Integrate LLD for MSVC targets; prepare for other targets #2142

kinke · 2017-05-26T19:46:37Z

No description provided.

…_LLD Results in a 7.5% bigger RelWithDebInfo ldc2.exe on Win64 with LLVM 3.9.1. LLD is currently enforced when building with LDC_WITH_LLD=ON. And LLD still doesn't support debuginfo (.pdb) generation for MSVC targets.

kinke · 2017-05-27T19:40:36Z

Alright, I just tested directly cross-compiling and -linking from Linux x64 to Win64. It works without much hassle.

Build this PR, ~~using CMake option -DLDC_WITH_LLD=ON~~ (edit: not required anymore), with a recent LLVM incl. LLD (I used the current release_40 branch for both LLVM and LLD on Ubuntu 16.04).
Copy over system libs and object files from a Visual C++ installation. I used Visual C++ 2015 x64 Update 3 with the latest Win10 SDK. I merged all .{lib,obj} files in most directories specified in the LIB environment variable (VS tools command prompt) into a single new directory, renamed all those files to lowercase (most Linux filesystems are case-sensitive) and packed them to an archive. Raw size nearly 0.5 GB, .tar.xz about 32 MB.
Copy over LDC runtime files, e.g., from the latest Win64 release package (all files in the lib64 subdirectory).
Tweak the LDC config by adding a section like this (note the -L lib directories [edit:] and the additional -link-internally switch):

x86_64-pc-windows-msvc:
{
    // default switches injected before all explicit command-line switches
    switches = [
        "-I/home/martin/ldc/runtime/druntime/src",
        "-I/home/martin/ldc/runtime/profile-rt/d",
        "-I/home/martin/ldc/runtime/phobos",
        "-L-L/home/martin/Downloads/libs_msvc14_x64",
        "-L-L/home/martin/Downloads/ldc2-1.3.0-beta1-win64-msvc/lib64", 
        "-link-internally",
        "-defaultlib=phobos2-ldc,druntime-ldc",
        "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
    ];
};

You'll then be able to produce Win64 executables and DLLs by simply using the -mtriple=x86_64-pc-windows-msvc switch with LDC. I produced a Phobos-hello-world and a dmd-testsuite runnable\eh.d executable, copied them over to my Windows box and they worked.

kinke · 2017-05-27T19:52:59Z

This also means that if we are allowed to redistribute the MS libs, we could provide a self-sufficient package for Windows, similar to the DMD package, without relying on any Visual C++ installation (and well basically depending on nothing at all except for Windows these days - no libconfig, no shared C runtime, no external linker & libs, ...). A self-sufficient Win64-only package (no 32-bit libs) would be a ~50 MB download, just to give you an idea.

dnadlinger · 2017-05-27T20:22:43Z

Nice! Do you know what the licensing situation is on the MS libraries?

dnadlinger · 2017-05-27T20:23:20Z

How big is the impact of LDC_WITH_LLD=ON? I'd be keen to keep the number of options to a minimum (that is, turn on the LLD code by default as soon as it is merged).

kinke · 2017-05-27T20:31:05Z

Do you know what the licensing situation is on the MS libraries?

Nope, no idea.

How big is the impact of LDC_WITH_LLD=ON?

Size-wise for the ldc2 executable currently +7.5% with just the COFF output capability (Win64, RelWithDebInfo, LLVM & LLD 3.9.1). Mach-O and ELF still to come.
It's just a quick hack for now so that I don't need to check the LLVM version and LLD availability.

kinke · 2017-05-28T15:51:18Z

I got rid of LDC_WITH_LLD as user-provided CMake option and try to detect LLVM 3.9+ incl. LLD headers & libs now instead.
AppVeyor produces LLD-enabled LDC builds and so does Travis for LLVM 4.0 (Linux and OS X), but LLD isn't tested as it's opt-in via -internal-linker for now (still experimental, no .pdb support etc.).
[I updated the instructions above accordingly.]

dnadlinger · 2017-05-28T15:52:30Z

driver/linker.cpp

    staticFlag("static", llvm::cl::ZeroOrMore,
               llvm::cl::desc("Create a statically linked binary, including "
                              "all system dependencies"));

+#if LDC_WITH_LLD
+static llvm::cl::opt<bool>
+    useInternalLinker("internal-linker", llvm::cl::ZeroOrMore, llvm::cl::Hidden,


Maybe -link-internally or something like that?

dnadlinger · 2017-05-28T15:53:13Z

CMakeLists.txt

+    if(MSVC)
+        list(APPEND LDC_LINKERFLAG_LIST lldCOFF.lib lldCore.lib lldDriver.lib)
+    else()
+        set(LDC_LINKERFLAG_LIST "-llldCOFF;-llldCore;-llldDriver;${LDC_LINKERFLAG_LIST}")


Is there an equivalent of llvm-config for this?

Not that I've seen, in fact I didn't find anything about how to embed LLD via CMake. On Linux the order matters (even inbetween these 3 new libs), so I put them in front. As the LLD libs don't have the LLVM prefix, we can't specify them in find_package()...

dnadlinger · 2017-05-28T16:00:02Z

CMakeLists.txt

+#
+set(LDC_WITH_LLD OFF)
+if(LDC_LLVM_VER GREATER 308)
+    if(EXISTS "${LLVM_INCLUDE_DIRS}/lld/Driver/Driver.h")


Maybe just use the standard CMake header search functionality? ${LLVM_INCLUDE_DIRS}/… (with the plural) looks quite wrong.

CHECK_INCLUDE_FILE() didn't work (although there's a previous line with include_directories(SYSTEM ${LLVM_INCLUDE_DIRS}) - no extra include dirs appear to be used for the check).

Does CMAKE_REQUIRED_INCLUDES do the trick?

Yep it does.

Hmm it doesn't for Travis (LLVM 4.0). :/

gcc needs an additional -std=c++11 flag; fixed now.
Edit: Dammit, clang (Travis OSX) still fails.
Edit2: Works with clang 3.8 here in my VM.

Damn, I missed that the first Travis OSX job now uses the SPIRV LLVM which apparently doesn't feature LLD. So all good, LLD is dectected correctly for the other job.

dnadlinger · 2017-05-28T16:01:36Z

driver/linker-msvc.cpp

+  }
+
+  for (unsigned i = 0; i < global.params.linkswitches->dim; i++) {
+    addSwitch(global.params.linkswitches->data[i]);


This is probably code that existed before, but you can use (*global.params.linkswitches)[i] or our range-based for wrapper.

kinke · 2017-05-29T17:27:59Z

Yesterday I tried integrating LLD for ELF and Mach-O targets as well. Sadly, there's an unknown number of conflicting command-line options then (which only show up after linking, when running LDC; additionally, only the first conflicting option is shown as I had to find out). I renamed the -dwarf-version one, but then -march popped up and I stopped, as I guess that wasn't all. [I experimented on Win64 with LLD 3.9.]

Oh and the ldc2.exe size grew by about 20% compared to LDC without LLD at all.

And infer CMake LDC_WITH_LLD automatically based on availability of LLVM 3.9+ with LLD headers & libs.

JohanEngelen · 2017-05-29T22:18:45Z

(about cmdline options: I think it'd be best to someday go the Clang route: put all LLVM options behind a -mllvm switch)

JohanEngelen · 2017-05-29T22:18:58Z

Very cool work btw!

kinke · 2017-05-30T20:40:04Z

The problem is that LLD's ELF/LTO.cpp drags in an LLVM header with quite a number of command-line options, quite a few colliding with our duplicates, but also many new options; at least some would also make sense for LDC.

The include has been removed in LLD master, but it's there for 3.9 and 4.0.
Edit: Nope, just moved here.

kinke · 2017-06-02T17:40:35Z

I quickly looked into using that shared LLVM header ourselves (in a separate .cpp, so that it's only pulled in if it's not already via LLD), but what's required is a hack. The actually required shared utility functions (setting up a TargetMachine's TargetOptions according to all codegen command-line options etc. - very handy) are all marked as static (linkage), so one can't 'export' them directly. LLD master has its own little .cpp file including the LLVM header and augmenting it with their external wrappers in the lld namespace. They use it to prevent conflicts inside LLD itself. ;)
Unfortunately, for us this means that we'd have to forward to these external LLD wrappers for LDC_WITH_LLD (edit: oh I think they only 'exported' a subset of the utility functions, not all we'd need), and otherwise include the LLVM header and implement the wrappers ourselves. And as soon as we want to integrate another LLVM project including that header, with its own wrapper functions in some unknown namespace, it goes boom.

Laeeth · 2017-06-22T05:43:07Z

licensing
https://www.visualstudio.com/en-us/productinfo/2015-redistribution-vs#visual-c-runtime

or you could just include a little call to download and install vcredist* as a script in the repo. (older versions usually require logging in with MS account, unfortunately, but curl should pick up latest one ). I don't know if vcredist installer works on wine.

kinke · 2017-06-22T08:19:54Z

We don't need to redistribute the DLLs (easy enough to install and only needed when running binaries created with non-default -mscrtlib=msvcrt[d]); it's about the static and import .lib files.

MartinNowak · 2017-07-09T22:14:44Z

They do support static linking, so one would think that their license supported some form of redistribution.
But non of those lib files are in Visual Studio 2015 Redistribution List.

d-random-contributor · 2017-12-14T13:38:53Z

They allow redistribution of libraries after they are run through linker :)

kinke changed the title ~~Move static lib generation from driver/linker.cpp to driver/archiver.cpp~~ Move static lib generation from driver/linker.cpp to driver/archiver.cpp [NFC] May 26, 2017

Move static lib generation from driver/linker.cpp to driver/archiver.cpp

ec1a54a

kinke force-pushed the archiver branch from a420877 to ec1a54a Compare May 26, 2017 20:25

Refactor MSVC link.exe driver into dedicated driver/linker-msvc.cpp

cb0c27c

kinke changed the title ~~Move static lib generation from driver/linker.cpp to driver/archiver.cpp [NFC]~~ Split up driver/linker.cpp [NFC] May 26, 2017

kinke force-pushed the archiver branch 2 times, most recently from 94d5ce6 to e0b34e2 Compare May 26, 2017 23:37

kinke added 2 commits May 27, 2017 02:23

Move GCC driver to dedicated driver/linker-gcc.cpp

ecbfa0e

Refactor generation of GCC linker command line

9c86ae9

kinke force-pushed the archiver branch from e0b34e2 to 9c86ae9 Compare May 27, 2017 00:27

kinke added 3 commits May 27, 2017 02:42

Incorporate LTO stuff in GCC ArgsBuilder

1339c02

Introduce ArgsBuilder::addLdFlag()

9416d33

Add first version of ld linker driver as ArgsBuilder specialization

ede2364

kinke changed the title ~~Split up driver/linker.cpp [NFC]~~ Split up driver/linker.cpp and prepare for (L)LD May 27, 2017

kinke force-pushed the archiver branch from 3cd00e3 to 00d5f9b Compare May 27, 2017 19:21

kinke changed the title ~~Split up driver/linker.cpp and prepare for (L)LD~~ [WIP] Integrate LLD for MSVC targets; prepare for other targets May 27, 2017

kinke mentioned this pull request May 27, 2017

Feature: Integrate LLD, the LLVM cross-linker #2028

Closed

kinke changed the title ~~[WIP] Integrate LLD for MSVC targets; prepare for other targets~~ Integrate LLD for MSVC targets; prepare for other targets May 28, 2017

dnadlinger approved these changes May 28, 2017

View reviewed changes

kinke force-pushed the archiver branch from 1444581 to 9982c5e Compare May 29, 2017 19:18

Add experimental -link-internally switch to enable LLD opt-in

9fd7fa2

And infer CMake LDC_WITH_LLD automatically based on availability of LLVM 3.9+ with LLD headers & libs.

kinke force-pushed the archiver branch from 9982c5e to 9fd7fa2 Compare May 29, 2017 20:45

kinke force-pushed the archiver branch 2 times, most recently from 7533746 to 2acddf9 Compare June 2, 2017 20:39

Range-ify usages of front-end Arrays

383c2d3

kinke force-pushed the archiver branch from 2acddf9 to 383c2d3 Compare June 3, 2017 22:32

kinke merged commit 2008400 into ldc-developers:master Jun 4, 2017

kinke deleted the archiver branch June 4, 2017 02:02

kinke mentioned this pull request Jun 4, 2017

Use LLVM headers for shared codegen command-line options #2148

Merged

kinke mentioned this pull request Apr 4, 2018

Cross-compiling from Linux to Windows with mulitarch/crossbuild #2643

Closed

3 tasks

wilzbach mentioned this pull request Apr 5, 2018

Improve the deployment script dlang-community/D-Scanner#598

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate LLD for MSVC targets; prepare for other targets #2142

Integrate LLD for MSVC targets; prepare for other targets #2142

kinke commented May 26, 2017

kinke commented May 27, 2017 •

edited

Loading

kinke commented May 27, 2017 •

edited

Loading

dnadlinger commented May 27, 2017

dnadlinger commented May 27, 2017 •

edited

Loading

kinke commented May 27, 2017

kinke commented May 28, 2017 •

edited

Loading

dnadlinger May 28, 2017 •

edited

Loading

kinke May 29, 2017

dnadlinger May 28, 2017

kinke May 28, 2017

dnadlinger May 28, 2017

kinke May 28, 2017

dnadlinger May 28, 2017

kinke May 29, 2017

kinke May 29, 2017

kinke May 29, 2017 •

edited

Loading

kinke Jun 2, 2017

dnadlinger May 28, 2017

kinke commented May 29, 2017 •

edited

Loading

JohanEngelen commented May 29, 2017

JohanEngelen commented May 29, 2017

kinke commented May 30, 2017 •

edited

Loading

kinke commented Jun 2, 2017 •

edited

Loading

Laeeth commented Jun 22, 2017

kinke commented Jun 22, 2017 •

edited

Loading

MartinNowak commented Jul 9, 2017

d-random-contributor commented Dec 14, 2017

Integrate LLD for MSVC targets; prepare for other targets #2142

Integrate LLD for MSVC targets; prepare for other targets #2142

Conversation

kinke commented May 26, 2017

kinke commented May 27, 2017 • edited Loading

kinke commented May 27, 2017 • edited Loading

dnadlinger commented May 27, 2017

dnadlinger commented May 27, 2017 • edited Loading

kinke commented May 27, 2017

kinke commented May 28, 2017 • edited Loading

dnadlinger May 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kinke May 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kinke commented May 29, 2017 • edited Loading

JohanEngelen commented May 29, 2017

JohanEngelen commented May 29, 2017

kinke commented May 30, 2017 • edited Loading

kinke commented Jun 2, 2017 • edited Loading

Laeeth commented Jun 22, 2017

kinke commented Jun 22, 2017 • edited Loading

MartinNowak commented Jul 9, 2017

d-random-contributor commented Dec 14, 2017

kinke commented May 27, 2017 •

edited

Loading

kinke commented May 27, 2017 •

edited

Loading

dnadlinger commented May 27, 2017 •

edited

Loading

kinke commented May 28, 2017 •

edited

Loading

dnadlinger May 28, 2017 •

edited

Loading

kinke May 29, 2017 •

edited

Loading

kinke commented May 29, 2017 •

edited

Loading

kinke commented May 30, 2017 •

edited

Loading

kinke commented Jun 2, 2017 •

edited

Loading

kinke commented Jun 22, 2017 •

edited

Loading