[WIP] internal lld linker

[japaric]

this commit embeds lld, the LLVM linker, into rustc. If the -Z use-lld
flag is passed to rustc, then rustc will use lld (via a library call),
instead of an external linker like cc, to link a Rust executable.

This has been tested in three different ways:

• Linking Hello World, but the -pie flag had to be omitted:

Note that lld doesn't implicitly pass libraries (-lc) or startup
object (crt1.o) like cc does, so one has to manually pass these to lld
via -C link-args:

$ rustc -Z use-lld \
        -C link-args='-dynamic-linker /lib64/ld-linux-x86-64.so.2 -L/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /usr/lib/x86_64-linux-gnu/crtn.o'
        hello.rs

$ ./hello
Hello world!

• Linking a Rust program with zero C dependencies, i.e. a Rust "kernel".
  Basically, I grabbed the "Hello, world" program from Intermezzos chapter 3 1
  and turned it into a Cargo project 2 then proceeded to link it using lld.

This linked correctly and after turning the executable into a ISO image,
it also worked perfectly under QEMU!

• Linking A bare metal program for the Cortex-M3 microcontroller.

This didn't require a toolchain (arm-none-eabi-gcc) or startup files
(newlib). This test has been turned into a run-make test for this feature.

Extra notes

I've only tested ELF executables, not dylibs, MachO or COFF.

lld supports ELFs, MachO and COFF formats. So in theory, with an
internal lld we could drop the dependency on a external linker pretty
much on all the platforms we support.

Relevant news: "lld now implements almost all of the functionality
necessary to function as a system linker for FreeBSD/amd64" -- FreeBSD
folk @ LLVM mailing list 3. The e-mail also mentions that
support for other archs is not quite ready for prime time.

---

The question is: Would it make sense to start slowly integrating lld
into rustc right now? It would land behind an unstable flag '-Z
use-lld' and we could print a warning message about it being
experimental whenever the flag is used.

Scenarios where it could be advantageous to not depend on a external
linker:

• Building bare-metal/C-free programs, e.g. x86 kernel
  development (already shown to work), embedded (microcontroller) ARM
  programs (once lld supports Thumb). Today, to build C-free Rust
  programs you still need a C linker (cc). Would be nice to drop the
  dependency on cc if you are not going to compile C code.
• @Diggsey mention that lld + the arm-musl targets effectively means one
  can cross compile ARM binaries without relying on anything external:
  no need for arm-gcc or arm-glibc. Just rustup target add $T and
  cargo build --target $T and you are done. This scenario hasn't been
  tested though.

Downsides:

• lld is still being actively developed. some architectures have limited
  support and features like linker scripts are not on parity with ld
  yet. However, as mentioned above, the FreeBSD folks have had success
  using it as a system linker on x86_64. At least, that architecture
  appears to be reaching stability.
• Maintenance. Upgrading LLVM means also having upgrading lld as those
  need to be keep in sync or lld won't compile. This cuts both ways: if
  we want to upgrade lld to support some new feature or fix some bug,
  it's likely we'll have to upgrade LLVM as well -- this possibly means
  updating LLVM more often.

P.S. This a PoC; the implementation is a mess/hack. It would have to be
cleaned up before merging it.

cc #9367
cc @alexcrichton @brson

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @eddyb (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[eddyb]

😮 Amazing!

As for knowing what flags to pass: clang -v prints this for me:

clang version 3.8.0 (tags/RELEASE_380/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/wdx8qncj8zg51g2rra89h9y95l6pd4xf-clang-3.8.0/bin
Found candidate GCC installation: /nix/store/haqh3hshg8w1779xgyzr4pk9q706ra30-gcc-5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0
Found candidate GCC installation: /nix/store/haqh3hshg8w1779xgyzr4pk9q706ra30-gcc-5.4.0/lib64/gcc/x86_64-unknown-linux-gnu/5.4.0
Selected GCC installation: /nix/store/haqh3hshg8w1779xgyzr4pk9q706ra30-gcc-5.4.0/lib64/gcc/x86_64-unknown-linux-gnu/5.4.0
Candidate multilib: .;@m64
Selected multilib: .;@m64

So we might be able to borrow some detection code from clang and only depend on a gcc installation, without actually invoking it to link.

EDIT: furthermore, we can do this in liblibc's build script 😆.
Using the platform's crt*.o is not doable through that though, AFAIK.

[bors]

☔️ The latest upstream changes (presumably #36117) made this pull request unmergeable. Please resolve the merge conflicts.

[japaric]

@eddyb

So we might be able to borrow some detection code from clang and only depend on a gcc installation, without actually invoking it to link.

I haven't read clang source code, but my guess is that it probably finds the system libraries by looking in relative paths from the location of the clang binary. So, basically $(dirname $(which clang))/../lib or something like that.

furthermore, we can do this in liblibc's build script

How would that work? If I build rustc on e.g. nixos, the library search path that gets hardcoded in liblibc would be /nix/store/.., but then that rustc wouldn't work on e.g. Ubuntu because that's no where system libraries are.

We have to decide how to solve this system library search path before lld can be used to build std programs. So far, we have been relying on gcc; this has worked well so far, even when cross compiling because the cross linker (arm-linux-gnueabihf-gcc) tells us where the cross compiled system libraries are. But without gcc, we either have to ask the user to supply the system library search path for each target, which is annoying for the user, or build some elaborate mechanism to auto-detect where the system libraries are even when cross compiling, which seems like it would get complicated quickly and be prone to ambiguity -- what if there are multiple sysroot for the same target triple but slightly tweaked codegen options (+neon and -neon for instance).

---

On another news, I tried to link a Rust program for x86_64-musl using lld but it errored with this message:

$ rustc -Z use-lld --target x86_64-unknown-linux-musl zero.rs
duplicate symbol: __tls_get_addr in /build/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-0a08e49eef7bbac5.rlib(__tls_get_addr.lo) and (internal)

Haven't look further.

[eddyb]

@japaric The relative thing is what gcc uses, but clang finds paths relative to gcc, not to itself.
As for detection in liblibc, I was thinking that the libc crate itself would track the native dependency on glibc or the respective platform C library, which would make liblibc-*.rlib distro-dependent.

It'd work for distro-built packages, but not distribution via, e.g. rustup, unless rustup itself took care of symlinking the libraries relative to rustc (which would work better with rustc handling the paths instead of having them stored in liblibc-*.rlib, anyway).

[alexcrichton]

Nice @japaric! It's actually pretty exciting to me to see how easy this is :)

I agree that the next big question is whether to start landing this in tree. To me I think the threshold for that is that we have at least a test for this functionality, but once we have that I'd be fine fleshing out more platforms as they arise. I think it's definitely correct leaving this behind an unstable -Z flag for now.

It's probably worth checking out the impact this has on the compiler though, just to make sure. For example:

• How much larger is the compiler binary-wise?
• Does the lld library actually compile on all our host platforms?

cc @brson

[brson]

This is sweet.

I think we should pursue this, as long is the implementation is cleanly separated from the existing linker code.

Some things I'd suggest:

• lld needs to be optional at build time. Distros will not want to take an lld dep, and that gives us a switch to shut it off if an upgrade goes bad.
• I'd prefer lld is its own crate and not conflated with either rustllvm or librustc_llvm.

[japaric]

@alexcrichton

To me I think the threshold for that is that we have at least a test for this functionality

Well, right now it can successfully link x86_64 ELFs but the produced binaries segfault when executed. At least that's the case for Rust programs that link to libc. As shown above, a Rust program that doesn't link to libc executes correctly, but that program must executed under QEMU because it's bare metal.

I don't know if just being able to link an executable passes the bar or if we should pursue integrating that bare metal program into our test suite; that would require qemu, perhaps a new test category (run-pass-qemu), etc.

How much larger is the compiler binary-wise?

Without this PR (stage1):

librustc_back 533568    (522K)
librustc_llvm 39763616  (38M)
total         134298492 (129M)

With this PR (stage1):

librustc_back 561888    (549K) +5.31%
librustc_llvm 42895600  (41M)  +7.88%
total         137487524 (132M) +2.37%

Does the lld library actually compile on all our host platforms?

Good question! I don't have a windows/osx box to test. Perhaps we could throw this into the build bots (bors' try) and see if it works?

Also, with the configure flag @brson suggested, we could, for now, simply not embed lld on the platforms where it doesn't compile.

@brson

the implementation is cleanly separated from the existing linker code.

That's a good idea. I'll try your suggestions in my next iteration.

[retep998]

While this PR isn't doing it, I'd like to point out that I am strongly opposed to any move to remove the ability to use the system linker. LLD will never achieve feature parity with all the system linkers out there, link.exe being one of the biggest examples‡. Also it wouldn't really buy us anything for -msvc targets anyway, because we still rely on a lot of libraries that come with the msvc toolchain, so we always have a working system linker.

_{‡Unless Microsoft seriously steps up their open source game and adds all the stuff to LLD so that it can handle LTCG stuff as well as PDB debuginfo and all those store app specific features, and other details. I doubt this will happen, but then again Microsoft did do Clang/C2 so who knows.}

[eddyb]

@retep998 rustc will likely always need to support invoking external linkers, it may just move away from it by default.

[alexcrichton]

Yeah I'd be fine throwing this to bors-dev at any point. @japaric I wonder if we could add a test for the linux MUSL case? E.g. some run-make test which is super gated and only runs in some situations, but that should in theory be our banner use case right? That is, if we have a standard library for x86_64 musl we should be always able to produce a binary.

[eddyb]

@retep998 Microsoft only needs to expose an API through which a gold or LLD plugin could be written.

[japaric]

I wonder if we could add a test for the linux MUSL case?

It doesn't work though ...

$ rustc -Z use-lld --target x86_64-unknown-linux-musl zero.rs
duplicate symbol: __tls_get_addr in /build/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-0a08e49eef7bbac5.rlib(__tls_get_addr.lo) and (internal)

Nor with lld-3.9-rc3 (this PR) or with lld-4.0 (yesterday-ish HEAD). The only case that works so far is C free executables.

[japaric]

@retep998 reports that lld can link Rust programs on msvc. And one can even use lld to link programs for msvc on Linux:

2016-09-04 21:00:52 WindowsBunnyFrustratedWithCargo JaparicFrustratedWithWindows: LLD as a separate binary does work for msvc windows. I've tried it recently and it does work
2016-09-04 21:01:14 WindowsBunnyFrustratedWithCargo People have even cross compiled rust code from linux to windows with it

I'll try that locally and see if I can turn it into a test.

[alexchandel]

Microsoft's link.exe LTCG option tells it to do "whole-program optimization", i.e. LTO, which LLD should eventually be able to do for msvc targets.

[eddyb]

@alexchandel Needs cooperation from Microsoft to actually invoke their equivalent of LLVM.

[japaric]

Update: lld-3.9 can link Rust binaries cross compiled for the thumb*-none-eabi* targets. This means
one can link Rust programs for Cortex-M microcontrollers without having arm-none-eabi-gcc installed
(yay, one less dependency). The issue is that linker script support in lld-3.9 is so immature that
one can't control the memory layout of the produced program and therefore the produced binaries
can't be flashed to real hardware (microcontrollers have very specific memory layout requirements).

However, linker script support in lld-HEAD is already good enough that you can control the memory
layout of Rust programs (It's not straightforward because the linker script support is not on par
with GNU ld but it's doable) and produce binaries that can be flashed to real hardware -- I've
tested this; the flashed program works but debug information is incomplete (lld likely dropped it):
stepi works but you can't see on which line of the source code you currently are and step sort
of just executes the program to the end.

@alexcrichton is there a schedule for the next llvm update? I'd like to update this PR to use the
latest lld (there has been a lot of work on improving lld's linker script support recently) but
that lld depends on LLVM-4.0 and we are using llvm-3.9-ish.

[alexcrichton]

Whoa, awesome! Currently there's no plans for the next LLVM update, but I don't think there's any particular reason to hold back either. Nowadays we just probably want to coordinate with emscripten to ensure that we update roughly at the same time.

[japaric]

@brson

Some things I'd suggest:

Done

---

I've also added a rmake test that we can link a bare metal executable for the ARM Cortex M without having any cross toolchain installed. This test needs to be disabled when lld support is not built into rustc (th default), but I'm not sure how to do that. Do we have an ignore option for that?

---

@alexcrichton

I'd like to throw this into the try buildbots but (a) this only works with rustbuild and (b) lld support is disabled by default. I'd have to toggle both rustbuild and lld to on by default before sending this to "try" but I'm not quite sure how to do that in the configure script. Any ideas?

---

P.S. the implementation is still a mess :-)

[alexcrichton]

Do we have an ignore option for that?

Not currently that I know of, but we proxy a bunch of random env vars down to compiletest so what's one more :)

I'd like to throw this into the try buildbots but (a) this only works with rustbuild and (b) lld support is disabled by default

Sure, I can throw to the bots. Wouldn't we want to enable lld by default though if we're building LLVM ourselves?

[rui314]

Hi, an LLD dev here. If you find any mislinking bug, please report it to me or to bugs.llvm.org. I'm happy to help solving issues.

[japaric]

@rui314

Thanks for chipping in!

I've had some problems linking programs with debuginfo enabled ("SHF_MERGE section not multiple of
entsize" or something like that) and statically linking programs against musl and I intend
to appropriately report those in bugs.llvm.org in due time (if they have not been already reported,
that's it).

---

@alexcrichton

Wouldn't we want to enable lld by default though if we're building LLVM ourselves?

Yes, I just pushed a commit that does that. If the try buildbots are using rustbuild (passing --enable-rustbuild to configure) then we can be sent this to bors try already. Otherwise, we would have to push another commit enabling rustbuild by default (withouth --enable-rustbuild)? (this is what I was referring to in my previous comment)

[alexcrichton]

There are a bunch of failures in dev, but they all look build related (not sure if anything to to tests)

[rui314]

@japaric LLD has a convenient option to report a bug. If you add --reproduce foo, it'll create foo.cpio archive file with all input files and command line options, so that we can run the same command with the same input very easily. When you report a bug, please use that feature and attach a created cpio file.

[bors]

☔️ The latest upstream changes (presumably #35021) made this pull request unmergeable. Please resolve the merge conflicts.

[japaric]

@rui314 Wow, that's nifty. Will make use of it.

---

Update: On Linux, I can now link Hello world for the host 🎉. But (a) I had to omit passing
-pie to lld (otherwise I get a binary that segfaults) and (b) I had to pass a few extra linker
arguments that are system dependent (basically, a bunch of paths). See below:

(The base of the diff is how rustc currently invokes lld when linking hello world)

 "ld.lld" \
     "--as-needed" \
     "-z" \
     "noexecstack" \
     "-L" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" \
     "hello.o" \
     "-o" \
     "hello" \
     "--gc-sections" \
-    "-pie" \
     "-L" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" \
     "--Bstatic" \
     "--Bdynamic" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcollections-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_unicode-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-411f48d3.rlib" \
     "$sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-411f48d3.rlib" \
     "-l" \
     "dl" \
     "-l" \
     "pthread" \
     "-l" \
     "gcc_s" \
     "-l" \
     "pthread" \
     "-l" \
     "c" \
     "-l" \
     "m" \
     "-l" \
     "rt" \
     "-l" \
     "util" \
     "-l" \
-    "compiler-rt"
+    "compiler-rt" \
+    -dynamic-linker \
+    /lib64/ld-linux-x86-64.so.2 \
+    -L/usr/lib/gcc/x86_64-linux-gnu/5 \
+    -L/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu \
+    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o \
+    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o \
+    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o

why we have to (explicitly) pass these to lld?

When we use cc as a linker, cc implicitly provides these paths to the dynamic linker and startup
objects plus library search paths to ld, the real linker, but lld doesn't know about those.
Whether we should make rustc search for these paths at link time or whether we should hard-code
these paths into rustc when rustc is bootstrapped is undecided.

In the meantime, should we (a) temporarily disable -pie when lld is used (and perhaps issue a
warning?) and (b) add a run-make test that builds and runs Hello world by passing hardcoded paths,
that (probably will only) work on the buildbot, to the linker -- basically something like this:

all:
    rustc -C link-args="-dynamic-linker $A -L$B -L$C $C/crt1.o $C/crti.o $C/crtn.o" -Z use-lld hello.rs 
    ./hello

where $A, $B and $C are chosen to make this pass on the buildbots.

Thoughts, @alexcrichton?

[m4b]

@rui314 I reported this issue about 6 months ago:

https://llvm.org/bugs/show_bug.cgi?id=27194

I haven't checked with latest to see if it's been fixed since, but it was never addressed in bug report afaik.

[jgouly]

Is there a reason to go for embedding LLD directly first, rather than teaching rustc to call it externally first, and then using the API once all the details for library paths etc have been worked out.

[japaric]

@jgouly

The advantage is that by embedding lld we can start using this feature for two use cases

• building programs that don't depend on any C system library e.g. bare-metal programs for
  microcontrollers or kernels/OSes.
• building MUSL programs (native or cross). These ship with their own libc.a and startup objects.

where using lld has the advantage of having to install one less thing: the cross toolchain (gcc and
libc). But if you have to install (and likely build yourself) an external lld then the advantage of
one less dependency is lost.

Also, for these two cases we don't need to decide how to handle the search of system libraries,
startup objects, etc. because those are not needed in these cases.

teaching rustc to call it externally first

This would require adding special logic to rustc to detect if lld is being used so it can adjust
the linker flags accordingly (this last part about the linker flags is included in this PR). This
approach would still have the problem of library paths.

[rui314]

@m4b LLD is three linkers in one executable. I'm the owner of COFF (Windows) and ELF (Unix) but Mach-O is maintained by other people.

[alexcrichton]

@japaric ok pushed to http://54.176.156.253/grid, may take a moment to show up though

[alexcrichton]

@japaric looks like the lld submodule may be busted?

[japaric]

@alexcrichton sorry, I forgot to push my local lld changes. Could you try again? Or should I rebase this first?

[alexcrichton]

Looks like rustbuild tests passed but perhaps makefile ones didn't?

[alexcrichton]

(also problems on Windows)

[bors]

☔️ The latest upstream changes (presumably #36456) made this pull request unmergeable. Please resolve the merge conflicts.

[cmr]

@japaric what's the status of this?

[japaric]

@cmr Waiting for a LLVM-UP (4.0) and for the upstream bugs to fix themselves so I get motivated enough to fix the WIndows build errors (yuck!). On a more serious note, I don't think I'll have time to work on this until the next month.

[alexcrichton]

Ok, closing this due to inactivity for now, but I'd be totally fine with a resubmission of this!

[LinusU]

@japaric Now that LLVM 4 has been merged, and rust already have some knowledge about lld -Z linker-flavor; do you think that work can resume on this?

[berkus]

I'd be very happy with lld used instead of GNU ld.