-Z linker-flavor

[japaric]

(Please read the commit message first)

This PR is an alternative to #36120 (internal lld linker). The
main goal of this PR is to make it possible to use LLD as a linker to allow
out of tree experimentation. Now that LLD is going to be shipped with LLVM 4.0,
it should become easier to get a hold of LLD (hopefully, it will be packaged by
Linux distros soon).

Since LLD is a multiarch linker, it has the potential to make cross compilation
easier (less tools need to be installed). Supposedly, LLD is also faster than
the gold linker so LLD may improve build times where link times are significant
(e.g. 100% incremental compilation reuse).

The place where LLD shines is at linking Rust programs that don't depend on
system libraries. For example, here's how you would link a bare metal ARM
Cortex-M program:

$ xargo rustc --target thumbv7m-none-eabi -- -Z linker-flavor=ld -C linker=ld.lld -Z print-link-args
"ld.lld" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps/app-de1f86df314ad68c.0.o" \
  "-o" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps/app-de1f86df314ad68c" \
  "--gc-sections" \
  "-L" \
  "$PWD/target/thumbv7m-none-eabi/debug/deps" \
  "-L" \
  "$PWD/target/debug/deps" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib" \
  "-Bstatic" \
  "-Bdynamic" \
  "$XARGO_HOME/lib/rustlib/thumbv7m-none-eabi/lib/libcore-11670d2bd4951fa7.rlib"

$ file target/thumbv7m-none-eabi/debug/app
app: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped, with debug_info

This doesn't require installing the arm-none-eabi-gcc toolchain.

Even cooler (but I'm biased) is that you can link Rust programs that use
steed (steed is a std re-implementation free of C dependencies for Linux
systems) instead of std for a bunch of different architectures without having
to install a single cross toolchain.

$ xargo rustc --target aarch64-unknown-linux-steed --example hello --release -- -Z print-link-args
"ld.lld" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/aarch64-unknown-linux-steed/lib" \
  "$PWD/target/aarch64-unknown-linux-steed/release/examples/hello-80c130ad884c0f8f.0.o" \
  "-o" \
  "$PWD/target/aarch64-unknown-linux-steed/release/examples/hello-80c130ad884c0f8f" \
  "--gc-sections" \
  "-L" \
  "$PWD/target/aarch64-unknown-linux-steed/release/deps" \
  "-L" \
  "$PWD/target/release/deps" \
  "-L" \
  "$XARGO_HOME/lib/rustlib/aarch64-unknown-linux-steed/lib" \
  "-Bstatic" \
  "-Bdynamic" \
  "/tmp/rustc.lAybk9Ltx93Q/libcompiler_builtins-589aede02de78434.rlib"

$ file target/aarch64-unknown-linux-steed/release/examples/hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped, with debug_info

All these targets (architectures) worked with LLD:

• aarch64-unknown-linux-steed
• arm-unknown-linux-steedeabi
• arm-unknown-linux-steedeabihf
• armv7-unknown-linux-steedeabihf
• i686-unknown-linux-steed
• mips-unknown-linux-steed
• mipsel-unknown-linux-steed
• powerpc-unknown-linux-steed
• powerpc64-unknown-linux-steed
• x86_64-unknown-linux-steed

---

The case where lld is unergonomic is linking binaries that depend on system
libraries. Like "Hello, world" for x86_64-unknown-linux-gnu. Because you have
to pass as linker arguments: the path to the startup objects, the path to the
dynamic linker and the library search paths. And all those are system specific
so they can't be encoded in the target itself.

$ cargo \
  rustc \
  --release \
  -- \
  -C \
  linker=ld.lld \
  -Z \
  linker-flavor=ld \
  -C \
  link-args='-dynamic-linker /lib64/ld-linux-x86-64.so.2 -L/usr/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1 /usr/lib/Scrt1.o /usr/lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1/crtbeginS.o /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.1/crtendS.o /usr/lib/crtn.o'

---

Another case where -Z linker-flavor may come in handy is directly calling
Solaris' linker which is also a multiarch linker (or so I have heard). cc
@binarycrusader

cc @alexcrichton
Heads up: [breaking-change] due to changes in the target specification format.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @pnkfelix (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[binarycrusader]

Yes, the Solaris linker is multi-arch. As long as it is provided well-formed objects it should be able to link them.

With that said, I intend to evaluate what it would take to get lld working on Solaris later this year and teach it some Solaris-specific tricks.

[binarycrusader]

Oh, and btw, this is awesome as it will make it easier for us to teach rust how to use the Solaris linker directly.

[nagisa]

This kinda collides and duplicates my effort to refactor target specs altogether but since it is pretty similar to what I'm going for wrt linkers I guess it's fine.

[nagisa]

The target spec field needn't be required field as it can just default to gnu, just like pretty much everything else does already anyway

[binarycrusader]

/cc @dhduvall

[alexcrichton]

These same four strings have been repeated quite a few times throughout this PR, perhaps they could be centralized to one location though?

[alexcrichton]

Most of this seems duplicated from a definition above, would it be possible to share code instead of duplicating?

[alexcrichton]

Is an option like this still needed?

[alexcrichton]

This all sounds like a great idea to me, thanks @japaric! I think it's some good internal refactoring regardless as well.

As a minor bikeshed, can we perhaps rename LinkerFlavor::Gnu to LinkerFlavor::Gcc? That should be more explicit that we're expecting a C compiler with gcc-compatible options.

[japaric]

These same four strings have been repeated quite a few times throughout this PR, perhaps they could be centralized to one location though?

Done.

Most of this seems duplicated from a definition above, would it be possible to share code instead of duplicating?

I got some deduplication by making LdLinker a wrapper around GccLinker. I could maybe get more deduplication if I create a wrapper around Command that strips -Wl, from arguments ...

Is an option like this still needed?

Yes. GccLinker is like a baseline interface but some aspects of it are tweaked based on the truthtiness of the is_like_osx, is_like_gnu and the is_like_solaris options. We could create more specific Linkers to handle those subtle differences and get right of those flags but then we would have SolarisLdLinker, OsxGccLinker, SolarisGccLinker, GnuGccLinker and GnuLdLinker or something like that.

As a minor bikeshed, can we perhaps rename LinkerFlavor::Gnu to LinkerFlavor::Gcc?

Done

[alexcrichton]

Ah yeah for deduplication of gcc/ld I was thinking we could just strip -Wl, and such, otherwise the meaty parts to deduplicate (the version script info, other rpath business, etc) is still duplicated.

Also could the "did you mean ..." be deduplicated as well? Just trying to contain the number of places we have to update when we inevitably add a new flavor of linker :)

Also yeah that's true about is_like_foo is just within the GccLinker for now. Let's leave that as-is for a future cleanup.

[alexcrichton]

It also looks like Travis may be failing?

[alexcrichton]

ping @japaric

[mrhota]

squeaky wheel @japaric

[alexcrichton]

Closing due to inactivity, but feel free to resubmit with a rebase!

[japaric]

Rebased.

Ah yeah for deduplication of gcc/ld I was thinking we could just strip -Wl

Done. Merged GccLinker and LdLinker into a single GccLinker that decides whether to add the -Wl based on the value of a boolean is_ld field.

Also could the "did you mean ..." be deduplicated as well?

Done.

It also looks like Travis may be failing?

That rmake test should be fixed now.

[japaric]

Now that we have the unstable book, should I document -Z linker-flavor in there? Are people likely to look for more info about -Z linker-flavor in there? Note that -Z linker-flavor is not behind a #![]-style feature gate as it's a compiler flag.

I forgot to ask in my previous comment: r? @alexcrichton

[bors]

☔️ The latest upstream changes (presumably #40805) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

⌛️ Testing commit e192fb3 with merge 1e2cdd2...

[TimNN]

@bors retry

• prioritizing rollup which this is included in

[bors]

⌛️ Testing commit e192fb3 with merge 3b5754e...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 3b5754e to master...

[tamird]

FYI: this style (e.g. "-l" "System") isn't supported by ld64 on darwin - that linker requires the style "-lSystem". I'm testing a PR that changes this.