Link with ld.gold by default #29974

brson · Nov 21, 2015

When using cc for linking rustc will, if gold is available (by looking for /usr/bin/ld.gold), pass -fuse-ld=gold to cc.

In some scenarios gold links much faster than ld. Servo uses it to considerably speed up linking. gold behaves nearly identically to ld (though I think there are rare corner cases that don't work still). I've run this through crater and everything there continues to link.

To disable, pass -C disable-gold.

rust-highfive · Nov 21, 2015

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

nagisa · Nov 21, 2015

nagisa reviewed Nov 21, 2015

View changes

nagisa · Nov 21, 2015

src/librustc_trans/back/linker.rs

@@ -199,6 +200,24 @@ impl<'a> Linker for GnuLinker<'a> {
    fn export_symbols(&mut self, _: &Session, _: &CrateTranslation, _: &Path) {
        // noop, visibility in object files takes care of this
    }
+
+    fn try_gold_linker(&mut self) {
+        let gold_exists = Path::new("/usr/bin/ld.gold").exists();


You should check for it in all pathes listed in $PATH, not hard-code /usr/bin.

This is very important on, e.g. NixOS, where /usr/bin only contains env.
EDIT: Seems to have been brought up by @nagisa already.

nagisa · Nov 21, 2015

nagisa reviewed Nov 21, 2015

View changes

nagisa · Nov 21, 2015

src/librustc_trans/back/linker.rs

+
+    fn try_gold_linker(&mut self) {
+        let gold_exists = Path::new("/usr/bin/ld.gold").exists();
+        let opt_out = self.sess.opts.cg.disable_gold;


I’d rather to just have a regular if here: if self.sess.opts.cg.disable_gold { return; }

alexcrichton · Nov 22, 2015

r=me with probing all of PATH instead of just /usr/bin

brson · Nov 23, 2015

I'm not sure I agree with probing $PATH. Are there known cases where gold appears somewhere else?

Edit: Ah, I'll just do it

petrochenkov · Nov 23, 2015

In MinGW ld.gold is in $PATH, but it can't produce Windows executables AFAIK. Won't this patch break everything on Windows by default?

brson · Nov 23, 2015

I'll conditionalize it on linux too then.

alexcrichton · Nov 23, 2015

I figured that if you go out of your way to install ld.gold to a custom location it may as well be nice for us to pick it up by default without requiring a modification of /usr/bin

nagisa · Nov 23, 2015

I'm not sure I agree with probing $PATH. Are there known cases where gold appears somewhere else?

I figured that if you go out of your way to install ld.gold to a custom location it may as well be nice for us to pick it up by default without requiring a modification of /usr/bin

Some distribution/OSes/package managers install to /usr/local/bin by default (and do not link /usr/bin/), then there’s NixOS based things which install to random-looking pathes and may link things into arbitrary pathes.

nagisa · Nov 23, 2015

nagisa reviewed Nov 23, 2015

View changes

nagisa · Nov 23, 2015

src/librustc/session/config.rs

@@ -515,6 +515,8 @@ options! {CodegenOptions, CodegenSetter, basic_codegen_options,
        "explicitly enable the cfg(debug_assertions) directive"),
    inline_threshold: Option<usize> = (None, parse_opt_uint,
        "set the inlining threshold for"),
+    disable_gold: bool = (false, parse_bool,


This is… a pretty confusing option (both its description and behaviour).

We already have -C linker which possibly swaps out gcc for something else and now we pass a special flag into linker assuming it is gcc-compatible (in absence of this option) without checking if specified linker is indeed compatible. That might make -C linker harder to use.

Maybe it would be better to have -C linker=gcc which overrides any default/dubious flags we might add?

We already have -C linker which possibly swaps out gcc for something else and now we pass a special flag into linker assuming it is gcc-compatible (in absence of this option) without checking if specified linker is indeed compatible. That might make -C linker harder to use

As far as I can tell that is an existing problem. Even if you override the linker, rustc assumes it takes either gcc or link.exe style arguments, depending on platform. IOW I tihnk -C linker is only good for specifying a different path to the expected linker, not for using a completely different 'style' of linker.

I'm not super worried about random codegen options being deprecated at some point, an option like this is pretty flavorful and likely to only be a one-time use case (if ever), so if we in the future just hide this and make it ignored by default then it's probably not too bad.

It may be the case that gcc -fuse-ld=gold -fuse-ld=not-gold (or some similar incantation) also works, so we could just be sure to allow that pattern, but I don't know if it exists. Either way this seems fine to me for now.

nagisa · Nov 23, 2015

Other than my nits, I really want this to land. ld.gold is, and always has been, a superior linker compared to the default.

michaelwoerister · Nov 24, 2015

🐤 👍

brson · Nov 24, 2015

I've updated it to search $PATH and to only work on linux. I haven't addressed @nagisa's comment about the confusingness of the -C disable-gold option. It's true that there are current and future configurations where it will not do anything, and also that -C linker is limited in what it actually does. I don't have a strong opinion about what the command line args look like here.

nagisa · Nov 24, 2015

nagisa reviewed Nov 24, 2015

View changes

nagisa · Nov 24, 2015

src/librustc_trans/back/linker.rs

+
+        let gold_exists = match env::var("PATH") {
+            Ok(env_path) => {
+                env_path.split(":").any(|p| {


Use split_paths instead.

nagisa · Nov 24, 2015

nagisa reviewed Nov 24, 2015

View changes

nagisa · Nov 24, 2015

src/librustc_trans/back/linker.rs

+            return;
+        }
+
+        let gold_exists = match env::var("PATH") {


There’s no reason to use var over var_os.

An example from another crate

nagisa · Nov 24, 2015

nagisa reviewed Nov 24, 2015

View changes

nagisa · Nov 24, 2015

src/librustc_trans/back/linker.rs

@@ -199,6 +201,37 @@ impl<'a> Linker for GnuLinker<'a> {
    fn export_symbols(&mut self, _: &Session, _: &CrateTranslation, _: &Path) {
        // noop, visibility in object files takes care of this
    }
+
+    fn try_gold_linker(&mut self) {
+        if !cfg!(target_os = "linux") {


I’m not necessarily opposed to having default on linux only, but did you consider the fact that FreeBSD, OpenBSD, DragonflyBSD, and similar UNIX-like OSes use ELF and are likely to have working ld.gold as well?

No I didn't. I'm reluctant to impose this on them without proof it works though. We have lots of visibility into Linux, little into the BSDs. If a BSD maintainer wants gold, perhaps they can submit a follow up verifying that it works.

alexcrichton · Nov 25, 2015

@petrochenkov

In MinGW ld.gold is in $PATH, but it can't produce Windows executables AFAIK. Won't this patch break everything on Windows by default?

I wouldn't be too worried about this, personally. That seems super unlikely to ever happen (if it doesn't support windows) and even if it does exist it'll exist as ld.gold.exe most likely which this isn't even probing for. @nagisa's point about OSes like FreeBSD and such also lean me towards just not checking the OS at all and blindly assuming that if ld.gold exists it's the "right linker" to use.

nagisa · Nov 25, 2015

A very relevant DragonflyBSD commit

brson · Dec 1, 2015

Gold only works reliably on x86 / x86_64 so this should probably also check that the target is x86 to avoid asking for gold on cross-compiles.

Edit: maybe arm?

Actually, checking that gold exists anywhere then asking for it for a cross-compile is probably bad. Maybe the check should be that the target == host and is a supported arch.

brson · Dec 1, 2015

@bstrie suggests this requires an RFC. cc @rust-lang/compiler

nagisa · Dec 1, 2015

@brson my ld.gold --help prints this:

ld.gold: supported targets: elf64-littleaarch64 elf64-bigaarch64 elf32-littleaarch64 elf32-bigaarch64 elf64-tradlittlemips elf32-tradlittlemips-nacl elf64-tradbigmips elf32-tradlittlemips-nacl elf32-tradlittlemips elf32-tradlittlemips-nacl elf32-tradbigmips elf32-tradlittlemips-nacl elf32-tilegx-be elf64-tilegx-be elf32-tilegx-le elf64-tilegx-le elf32-bigarm elf32-bigarm-nacl elf32-littlearm elf32-littlearm-nacl elf64-powerpcle elf64-powerpc elf32-powerpcle elf32-powerpc elf64-sparc elf32-sparc elf32-x86-64 elf32-x86-64-freebsd elf32-x86-64-nacl elf64-x86-64 elf64-x86-64-freebsd elf64-x86-64-nacl elf32-i386 elf32-i386-freebsd elf32-i386-nacl
ld.gold: supported emulations: aarch64_elf64_le_vec aarch64_elf64_be_vec aarch64_elf32_le_vec aarch64_elf32_be_vec elf64-tradlittlemips elf32-tradlittlemips-nacl elf64-tradbigmips elf32-tradlittlemips-nacl elf32-tradlittlemips elf32-tradlittlemips-nacl elf32-tradbigmips elf32-tradlittlemips-nacl elf32tilegx_be elf64tilegx_be elf32tilegx elf64tilegx armelfb armelfb_nacl armelf armelf_nacl elf64lppc elf64ppc elf32lppc elf32ppc elf64_sparc elf32_sparc elf32_x86_64 elf32_x86_64_nacl elf_x86_64 elf_x86_64_nacl elf_i386 elf_i386_nacl

Are you claiming most of these targets do not work?

brson · Dec 1, 2015

@nagisa No, I'm claiming I have no evidence they do. I know that I read just today that gold's powerpc support is incomplete.

Edit: I see my previous statement was strongly saying gold only works on x86. That was a mistake.

brson · Dec 1, 2015

I made this patch more defensive. Now it only uses gold when ld.gold exists, the host is linux, the host and the target are roughly the same, and the target is x86/x86_64. As @nagisa points out there are other platforms that gold support, but I have most confidence this won't break anything on x86 linux.

xitep · Dec 31, 2015

Hello,

being bound to Debian Wheezy, I'm now forced to use -C disable-gold due to gcc not supporting -fuse-ld=gold on this platform:

> cc -fuse-ld=gold -v
cc: error: unrecognized command line option ‘-fuse-ld=gold’
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.7.2 (Debian 4.7.2-5)

Unfortunately, I'm stuck with cargo based projects containing a custom build.rs as I haven't found a way to pass disable-gold to the compilation of that script (either directly in my project or any of its dependencies.) Any ideas how I could help myself without having to upgrade gcc?

eddyb · Dec 31, 2015

@xitep You can write a wrapper shell script for rustc that passes -C disable-gold and the rest of the arguments to the real binary.

xitep · Dec 31, 2015

@eddyb thank you so much for the idea! it works nicely.

nagisa · Jan 1, 2016

This might have killed play.rust-lang.org. see e.g. https://play.rust-lang.org/?gist=5dc29f8c79c267056ca2&version=nightly cc @alexcrichton @brson

birkenfeld · Jan 5, 2016

Another bit of incompatibility: gold does not have the same default library search path; in particular, it does not search in /usr/local/lib. This has broken building bindings for me, especially since there doesn't seem to be an easy way to add a libdir in Cargo (without build script)?

eddyb · Jan 5, 2016

@birkenfeld That sounds like a distro bug rather than rustc's fault, unless it's intentional, in which case... what's the reasoning?

birkenfeld · Jan 5, 2016

You mean, the library being in /usr/local? That's a reasonable location for self-built or site-distributed binaries, and the libs I'm writing bindings are not distributed by the system package manager.

eddyb · Jan 5, 2016

@birkenfeld I was referring to "gold does not have the same default library search path".

birkenfeld · Jan 5, 2016

Ah! That, as far I could find out from quick Googling, is "normal" behavior for gold. Apparently ld.bfd includes the runtime linker search path (i.e. ld.so.conf and LD_LIBRARY_PATH) when looking for libs, while gold doesn't (but it respects LIBRARY_PATH).

Possibly gold's behavior is cleaner here, I don't know. But it's definitely annoying to trip over these kinds of problems without a good way to disable gold from Cargo.

eddyb · Jan 5, 2016

@birkenfeld if gold repsects LIBRARY_PATH, can't you just put /usr/local/lib in there?

birkenfeld · Jan 5, 2016

Yes, I can (and I will).

Just noting this as a data point to consider before stabilizing this change (one which likely doesn't show up in a crater run).

nagisa · Jan 5, 2016

ld.gold not respecting ld.so.conf would be pretty crazy. That being said, ldconfig doesn’t search /usr/local/lib by default, so the behaviour might be as expected.

brson · Jan 8, 2016

This broke playpen on nightly.

brson · Jan 8, 2016

being bound to Debian Wheezy, I'm now forced to use -C disable-gold due to gcc not supporting -fuse-ld=gold on this platform:

This seems really bad. So when you use the Rust downloaded from official sources, it doesn't work on Debian? I think rustc needs to be able to figure out this scenario.

brson · Jan 8, 2016

@xitep Do you know how we could identify configurations like yours and automatically disable gold?

brson · Jan 8, 2016

@xitep @birkenfeld I filed issues for both the regression you identified.

birkenfeld · Jan 8, 2016

Thanks!

xitep · Jan 8, 2016

@brson many thanks! so far i haven't searched intensively, but it looks like something along the lines of cc -fuse-ld=gold -v 2>&1 | grep 'unrecognized command line option' would be necessary; this would be agnostic to the host environment merely relying on cc itself. but it would require an extra invocation of the compiler. btw: relying on the exit code of cc is not possible really in this case as it ways results in exit status zero when being passed -v. however, -v is used to make it happy with no input files specified.

you can see the full output on my debian wheezy (stable) of the cc -fuse-ld=gold -v command in my comment above.

lhecker · Jan 10, 2016

I think I should mention, @brson, that ld.gold is also a bit more error prone on non-standard filesystems... It fails to link everytime I use it on a shared folder (the "hgfs" filesystem) inside a VMware instance. While I don't think that this is a huge issue I would recommend to note somewhere that ld.gold is at fault here (and not Rust). Maybe even directly in the output of rustc...

brson · Jan 12, 2016

Thanks for the interesting note @lhecker.

brson · Jan 12, 2016

@xitep Thanks for the investigation on detecting -fuse-ld. I'm noting it in the bug.

rust-highfive assigned alexcrichton Nov 21, 2015

brson force-pushed the brson:gold branch from e6b9c48 to b5cc5d7 Nov 21, 2015

bluss added the relnotes label Nov 21, 2015

brson referenced this pull request Dec 1, 2015
Closed
Investigate the impact of using Gold for linking rather than the default linker on systems where it is available #29938

brson added T-compiler I-nominated labels Dec 1, 2015

brson force-pushed the brson:gold branch from 2da4c30 to 4dc0e89 Dec 1, 2015

niconii referenced this pull request Jan 1, 2016
Closed
Playpen currently fails to compile anything in nightly channel #165

This was referenced Jan 8, 2016

Open

rustc should attempt to use the gold linker by default, but this breaks Debian Wheezy because of obsolete gcc #30783

Open

Linking with gold does not search /usr/local/lib #30784

lhecker referenced this pull request in zonyitoo/coio-rs Jan 10, 2016

Fixed merge errors with 6131325

Loading status checks…

a04888e

larsbergstrom referenced this pull request Jan 11, 2016
Merged
Update to rustc 1.7.0-nightly (b4707ebca 2015-12-27) #9093

brson removed the relnotes label Jan 14, 2016

brson referenced this pull request Jan 14, 2016
Merged
Revert "Link with ld.gold by default" #30913

bors added a commit that referenced this pull request Jan 14, 2016

Auto merge of #30913 - brson:nogold, r=alexcrichton
This reverts commit 34dc0e0. cc #30783 #30784 #29974 r? @alexcrichton

c12c42d

SimonSapin referenced this pull request Nov 27, 2017
Merged
Use rustup.rs instead of custom bootstrap #19395

rust-lang/rust

Join GitHub today

Link with ld.gold by default #29974

Conversation

brson commented Nov 21, 2015

rust-highfive assigned alexcrichton Nov 21, 2015

This comment has been minimized.

rust-highfive commented Nov 21, 2015

brson force-pushed the brson:gold branch from e6b9c48 to b5cc5d7 Nov 21, 2015

nagisa reviewed Nov 21, 2015 View changes

This comment has been minimized.

This comment has been minimized.

nagisa reviewed Nov 21, 2015 View changes

This comment has been minimized.

bluss added the relnotes label Nov 21, 2015

This comment has been minimized.

alexcrichton commented Nov 22, 2015

This comment has been minimized.

brson commented Nov 23, 2015

This comment has been minimized.

petrochenkov commented Nov 23, 2015

This comment has been minimized.

brson commented Nov 23, 2015

This comment has been minimized.

alexcrichton commented Nov 23, 2015

This comment has been minimized.

nagisa commented Nov 23, 2015

nagisa reviewed Nov 23, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nagisa commented Nov 23, 2015

This comment has been minimized.

michaelwoerister commented Nov 24, 2015

This comment has been minimized.

brson commented Nov 24, 2015

nagisa reviewed Nov 24, 2015 View changes

This comment has been minimized.

nagisa reviewed Nov 24, 2015 View changes

This comment has been minimized.

This comment has been minimized.

nagisa reviewed Nov 24, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Nov 25, 2015

This comment has been minimized.

nagisa commented Nov 25, 2015

This comment has been minimized.

brson commented Dec 1, 2015

brson referenced this pull request Dec 1, 2015

This comment has been minimized.

brson commented Dec 1, 2015

brson added T-compiler I-nominated labels Dec 1, 2015

This comment has been minimized.

nagisa commented Dec 1, 2015

This comment has been minimized.

brson commented Dec 1, 2015

This comment has been minimized.

brson commented Dec 1, 2015

brson force-pushed the brson:gold branch from 2da4c30 to 4dc0e89 Dec 1, 2015

This comment has been minimized.

xitep commented Dec 31, 2015

This comment has been minimized.

eddyb commented Dec 31, 2015

This comment has been minimized.

xitep commented Dec 31, 2015

This comment has been minimized.

nagisa commented Jan 1, 2016

niconii referenced this pull request Jan 1, 2016

This comment has been minimized.

birkenfeld commented Jan 5, 2016

This comment has been minimized.

eddyb commented Jan 5, 2016

This comment has been minimized.

birkenfeld commented Jan 5, 2016

This comment has been minimized.

eddyb commented Jan 5, 2016

This comment has been minimized.

brson force-pushed the brson:gold branch from `e6b9c48` to `b5cc5d7` Nov 21, 2015

nagisa reviewed Nov 21, 2015

View changes

nagisa reviewed Nov 21, 2015

View changes

nagisa reviewed Nov 23, 2015

View changes

nagisa reviewed Nov 24, 2015

View changes

nagisa reviewed Nov 24, 2015

View changes

nagisa reviewed Nov 24, 2015

View changes

brson force-pushed the brson:gold branch from `2da4c30` to `4dc0e89` Dec 1, 2015