Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linking with LLD #39915

Open
bstrie opened this issue Feb 17, 2017 · 103 comments
Open

Linking with LLD #39915

bstrie opened this issue Feb 17, 2017 · 103 comments

Comments

@bstrie
Copy link
Contributor

@bstrie bstrie commented Feb 17, 2017

LLVM 4.0 is shipping with LLD enabled, though AFAIK it is not yet production-ready on all platforms. I believe we've got an LLVM upgrade planned soon to resolve AVR/emscripten problems anyway, so now's the time to start determining what we might need to do to support it, how it impacts compiler performance/binary size/runtime performance compared to our usual linkers, and what platforms on which we might want to enable it by default.

Current status (2020-04-24) summarized at #39915 (comment)

Possible problems:

  • LLD has internal parallelism, and is not documented to have jobserver support, which can lead to poor contention resolution in practice
@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Feb 17, 2017

See also a PoC in #36120.

@retep998
Copy link
Member

@retep998 retep998 commented Feb 18, 2017

LLD may be a very good candidate for MinGW targets, because we currently bundle a linker with them anyway and MinGW's linker has a variety of issues ranging from a lack of ASLR to no bigobj support. If we can somehow also bring along the necessary mingw libraries when cross compiling and not just native targeting (which rustup's mingw package is currently limited to), then that would enable rust cross compilation from linux out of the box, which would be a huge improvement over the existing situation where people get MinGW from their distro and then run into issues because distros almost always use an incompatible MinGW.

LLD is not a good candidate for natively targeting MSVC targets, due to several reasons, the primary reason being the lack of debuginfo support. Cross compiling to MSVC targets requires libraries that can't be redistributed so we can't support that out of the box anyway.

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Feb 19, 2017

The tracking issue for upgrading to LLVM 4.0 is #37609 .

@binarycrusader
Copy link
Contributor

@binarycrusader binarycrusader commented Feb 20, 2017

For the record lld is definitely not ready for Solaris targets. But on Solaris, there's no reason that I'm aware of to use lld instead of the native ld instead. We've already been looking at what it would take to have rust use Solaris ld on Solaris instead of using gcc for linking.

@whitequark
Copy link
Member

@whitequark whitequark commented Feb 20, 2017

@binarycrusader One reason to use lld is when building for Solaris, not on Solaris.

@japaric
Copy link
Member

@japaric japaric commented Feb 21, 2017

PR #40018 adds a -Z linker-flavor flag to rustc to make it possible to use LLD as a linker. That PR doesn't embed LLD in rustc but allows out of tree experimentation with it.

@binarycrusader ^ that may help with your experiment of directly using Solaris' ld instead of gcc.

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Apr 25, 2017

We now appear to be running on LLVM 4.0. @japaric , does this mean that the linker-flavor flag can now be easily used to compare and contrast LLD with the system linker?

@japaric
Copy link
Member

@japaric japaric commented Apr 29, 2017

@bstrie #40018 landed a few weeks ago. Since that landed one has been able to use -Z linker-flavor=ld -C linker=ld.lld to use an external LLD binary as a linker. Note that, unlike gcc, LLD doesn't know where the system libraries are so you'll have to pass the library search path to the linker using -C link-args='-L ...' if you are linking to any system library.

What LLVM 4.0 helps with is merging LLD into rustc. With that change we wouldn't require an external linker in some scenarios like linking MUSL binaries or bare metal programs. I say some scenarios because most targets require linking to system libraries where you will run into the library search path problem I mentioned above. For those targets, LLD won't work out of the box. It's not clear how and where to solve that problem and without a solution for that we can't switch to LLD for the most important (tier 1) targets, which the reduces the appeal of embedding LLD into rustc in the first place.

@whitequark
Copy link
Member

@whitequark whitequark commented Apr 29, 2017

@japaric What are the arguments against embedding (sysroot-relative library) search paths, as well as things like -lc -lpthread crt0.o, directly into rustc? After all, some component of the toolchain has to embed them as we don't have any standard for the platforms to follow, and binutils is not a good golden source of this knowledge.

The only drawback I can think of is the situation where the same triple would have different search paths on different flavors of systems (which will likely be exclusive to Linux/glibc triples, and especially bad on platforms with multilib). In that case, I believe clang snoops the OS name and hardcodes OS-specific conventions, which seems bad but probably unavoidable if one wants to distribute a single binary that runs on any Linux (and doesn't need the system linker).

@iainnicol
Copy link

@iainnicol iainnicol commented Jun 16, 2017

@retep998 I had a brief look at lld a few months ago. I wasn't able to cross compile (cross link?) an .exe from Linux. It seemed like lld only supports platform native formats.

Hopefully I'm mistaken.

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Sep 8, 2017

Tagging this as a performance bug, since according to LLD's benchmarks it seems to outperform GNU ld by a factor of ten, and linking performance is a large component of compiler speed currently.

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Sep 8, 2017

Er, forgot to link the benchmarks: https://lld.llvm.org/#performance

(Relevant today since LLVM 5.0 has just been released.)

@tpimh
Copy link

@tpimh tpimh commented Sep 9, 2017

Linking with LLD is much faster than bfd or gold, but I doubt that using it will significantly improve overall performance. Still I think this issue is important and should be a priority.

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented Sep 9, 2017

@tpimh I'm actually not entirely sure whether the I-slow tag is supposed to represent runtime performance bugs or compiletime performance bugs, I was intending it as the latter. And IME when I look at time-passes output linking is usually in the top three longest phases, significantly longer than most, so even cutting linking time in half would probably be a huge win (especially for big things like Servo and rustc).

@jonas-schievink
Copy link
Member

@jonas-schievink jonas-schievink commented Sep 9, 2017

@bstrie I-slow is for bad runtime performance, I-compiletime is for compiler perf last time I checked

@iainnicol
Copy link

@iainnicol iainnicol commented Sep 17, 2017

Good news for anybody interested in the obscure topic of cross linking from Linux to Windows. I said earlier it wasn't possible with lld, but that's only true for lld's ld flavor. It's possible for lld's link.exe flavour (lld-link).

Specifically for Rust we can do this today with a couple code changes.

  1. We need to compile a very small subset of mingw-w64's CRT into .o object files. Namely some thread local storage init. We also need chkstk.

  2. lld does not like MinGW's usual import libraries. Instead we need to build the .def files into .lib files ourselves, using lld-link or llvm-dlltool

  3. Modify lld to treat IMPORT_NAME_NOPREFIX like
    IMPORT_NAME_UNDECORATE, because even with step 2 the .libs aren't perfect

  4. Modify Rust's seh.rs to replace TYPE_INFO_VTABLE with ptr::null(). Required because the symbol ??_7type_info@@6B@ isn't defined in MinGW. Then build and install Rust.

  5. Use .cargo/config to specify a custom wrapper script as the linker.

  6. Our wrapper linker script should invoke lld-link mostly using the parameters it is passed. However we must make a few tweaks:

    a) Fix filename casing e.g. change AdvAPI32.Lib to advapi32.lib

    b) Modify the .def file Rust generates to prefix symbols with an extra underscore

    c) Override the entry point (/entry). Required probably due to a name mangling issue.

    d) Append the mingw-crt object files you compiled in step 1

  7. Build your Rust project using xargo --target=i686-pc-windows-msvc

Doing the above steps allows me to cross compile Rust code. I can even panic and catch panics using Rust's SEH-based unwinding.

@retep998
Copy link
Member

@retep998 retep998 commented Sep 17, 2017

@iainnicol You're mixing the msvc target with MinGW bits, which is why you have to do all those weird modifications. If you just copy over the libraries from an existing VC++ installation then you can use lld-link normally without all those modifications or any MinGW bits.

@whitequark
Copy link
Member

@whitequark whitequark commented Sep 17, 2017

But I don't want to use an existing VC++ installation. There isn't even any way to get one without spending something like eight hours downloading and installing junk, much less to redistribute.

@rpjohnst
Copy link
Contributor

@rpjohnst rpjohnst commented Sep 17, 2017

The standalone build tools are much lighter-weight, unless that's already what you're referring to, in which case perhaps we ought to put some work into improving or recreating what MinGW did so it's actually compatible with MSVC.

@whitequark
Copy link
Member

@whitequark whitequark commented Sep 17, 2017

The standalone build tools are much lighter-weight

I haven't realized Microsoft distributes those. Could you link to them? Is there any reasonable way to extract the installation archive without actually running it i.e. is it an msi or something similar?

@rpjohnst
Copy link
Contributor

@rpjohnst rpjohnst commented Sep 17, 2017

Here they are: http://landinghub.visualstudio.com/visual-cpp-build-tools

Both the 2015 and 2017 versions are exes, but you may be able to convince the 2017 exe to give you what you want via this: https://docs.microsoft.com/en-us/visualstudio/install/install-vs-inconsistent-quality-network

@retep998
Copy link
Member

@retep998 retep998 commented Sep 17, 2017

If we really want to do this right for Windows, we'd first of all need #30027 to eliminate the need for the Windows SDK or MinGW's import libraries. Then all we'd have left is to replace the CRT bits with our own pure Rust versions (math/memory functions, entry point, a few other runtime bits Rust needs) and we'd have a fully self-contained Rust toolchain that can create Windows binaries! The downside of this is that you wouldn't be able to statically link C/C++ code because that relies very heavily on linking in the appropriate CRT from either MinGW or VC++. Of course the whole point of Rust is to rewrite everything in Rust, so this isn't really much of an issue.

@Gankra
Copy link
Contributor

@Gankra Gankra commented Apr 23, 2020

This bug is a bit of a mess, so here's my best shot at a quick summary of the current situation, for folks who want to move this forward.

What is lld

A linker that's part of the llvm project, which is desirable for two reasons:

  • it's very friendly to cross compilation (hence its emphasis for embedded targets)
  • it's very fast (often runs in half the time as Gold -- linking can take several minutes for big projects (rustc, servo, etc.) and linking can be a huge % of the compile with incremental builds, so halving this runtime is a Big Deal.)

Things Rust does with lld today

  • Rust currently ships its own copy of lld on most platforms as a binary it calls rust-lld
  • rust-lld is used by default on many bare-metal targets
  • rust-lld is used by default for wasm
  • (?) you can explicitly request rust-lld be used by using "-C linker-flavor" (fuzzy on what exactly this does on non-bare-metal platforms, see below)

Problems for using rust-lld in more places (i.e. desktop linux/mac/windows)

  • The macOS (Mach-O) backend for lld is broken and abandoned
    • a rewrite from scratch has begun, but it's early days
  • On linux/unix platforms, you aren't supposed to directly invoke ld/lld. You're supposed to invoke the linker through your system c compiler (i.e. gcc), whose responsibility it is to discover system symbols like crt1.o and provide them to ld. This means we can't "just" use rust-lld; we must feed it into gcc/clang/whatever. (we don't want to implement this system symbol logic ourselves)
    • In general, you can't provide the linker as a path, you must inject it into the C compiler's search path as "ld"
    • Alternatively, you can do the same thing but inject it as "ld.lld", and pass "-fuse-ld=lld"
      • This may be important, apparently lld does clang-style binary name detection of if it's being executed as "ld" or "ld.lld" (needs investigation)
      • Unfortunately -fuse-ld=lld is only part of GCC 9, so we may require feature/version detection to use it (clang has had it for a long time)
  • windows-msvc is apparently in good shape, and seems to have some limited support for using rust-lld in the backend, but I'm unclear on what needs to be done here.
  • windows-mingw seems to be in roughly the same place as linux/unix, except you're liable to get an ancient GCC, and things are a bit wonky because pseudo-windows-linux isn't exactly a well-tested configuration?

Also just in general, lld is newer, it's not the default for most OSes, random compat bugs are almost certainly going to come up if we start using this in more places.

I have filed two metabugs for focused efforts on using (rust-)lld by default on two platforms:

  • #71515 - x64 Ubuntu 20.04 LTS (and more broadly all the x64 ELF platforms)
  • #71520 - x64 msvc windows

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Apr 24, 2020

windows-msvc is apparently in good shape, and seems to have some limited support for using rust-lld in the backend, but I'm unclear on what needs to be done here.

LLD + windows-msvc is in pretty good state, I'm currently using this setup for rustc development.

All the necessary support for lld-link is in place in rustc backend, but there are bugs like #68647.

@Hello71
Copy link

@Hello71 Hello71 commented Apr 24, 2020

  • This may be important, apparently lld does clang-style binary name detection of if it's being executed as "ld" or "ld.lld" (needs investigation)

it does, but ld and ld.lld are the same mode: https://github.com/rust-lang/llvm-project/blob/rustc/10.0-2020-02-05/lld/tools/lld/lld.cpp (same since 8.0-2019-01-16)

  • Unfortunately -fuse-ld=lld is only part of GCC 9, so we may require feature/version detection to use it (clang has had it for a long time)

since ld.lld is the same as ld, and according to https://patches-gcc.linaro.org/patch/11148/ the only change with -fuse-ld=lld is to run ld.lld instead of ld, if we use PATH injection, it should be fine. I think locking this out to gcc 9+ isn't good though: debian stable only has 8.3, and bullseye probably won't be released until 2021.

  • windows-mingw seems to be in roughly the same place as linux/unix, except you're liable to get an ancient GCC, and things are a bit wonky because pseudo-windows-linux isn't exactly a well-tested configuration?

mingw-w64 6.0.0, released 2018-09-16, has gcc 8.3.0, which doesn't have -fuse-ld=lld, but is still reasonably new. mingw-w64 7.0.0, released 2019-11-11, has gcc 9.3.0, which does have -fuse-ld=lld. Debian buster (stable) has 6.0.0, bullseye (testing) has 7.0.0. Debian stretch (oldstable) only has 5.0.1 with gcc 6.3.0, but I think it would be reasonable to require latest debian stable for lld support if there are significant issues with gcc 6.3.

@Gankra
Copy link
Contributor

@Gankra Gankra commented Apr 24, 2020

I have filed two metabugs for focused efforts on using (rust-)lld by default on two platforms:

  • #71515 - x64 Ubuntu 20.04 LTS (and more broadly all the x64 ELF platforms)
  • #71520 - x64 msvc windows

@davidbarsky
Copy link

@davidbarsky davidbarsky commented Oct 6, 2020

About the macOS (Mach-O) port of lld: it seems to work, or at the very least, be in significantly better shape since #39915 (comment) was written!

Using this LLVM commit, I built lld and set it as a project-specific linker for tokio-rs/tracing. I then built tracing against nightly-x86_64-apple-darwin and ran all tests successfully. I'm especially happy about about the (debug) build times:

  • With ld, a clean cargo build took 35 seconds.
  • With lld, a clean cargo build took 20 seconds.

Note that:

  • these performance numbers came from a recent MacBook Pro with 32GB of RAM and an 8-Core i9, and
  • there are a few outstanding issues for lld and Mach-O.

@trevyn
Copy link
Contributor

@trevyn trevyn commented Oct 7, 2020

@davidbarsky Cool! Out of curiosity, how does that performance compare with zld? ( https://github.com/michaeleisel/zld )

Also, did you account for thermals? MBPs go into thermal throttling very quickly, especially with the default fan speed profile. Just waiting until the bottom of the machine near the hinge is cool to the touch before doing a run should help consistency.

@advancedwebdeveloper
Copy link

@advancedwebdeveloper advancedwebdeveloper commented Nov 10, 2020

I just such a bug, on Ubuntu 16 i686

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Nov 20, 2020

Potential problem for using rust-lld on Arm64 / aarch64 / Apple Silicon macOS

All executables are required to be code-signed on the new Arm64 macOS machines. By default, the system linker does that for you:

% cargo new hello-world
     Created binary (application) `hello-world` package

% cd hello-world

% cargo build
   Compiling hello-world v0.1.0 (/private/tmp/hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 5.03s

% codesign -dvvv target/debug/hello-world
Executable=/private/tmp/hello-world/target/debug/hello-world
Identifier=hello_world
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=3140 flags=0x20002(adhoc,linker-signed) hashes=95+0 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=2fd59b9215be8cfaa04c619323546803bd8ee109
CandidateCDHashFull sha256=2fd59b9215be8cfaa04c619323546803bd8ee1091eab723ddedc127abacd9e6b
Hash choices=sha256
CMSDigest=2fd59b9215be8cfaa04c619323546803bd8ee1091eab723ddedc127abacd9e6b
CMSDigestType=2
CDHash=2fd59b9215be8cfaa04c619323546803bd8ee109
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

If someone tries to get lld working on macOS, we should verify that the code signing still happens for free.

@tschuett
Copy link

@tschuett tschuett commented Feb 18, 2021

[lld-macho] add code signature for native arm64 macOS
https://reviews.llvm.org/D96164

@bstrie
Copy link
Contributor Author

@bstrie bstrie commented May 22, 2021

Since a lot of people are invested in this, I should note that enabling LLD by default on Linux (#71515) and Windows (#71520) both seem relatively close to completion, but they each need a champion willing to put in the last bit of work to push them over the finish line.

@dvc94ch
Copy link
Contributor

@dvc94ch dvc94ch commented Dec 21, 2021

so was wondering if it was possible to cross link for macos using lld and the macos sdk. I get this error:

error: linking with `rust-lld` failed: exit status: 1
  |
  = note: "rust-lld" "-flavor" "darwin" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.1k1r2peqlms382uw.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.1suwbij9vuhemtp7.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.28of16y6dy4ftj2l.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.2cqhpy3ctqkp1udg.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.2mgjts3992b5y0bz.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.4438dslzay5e4h5g.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.49lwkudf6wetod32.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.4o3qjqrb84o9o4w5.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.4o5cczgq0rwzc52s.rcgu.o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755.4kvw9qe301n2oclv.rcgu.o" "-L" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps" "-L" "/home/dvc/cloudpeer/cross-hw/target/debug/deps" "-L" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/libcross_hw-69c8e78ff0eca6e4.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libstd-dd8a82589e0cba34.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-8c04c8bd0d1a8900.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libobject-c6a4ae86ed2c40d0.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libmemchr-f9ab4d1b2e38b05e.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-002c7b677ad6c512.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libgimli-a3f3d9f86c37973f.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-8b14bcf2354140fd.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-d6f2fd91ec8bbbcc.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-24c80e37fb5b15c5.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-edb9b11fa36b4795.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libunwind-769780536fb7ef9b.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-d37c37a3a3ac2b0c.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/liblibc-c1bdc4c1f89760ef.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/liballoc-750380e9c94de9ce.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1108e622f5a15c3d.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcore-43af7053e70b1eed.rlib" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-3a81ebf6a3abbdee.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/home/dvc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/home/dvc/cloudpeer/cross-hw/target/x86_64-apple-darwin/debug/deps/cross_hw-70d0fd826bb4c755" "-dead_strip"
  = note: rust-lld: error: must specify -arch

my ~/.cargo/config looks like this:

[target.x86_64-pc-windows-msvc]
rustflags = [
    "-C", "target-feature=+crt-static",
    "-Lnative=/home/dvc/xwin/crt/lib/x86_64",
    "-Lnative=/home/dvc/xwin/sdk/lib/um/x86_64",
    "-Lnative=/home/dvc/xwin/sdk/lib/ucrt/x86_64",
]
linker = "rust-lld"

[target.x86_64-apple-darwin]
linker = "rust-lld"

@bjorn3
Copy link
Contributor

@bjorn3 bjorn3 commented Dec 21, 2021

Directly using -Clinker=rust-lld only works for wasm and windows. On all unix platforms you need gcc or clang to invoke the linker for you as they pass a lot of arguments to the linker necessary for a successful compilation like the path to system libraries or in this case the system architecture to link for. The way to make gcc and clang invoke lld would be to use -fuse-ld=lld as argument (-Clink-arg=-fuse-ld=lld when using rustc) if lld is in your path. To use the lld version shipped with rustc you can use -Zgcc-ld=lld.

@dvc94ch
Copy link
Contributor

@dvc94ch dvc94ch commented Dec 21, 2021

building with -Zgcc-ld=lld doesn't seem to work either:

 = note: cc: error: unrecognized command-line option '-arch'
          cc: error: unrecognized command-line option '-fuse-ld=/home/dvc/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld64'

got a bit further with the previous approach:

[target.x86_64-apple-darwin]
rustflags = [
    "-C", "link-arg=-arch",
    "-C", "link-arg=x86_64",
    "-C", "link-arg=-platform_version",
    "-C", "link-arg=macos",
    "-C", "link-arg=10.10",
    "-C", "link-arg=10.10",
    "-L/home/dvc/MacOSX10.10.sdk/usr/lib",
]
linker = "rust-lld"

but now I get errors like this:

rust-lld: error: unable to locate re-export with install name /usr/lib/system/libcache.dylib

@dvc94ch
Copy link
Contributor

@dvc94ch dvc94ch commented Dec 22, 2021

so got it to compile like this:

[target.x86_64-apple-darwin]
rustflags = [
    "-C", "link-arg=-target",
    "-C", "link-arg=x86_64-apple-darwin",
    "-C", "link-arg=-fuse-ld=lld",
    "-C", "link-arg=-mmacosx-version-min=10.10",
    "-C", "link-arg=-v",
    "-C", "link-arg=-isystem",
    "-C", "link-arg=/home/dvc/MacOSX10.10.sdk",
    "-L", "/home/dvc/MacOSX10.10.sdk/usr/lib",
    "-L", "/home/dvc/MacOSX10.10.sdk/usr/lib/system",
    "-C", "link-arg=-F",
    "-C", "link-arg=/home/dvc/MacOSX10.10.sdk/System/Library/Frameworks",
]
linker = "clang"

get some warnings though: nope was due to some env variables set during experimenting

and I have no idea if the binary actually works as I don't have a mac to test it on yet.

@nsabovic
Copy link

@nsabovic nsabovic commented Dec 24, 2021

I compiled a relatively large project with ~450 dependencies that's a GRPC service that does raytracing, so a bunch of different kinds of dependencies. I have an M1 Mac so my config was:

[target.aarch64-apple-darwin]
rustflags = [
    "-C", "link-arg=-target",
    "-C", "link-arg=aarch64-apple-darwin",
    "-C", "link-arg=-fuse-ld=lld",
    "-C", "link-arg=-mmacosx-version-min=12.1",
    "-C", "link-arg=-v",
    "-C", "link-arg=-isystem",
    "-C", "link-arg=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk",
    "-L", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib",
    "-L", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/system",
    "-C", "link-arg=-F",
    "-C", "link-arg=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks",
]
linker = "clang"

For this to work I used Xcode.app and LLVM installed with Homebrew (brew install llvm).

The binaries worked just fine, the test suite passed. Looking at the file sizes, they were comparable if a tiny bit larger with lld.

-rwxr-xr-x  1 redacted  staff  18324905 Dec 24 11:04 ./std/debug/test
-rwxr-xr-x  1 redacted  staff  17645152 Dec 24 11:01 ./lld/debug/test
-rwxr-xr-x  1 redacted  staff   2852265 Dec 24 11:05 ./std/release/test
-rwxr-xr-x  1 redacted  staff   2872608 Dec 24 11:03 ./lld/release/test

Compile times were also comparable if a tiny bit faster with lld.

std: cargo build  165.67s user 14.97s system 542% cpu 33.287 total
lld: cargo build  160.69s user 15.24s system 535% cpu 32.852 total

std: cargo build --release  279.84s user 15.70s system 440% cpu 1:07.08 total
lld: cargo build --release  276.73s user 16.06s system 439% cpu 1:06.59 total

@dvc94ch
Copy link
Contributor

@dvc94ch dvc94ch commented Dec 24, 2021

Working on a cross compile toolchain for all platforms using distro supplied clang/lld. Copies the relevant parts from a GitHub runner and publishes a release. Still tweaking the .cargo/config for the example app to build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet