Link against LLVM libunwind (statically) instead of libgcc_s (dynamically) on Linux targets

[vadorovsky]

Follow up for the issue rust-lang/rustup#2213, which is probably more accurate to be discussed and potentially fixed in rustc rather than rustup.

Problem

The default Rust builds (like the ones shipped in rustup) are dynamically linked against libgcc_s, which forces users to install GCC-related packages even on distributions which don't ship the GNU userland by default (Alpine, Gentoo with llvm-musl profile etc.). Without libgcc installed, users face the following error:

Error loading shared library libgcc_s.so.1: No such file or directory (needed by /home/vadorovsky/.rustup/toolchains/stable-x86_64-unknown-linux-musl/bin/cargo)

Potential solution

LLVM libunwind can be just used as a drop-in replacement for libgcc_s without causing any issues. In fact, I did the following as a workaround on my Gentoo (llvm-musl profile) installation, which doesn't have GCC nor glibc installed:

ln -s /usr/lib/libunwind.so.1 /usr/lib/libgcc_s.so.1
ln -s /usr/lib/libunwind.so /usr/lib/libgcc_s.so

and it works. I'm daily driving Rust on that machine without any issues.

So to make it easier for everyone, we could probably just link libunwind statically for all Linux targets (or at least the musl ones) by default, since the LLVM backend is the default one anyway. One thing to consider could be an opt-in way to use libgcc_s (which would make a lot of sense when using GCC backend).

[Noratrieb]

Systems without a GNU userland shouldn't use the *-linux-unknown-gnu toolchains, which explicitly require a GNU userland. Your supposed to use *-linux-unknown-musl.

[vadorovsky]

The same problem happens on *-linux-unknown-musl toolchains - it still requires libgcc_s. The error I pasted is from a musl toolchain installed with rustup.

[Noratrieb]

Ah, I see. The host tools for x86_64-unknown-linux musl are built with -Ctarget-feature=-crt-static, which allows dylibs and links to libc dynamically. But this does indeed add a dependency on libgcc_s.so.1, libc.so.6 and ld-linux-x86-64.so.2. I know that alpine patches the rustc they package anyways, because our toolchains are basically unusable on Alpine if you have any C deps, because they default to statically linking libc on musl (a mistake with an accepted proposal to fix, but no one having put in the work of actually fixing). It would be useful to know how Alpine mitigates this problem with libgcc_s for their builds.
Also cc @bjorn3 for linking questions :)

[bjorn3]

So to make it easier for everyone, we could probably just link libunwind statically for all Linux targets (or at least the musl ones) by default

That will break linking against C++ code. There can only be a single unwinder runtime in the entire program if you want to be able to unwind across dynamic library boundaries, which we support using extern "C-unwind". For this reason we link against the "system" unwinder, which on most linux targets is libgcc_s.so. For musl we do link against libunwind when producing a statically linked executable though:

rust/library/unwind/src/lib.rs

Line 56 in e51e98d

#[link(name = "unwind", kind = "static", modifiers = "-bundle", cfg(target_feature = "crt-static"))]

And when compiling the standard library you can enable the system-llvm-libunwind feature to link against libunwind.so by default.

[12101111]

LLVM libunwind is not a drop-in replacement for libgcc_s.so.1. If you want a non GCC version of libgcc_s, please use llvm-libgcc. I have an ebuild for gentoo user

They are different because libgcc_s (and llvm-libgcc) provide more symbols than libunwind.

llvm-readelf --dt ~/.rustup/toolchains/nightly-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/bin/rust-lld  | grep GCC | grep "__"
    44: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __floatunditf@GCC_4.2.0
    79: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __divtf3@GCC_3.0
   109: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __clear_cache@GCC_3.0
   160: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __extendsftf2@GCC_3.0
   177: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __eqtf2@GCC_3.0
   194: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __trunctfdf2@GCC_3.0
   210: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __multf3@GCC_3.0
   234: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __letf2@GCC_3.0
   246: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __floatsitf@GCC_3.0

llvm-readelf --dt ~/.rustup/toolchains/nightly-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/bin/rust-lld | grep GCC | grep "__"
    51: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __floatunditf@GCC_4.2.0
    92: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __divtf3@GCC_3.0
   113: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __clear_cache@GCC_3.0
   122: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __unordtf2@GCC_4.5.0
   162: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __extendsftf2@GCC_3.0
   197: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __trunctfdf2@GCC_3.0
   205: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __getf2@GCC_3.0
   212: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __multf3@GCC_3.0
   228: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __letf2@GCC_3.0
   233: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __floatsitf@GCC_3.0
   237: 0000000000000000     0 FUNC    GLOBAL DEFAULT   UND __gttf2@GCC_3.0

As the doc of llvm-libgcc say, libgcc_s.so is a required dependency in the Linux standard base, so we should assume it's installed.

And as @bjorn3 say, rustc will load both libstd-*.so and librustc_driver-*.so, and more dynamic libraries (e.g. compiled proc macro, and libLLVM-17-rust-*.so for some target).

If libstd-*.so has a statically linked libunwind, libLLVM and librustc_driver need to link to that libstd-*.so and do not statically link another libunwind. But it's not possible. when rustc is bootstraping, libLLVM, librustc_driver and rustc are compiled first and then this new rustc compile libstd.

[bjorn3]

But it's not possible. when rustc is bootstraping, libLLVM, librustc_driver and rustc are compiled first and then this new rustc compile libstd.

The libstd.so that rustc itself is linked against is compiled while building the previous bootstrap stage, so that is not the issue. What is an issue however is that libLLVM doesn't link against libstd and much more importantly that proc macros (which rustc dlopen's) don't link against libstd.so either but statically link it.

LLVM libunwind is not a drop-in replacement for libgcc_s.so.1.

In the case of pure rust code we only use libgcc_s for the unwind runtime. The compiler intrinsics are defined by the compiler_builtins crate (unless libgcc or compiler-rt is statically linked and takes precedence). For the C code that is shipped with rustc (LLVM, lld, ...) it is indeed a potential problem though.

[vadorovsky]

@12101111 amazing, thanks for clarification about llvm-libgcc and for the ebuild! I think it solves problem for me.

Also, @bjorn3, thanks for explanation why linking libunwind statically wouldn't work. I'm closing this issue then.