Error "cannot find -lunwind" when compiling static mips-unknown-linux-musl

[wcampbell0x2a]

I am using https://github.com/richfelker/musl-cross-make to generate a mips-linux-musl compiler to build mips-unknown-linux-musl, as it's tier 3 and needs -Zbuild-std.

First off, I needed to add a couple of -L to the config, in order to not have Could not find crt0 problems. I don't know if this is a rustc bug, or a cross-compiler bug.

However, the bug that I can't figure out, is that for static binaries(and/or panic=abort) it still needs the -lunwind library. Nevertheless, It seems like this should be something it finds from the ~/.rustup/toolchains/, but nothing I try helps rustc find this library?

I tried this code:

src/main.rs:

fn main() {
    println!("test");
}

Cargo.toml:

[profile.release]
strip = true
codegen-units = 1
lto = true
panic = "abort"
opt-level = "z"

.cargo/config:

[build]
rustflags = [
	"-C", "target-feature=+crt-static",
	"-L", "/home/wcampbell/projects/wcampbell/musl-cross-make/output/mips-linux-musl/lib",
	"-L", "/home/wcampbell/projects/wcampbell/musl-cross-make/output/lib/gcc/mips-linux-musl/9.4.0/",
]

[target.mips-unknown-linux-musl]
linker = "/home/wcampbell/projects/wcampbell/musl-cross-make/output/bin/mips-linux-musl-gcc"

[unstable]
build-std = ["core", "compiler_builtins", "alloc", "std", "panic_abort"]

I expected to see this happen: Statically compiled mips-musl binary with -Zbuild-std

Instead, this happened:

$ cargo build --bin mempeek --locked --release --target mips-unknown-linux-musl
error: linking with `/home/wcampbell/projects/wcampbell/musl-cross-make/output/bin/mips-linux-musl-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/wcampbell/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/wcampbell/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/home/wcampbell/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl" VSLANG="1033" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/bin/mips-linux-musl-gcc" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/mips-linux-musl/lib/crt1.o" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/mips-linux-musl/lib/crti.o" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/lib/gcc/mips-linux-musl/9.4.0/crtbegin.o" "/tmp/rustc2DONcX/symbols.o" "/home/wcampbell/projects/wcampbell/mempeek/target/mips-unknown-linux-musl/release/deps/mempeek-840512eed390a4d1.mempeek.eea5b45178c2269e-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/home/wcampbell/projects/wcampbell/mempeek/target/mips-unknown-linux-musl/release/deps" "-L" "/home/wcampbell/projects/wcampbell/mempeek/target/release/deps" "-L" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/mips-linux-musl/lib" "-L" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/lib/gcc/mips-linux-musl/9.4.0/" "-L" "/home/wcampbell/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib" "-Wl,-Bstatic" "-lunwind" "-lc" "/home/wcampbell/projects/wcampbell/mempeek/target/mips-unknown-linux-musl/release/deps/libcompiler_builtins-2264a038ec2e250c.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "/home/wcampbell/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib" "-L" "/home/wcampbell/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/self-contained" "-o" "/home/wcampbell/projects/wcampbell/mempeek/target/mips-unknown-linux-musl/release/deps/mempeek-840512eed390a4d1" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/lib/gcc/mips-linux-musl/9.4.0/crtend.o" "/home/wcampbell/projects/wcampbell/musl-cross-make/output/mips-linux-musl/lib/crtn.o"
  = note: /home/wcampbell/projects/wcampbell/musl-cross-make/output/bin/../lib/gcc/mips-linux-musl/9.4.0/../../../../mips-linux-musl/bin/ld: cannot find -lunwind
          collect2: error: ld returned 1 exit status

Meta

rustc --version --verbose:

rustc --version --verbose
rustc 1.77.0-nightly (5bd5d214e 2024-01-25)
binary: rustc
commit-hash: 5bd5d214effd494f4bafb29b3a7a2f6c2070ca5c
commit-date: 2024-01-25
host: x86_64-unknown-linux-gnu
release: 1.77.0-nightly
LLVM version: 17.0.6

[wcampbell0x2a]

@rustbot label -O-MIPS -O-musl -Zbuild-std

[rustbot]

Error: Label Zbuild-std can only be set by Rust team members

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[wcampbell0x2a]

@rustbot label +O-MIPS +O-musl

[saethlin]

libunwind is still required for aborting panics, because those still print backtraces.

The std feature panic_immediate_abort exists to disable all panic formatting and backtrace printing so that all the code that's used for it can be optimized away. It may or may not work here; it has no tests and I know it has users but I kind of maintain it on the side. I tried writing a run-make test for it and couldn't figure out how run-make tests work.

So I suppose there's a question of what you want to achieve here: Do you want a working panic=abort build, or do you want a build that doesn't depend on libunwind?

[wcampbell0x2a]

I guess I'm less concerned about panic_immediate_abort and panic=abort (although I use these for embedded projects).

I'm more concerned (if that's the right word) about how to resolve the library path errors when cross-compiling and what rustc expects of me when using -Zbuild-std.

I'm not sure about "do you want a build that doesn't depend on libunwind", although I guess with panic=abort the linker would remove the link invocation, as the symbol wouldnt be required (ideally).

[aroberts87]

This issue is timely for me because I'm doing the exact workflow as described above and getting the same results. I am also using https://github.com/richfelker/musl-cross-make to build a musl compiler, with the major difference for me that I'm targeting a different tier 3 platform, s390x-unknown-linux-musl.

The project I'm trying to build is sccache.

.cargo/config.toml:

[target.s390x-unknown-linux-musl]
rustflags = [
	"-C", "target-feature=+crt-static",
	"-C", "panic=abort",
	"-L/opt/s390x-linux-musl/s390x-linux-musl/lib",
	"-L/opt/s390x-linux-musl/lib/gcc/s390x-linux-musl/9.4.0/"
]
linker = "s390x-linux-musl-gcc"
ar = "s390x-linux-musl-gcc-ar"

Cargo.toml:

[profile.release]
codegen-units = 1
lto = true
strip = true
panic = "abort"

Build Command:

export PATH="/opt/s390x-linux-musl/bin/:$PATH"
cargo build -Zbuild-std=std,panic_abort -Zbuild-std-features=panic_immediate_abort --locked --bin sccache --features=openssl/vendored

Observed Error:

  = note: /opt/s390x-linux-musl/bin/../lib/gcc/s390x-linux-musl/9.4.0/../../../../s390x-linux-musl/bin/ld: cannot find -lunwind
          collect2: error: ld returned 1 exit status

rustc --version --verbose:

rustc --version --verbose 
rustc 1.77.0-nightly (595bc6f00 2024-01-05)
binary: rustc
commit-hash: 595bc6f00369475047538fdae1ff8cea692ac385
commit-date: 2024-01-05
host: x86_64-unknown-linux-gnu
release: 1.77.0-nightly
LLVM version: 17.0.6

I'm pretty new to the Rust ecosystem and most of my configuration in .cargo/config.toml is based on solutions to other posts that I found while attempting to find a solution to this problem, but when I found this issue and realized it was filed today I wanted to chime in as a second data point in case this is more than the user error that I thought I was having.

[wcampbell0x2a]

@aroberts87 fyi, ar is deprecated: https://doc.rust-lang.org/cargo/reference/config.html#targettriplear

[wcampbell0x2a]

Another question, for build-std and this issue, is libunwind expected to be in my tool chain, or is this something that rustc doesn't have packaged because it's a tier 3?

I guess with LTO and panic=abort the functions would be removed anyway( should have no call sites), so could it just be an "empty functions providing symbols" library?

[bjorn3]

libc and libunwind for musl targets are packed in the rust-std component for the respective target, which for tier 3 targets doesn't exist at all.

With panic=abort we still need libunwind for generating backtraces (which is what panic_immediate_abort disables among other things) Also when using extern "C-unwind" with panic=abort we have to ensure that unwinding into rust code aborts to avoid UB in that case, which we do using a personality function that calls several libunwind functions.

[wcampbell0x2a]

Is this a problem for every tier 3 target with std support? If not, how do they build libunwind?

[saethlin]

I think this is the file that introduces that -l flag, it introduces it by putting #[link(name = "unwind")] on an empty extern "C" block: https://github.com/rust-lang/rust/blob/c29082fe7dc6e902169cacbae165562a7e4a1fd6/library/unwind/src/lib.rs

The llvm-libunwind feature is toggled by the features of the same name in library/std/Cargo.toml which are in turn toggled by code in src/bootstrap.

I see musl is special-cased in both places. I don't know why.

[wcampbell0x2a]

This might be related to #61367

[wcampbell0x2a]

I guess if I could optionally pick to use my libgcc, I would have the Unwind symbols:

> readelf -s lib/gcc/mips-linux-musl/9.4.0/libgcc.a | grep Unwind
    18: 000034f0   328 FUNC    LOCAL  DEFAULT    1 _Unwind_RaiseExc[...]
    19: 00003638   388 FUNC    LOCAL  DEFAULT    1 _Unwind_ForcedUn[...]
    20: 00003aec     8 FUNC    LOCAL  DEFAULT    1 _Unwind_DebugHook
    38: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_Find_FDE
    42: 000037bc   148 FUNC    GLOBAL HIDDEN     1 _Unwind_GetGR
    43: 00003850    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetCFA
    44: 0000385c   164 FUNC    GLOBAL HIDDEN     1 _Unwind_SetGR
    45: 00003900    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetIP
    46: 0000390c    28 FUNC    GLOBAL HIDDEN     1 _Unwind_GetIPInfo
    47: 00003928     8 FUNC    GLOBAL HIDDEN     1 _Unwind_SetIP
    48: 00003930    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetLangu[...]
    49: 0000393c    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetRegio[...]
    50: 00003948    68 FUNC    GLOBAL HIDDEN     1 _Unwind_FindEncl[...]
    51: 0000398c    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetDataR[...]
    52: 00003998    12 FUNC    GLOBAL HIDDEN     1 _Unwind_GetTextR[...]
    54: 00003af4   696 FUNC    GLOBAL HIDDEN     1 _Unwind_RaiseExc[...]
    55: 00003dac   468 FUNC    GLOBAL HIDDEN     1 _Unwind_ForcedUnwind
    56: 00003f80   504 FUNC    GLOBAL HIDDEN     1 _Unwind_Resume
    57: 00004178   492 FUNC    GLOBAL HIDDEN     1 _Unwind_Resume_o[...]
    58: 00004364    32 FUNC    GLOBAL HIDDEN     1 _Unwind_DeleteEx[...]
    59: 00004384   376 FUNC    GLOBAL HIDDEN     1 _Unwind_Backtrace
    19: 00000f58  1452 FUNC    LOCAL  DEFAULT    1 _Unwind_IterateP[...]
    62: 000022d0   692 FUNC    GLOBAL HIDDEN     1 _Unwind_Find_FDE
    23: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_GetTextR[...]
    24: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_GetRegio[...]
    26: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_GetDataR[...]
    28: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_GetLangu[...]
    29: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_GetIPInfo
    30: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_SetGR
    31: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _Unwind_SetIP

[Ragnt]

I'm having these same issues. I'm racking my brain trying to figure out HOW rust is basically useless on mips with either uclibc (no method named si_status found for struct siginfo_t) or musl (unwind is unavailable, even in buildroot, for mips-musl).

[saethlin]

I'm racking my brain trying to figure out HOW rust is basically useless on mips

There are a few problems:

LLVM does not have a reliable MIPS backend, so we cannot have MIPS CI, or all work on the compiler and library grinds to a halt whenever a new LLVM MIPS bug is discovered. This happened enough for MIPS bugs to become a meme. We didn't just demote the targets immediately, we demoted them after the amount of pain was so high that everyone we relieved to see it happen. rust-lang/compiler-team#648

Most of the MIPS targets do not even have a target maintainer. Those, if they exist, are documented here: https://doc.rust-lang.org/nightly/rustc/platform-support.html. There are some other -uclibc targets which do actually have a target maintainer, perhaps you want to @ one of those on a new issue.

[Ragnt]

To be clear, my frustration isn't without understanding. I've been reading about the lack of LLVM support for MIPS and know there is only so much the compiler team can do.

[wcampbell0x2a]

I forgot to copy paste a solution I found in cross:

A solution to this is just to create a symbolic link the libgcc.a into libuwind.a:

ln -s musl-cross-make/output/lib/gcc/mips-linux-musl/11.2.0/libgcc.a /musl-cross-make/output/lib/gcc/mips-linux-musl/11.2.0/libunwind.a

A better solution could be found within rustc, but that will take more effort and investigation.

[jdmitrovic-syrmia]

I don't get an error with building or running the Hello World testcase.

I compiled Rust project with the following config.toml:

change-id = 125535
profile = 'compiler'

[build]
target = ["x86_64-unknown-linux-gnu", "mips-unknown-linux-musl"]
configure-args = []

[llvm]
assertions = true
targets = "Mips;X86"
download-ci-llvm = false

[target.x86_64-unknown-linux-gnu]
llvm-config = "/home/syrmia/llvm-project/build-rust/bin/llvm-config"

[target.mips-unknown-linux-musl]
llvm-config = "/home/syrmia/llvm-project/build-rust/bin/llvm-config"
linker = "/home/syrmia/musl-cross-make/output/bin/mips-linux-musl-gcc"
runner = "qemu-mips"
llvm-libunwind = 'in-tree'
musl-root = "/home/syrmia/musl-cross-make/output/"
musl-libdir = "/home/syrmia/musl-cross-make/output/mips-linux-musl/lib"
crt-static = true

I got to this config by consulting the config.example.toml. This is the config.toml of the Hello World test:

[build]
rustflags = [
	"-C", "target-feature=+crt-static"
]

[target.mips-unknown-linux-musl]
linker = "/home/syrmia/musl-cross-make/output/bin/mips-linux-musl-gcc"
runner = "qemu-mips"

[unstable]
build-std = ["core", "compiler_builtins", "alloc", "std", "panic_abort"]

As you can see, nothing much was changed, I just found additional values that must be set in the config.toml of the Rust project. Cargo.toml is omitted, as it it exactly the same as in the opening message. Although, I have used a more recent LLVM (llvm/llvm-project@f6ace2bc15bf).