Builds on *-unknown-linux-musl that link to C libaries produce broken binaries

[WhyNotHugo]

Problem

I'm trying to build the following minimal example:

fn main() {
    _ = xkbcommon::xkb::Context::new(0);
    println!("Hello, world!");
}

I'm buildling on aarch64-unknown-linux-musl, and my target is aarch64-unknown-linux-musl (e.g.: there's no cross-compiling going on here).

The above builds fine, but the resulting binary fails to run:

> cargo run  
   Compiling libc v0.2.137
   Compiling memmap2 v0.5.7
   Compiling xkbcommon v0.5.0
   Compiling hello v0.1.0 (/home/user/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 6.67s
     Running `target/debug/hello`
error: could not execute process `target/debug/hello` (never executed)

Caused by:
  No such file or directory (os error 2)

The resulting binary is clearly linked against glibc, which is not present here:

> ldd target/debug/hello
	/lib/ld-linux-aarch64.so.1 (0xffffacf0b000)
	libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0xffffacebc000)
	libc.musl-aarch64.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffacf0b000)
> readelf -l target/debug/hello

Elf file type is EXEC (Executable file)
Entry point 0x405328
There are 9 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R      0x8
  INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x000000000000001b 0x000000000000001b  R      0x1
      [Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x00000000000581dc 0x00000000000581dc  R E    0x10000
  LOAD           0x0000000000058e28 0x0000000000468e28 0x0000000000468e28
                 0x0000000000007338 0x0000000000008e40  RW     0x10000
  DYNAMIC        0x000000000005fcb0 0x000000000046fcb0 0x000000000046fcb0
                 0x00000000000001f0 0x00000000000001f0  RW     0x8
  TLS            0x000000000005d758 0x000000000046d758 0x000000000046d758
                 0x0000000000000028 0x0000000000000050  R      0x8
  GNU_EH_FRAME   0x0000000000056124 0x0000000000456124 0x0000000000456124
                 0x000000000000132c 0x000000000000132c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x0000000000058e28 0x0000000000468e28 0x0000000000468e28
                 0x00000000000071d8 0x00000000000071d8  R      0x1

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .debug_gdb_scripts .eh_frame_hdr .gcc_except_table 
   03     .eh_frame .tdata .init_array .fini_array .data.rel.ro .dynamic .got .data .bss 
   04     .dynamic 
   05     .tdata .tbss 
   06     .eh_frame_hdr 
   07     
   08     .eh_frame .tdata .init_array .fini_array .data.rel.ro .dynamic .got 

Steps

rustup default nightly
mkdir lalala
cd lalala
cargo init
cargo add xkbcommon
cat > src/main.rs <<EOF
fn main() {
    _ = xkbcommon::xkb::Context::new(0);
    println!("Hello, world!");
}
EOF
cargo run

Possible Solution(s)

Not sure, I think there's something wrong with the binaries that rustup ships.

Notes

Using cargo/rust/etc from Alpine repositories works fine. It's the binaries pulled by rustup that are faulty somehow.

Originally reported at the library I was trying to use (rust-x-bindings/xkbcommon-rs#37), but eventually realised it seems to be an issue with the binaries that rustup pulls.

Rustup version

rustup 1.25.1 :: v20220715+1493 (394daf022 2022-08-03) dirty 2 modifications
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.65.0 (897e37553 2022-11-02)`

Installed toolchains

Default host: aarch64-unknown-linux-musl
rustup home:  /home/user/.rustup

stable-aarch64-unknown-linux-musl (default)
rustc 1.65.0 (897e37553 2022-11-02)

[WhyNotHugo]

Edit: updated reproduction steps to make them copy-pasteable

[WhyNotHugo]

Updating title since I've now reproduced this on amd64 as wel.

[rbtcollins]

This should be moved to the rust repository I think; rustup isn't involved in the construction of the binaries in the distribution packages

[WhyNotHugo]

There's a likely explanation and some more thorough research on this bug in the downstream issue tracker.

[rbtcollins]

Ok, so the summary is mixed static-and-dylib linking on musl without the static-crt option The link from @WhyNotHugo has more details

[WhyNotHugo]

Still an issue with rustc 1.71.0-nightly (2c41369ac 2023-05-13).

cc: @Amanieu

[WhyNotHugo]

I think that the patch in Alpine that tweaks this behaviour is musl-fix-linux_musl_base.patch.

I'm not 100% certain though, there's a few other downstream patches.

[nekopsykose]

i don't think the selfcontained crt object set makes a difference for what default ldso is used. you can probably tweak that patch and leave the first lines untouched, and keep only the base.crt_static_default = false; override, and rebuild and check if that breaks it by default. my guess is it wouldn't make it ld-linux. but i could be wrong :)

[Amanieu]

I think this happening because libxkbcommon.a is not found so it links against libxkbcommon.so.1, which itself dynamically links to glibc. If you install libxkbcommon.a, does this fix the issue?

[WhyNotHugo]

lib/libxkbcommon.so.0 does not link to glibc:

> ldd /usr/lib/libxkbcommon.so.0
	/lib/ld-musl-x86_64.so.1 (0x7fd7dddb4000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fd7dddb4000)

There's no glibc on this host.

I tried installing libxkbcommon-static (which provides /usr/lib/libxkbcommon.a), but the result is the same, the resulting binary segfaults:

>  strace ./target/debug/reproducer 
execve("./target/debug/reproducer", ["./target/debug/reproducer"], 0x7ffcf1583680 /* 77 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7f42a765c8e8) = 0
set_tid_address(0x7f42a765ca98)         = 13314
poll([{fd=0, events=0}, {fd=1, events=0}, {fd=2, events=0}], 3, 0) = 0 (Timeout)
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f42a7636006}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGSEGV, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x7f42a760d1a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x7f42a7636006}, NULL, 8) = 0
rt_sigaction(SIGBUS, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=0x7f42a760d1a0, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x7f42a7636006}, NULL, 8) = 0
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f42a75e3000
mprotect(0x7f42a75e3000, 4096, PROT_NONE) = 0
sigaltstack({ss_sp=0x7f42a75e4000, ss_flags=0, ss_size=8192}, NULL) = 0
brk(NULL)                               = 0x555556e6f000
brk(0x555556e71000)                     = 0x555556e71000
mmap(0x555556e6f000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x555556e6f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f42a75e2000
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f42a7636006}, NULL, 8) = 0
rt_sigreturn({mask=[]})                 = 139924253016624
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV +++
zsh: segmentation fault  strace ./target/debug/reproducer

The output of ldd has changed since my initial bug report:

/lib/ld-musl-x86_64.so.1 (0x7f8cf53cd000)
libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0x7f8cf5318000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f8cf53cd000)

[Amanieu]

The problem is that rustc is expecting to build a static binary (and is passing the static-exe crt*.o files for it to the linker), but the linker is somehow ending up producing a dynamically linked binary.

I believe this is because of the -Bdynamic flag passed to the linker, which is incorrect when building a statically linked binary. This happens because the xkbcommon crate is using #[link = "xkbcommon"] with no kind attribute, which defaults to dylib.

This could be fixed in the xkbcommon crate by checking for the crt-static feature and using kind = "static" in that case.

[Amanieu]

Incidentally this seems to be a duplicate of #102993.

[nekopsykose]

I believe this is because of the -Bdynamic flag passed to the linker, which is incorrect when building a statically linked binary. This happens because the xkbcommon crate is using #[link = "xkbcommon"] with no kind attribute, which defaults to dylib.

This could be fixed in the xkbcommon crate by checking for the crt-static feature and using kind = "static" in that case.

i had arrived at a conclusion of something like this long ago (downstream in alpine, someone diffed a working/broken linker invocation and found exactly this Bdynamic issue, but we never followed it up ourselves). the issue is that this is inherently quite fragile- there aren't any warnings or failures or anything of the sort, just a silently added -Bdynamic in certain linkage cases that immediately yield you a broken output. perhaps there should be a diagnostic that having +crt-static with static-exe and any dynamic output is fundamentally broken (at least on -musl)?

diagnostic aside however, the fact that arbitrary crates can have a #[link] that would break this (via defaulting to something that adds the dynamic) means something would have to change to allow it to work without requiring every single crate out there to start passing around the type.. (i.e. a smarter default, perhaps defaulting on +-crt-static setting).

Incidentally this seems to be a duplicate of #102993.

yes, it seems to be exactly that issue. it seems that these discussions always become slightly tangential however, in that the discussion ends up centering around 'whether -musl is dynamic by default' or similar, instead of the root cause of +crt-static producing the broken output regardless of what the default is or not (maybe just a pet peeve of mine).

i notice you found why it does that default Bdynamic however, so thank you :) i was looking a long time ago and didn't come upon that exact location.

[Amanieu]

Can you give #111698 a try to see if it fixes the issue? I'm having a bit of trouble building a proper musl target on my end.