Error linking standard library for x86_64-unknown-linux-musl target

[outkine]

I'm having an issue compiling to musl. I'm using cross:

cross build --target=x86_64-unknown-linux-musl --release --all-features

mimalloc compiles, but the final executable does not:

   Compiling libmimalloc-sys v0.1.15
   Compiling mimalloc v0.1.19
   Compiling lambda v0.1.0 (/project/runners/lambda)
error: linking with `x86_64-linux-musl-gcc` failed: exit code: 1
  |
  = note: "x86_64-linux-musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib" "/target/x86_64-unknown-linux-musl/release/deps/lambda-c655d3ea2f4baa0c.lambda.a24i26eq-cgu.5.rcgu.o" "-o" "/target/x86_64-unknown-linux-musl/release/deps/lambda-c655d3ea2f4baa0c" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/target/x86_64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build" "-L" "/target/x86_64-unknown-linux-musl/release/build/openssl-sys-dedc95d14540c99d/out/openssl-build/install/lib" "-L" "/target/x86_64-unknown-linux-musl/release/build/wasmer-runtime-core-23180f8bf205f7d9/out" "-L" "/target/x86_64-unknown-linux-musl/release/build/blake3-d963233bac6941d5/out" "-L" "/target/x86_64-unknown-linux-musl/release/build/blake3-d963233bac6941d5/out" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/tmp/rustclY9a9C/libwasmer_runtime_core-9b88a930581322af.rlib" "/tmp/rustclY9a9C/libblake3-85fe686220555d07.rlib" "/tmp/rustclY9a9C/libopenssl_sys-7655b1d66f49ab3a.rlib" "-Wl,--start-group" "/tmp/rustclY9a9C/libbacktrace_sys-c7370facec3b424f.rlib" "/tmp/rustclY9a9C/libunwind-150ab9ec13f39e6d.rlib" "/tmp/rustclY9a9C/liblibc-e131fd18530c560e.rlib" "-Wl,--end-group" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-8ce6a54cdd8c1827.rlib" "-Wl,-Bdynamic" "-lmimalloc-secure" "-static" "/rust/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o"
  = note: /target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build/libmimalloc-secure.a(options.c.o): In function `mi_vfprintf.part.2':
          options.c:(.text.mi_vfprintf.part.2+0x43): undefined reference to `stdout'
          /target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build/libmimalloc-secure.a(options.c.o): In function `mi_option_get':
          options.c:(.text.mi_option_get+0xb8): undefined reference to `strncat'
          options.c:(.text.mi_option_get+0x176): undefined reference to `toupper'
          options.c:(.text.mi_option_get+0x227): undefined reference to `toupper'
          /target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build/libmimalloc-secure.a(stats.c.o): In function `_mi_stats_print':
          stats.c:(.text._mi_stats_print+0x5ad): undefined reference to `getrusage'
          /target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build/libmimalloc-secure.a(os.c.o): In function `_mi_os_numa_node_count_get':
          os.c:(.text._mi_os_numa_node_count_get+0x88): undefined reference to `access'
          /target/x86_64-unknown-linux-musl/release/build/libmimalloc-sys-fa3bdc28ec03bf95/out/./build/libmimalloc-secure.a(os.c.o): In function `_mi_os_numa_node_get':
          os.c:(.text._mi_os_numa_node_get+0xe0): undefined reference to `access'
          collect2: error: ld returned 1 exit status
          
error: aborting due to previous error

error: could not compile `lambda`.

You can view the --verbose version here. My Cargo.toml:

[package]
name = "lambda"
version = "0.1.0"
authors = ["Anton <antonoutkine@gmail.com>", "Noah <33094578+coolreader18@users.noreply.github.com>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
logic = { path = "../../logic" }
lambda = { git = "https://github.com/awslabs/aws-lambda-rust-runtime/", branch = "master" }
rusoto_core = "0.44.0"
rusoto_sqs = "0.44.0"
serde_json = "1.0.55"
serde = { version = "1.0.114", features = ["derive"] }
serde_with = { version = "1.4.0", features = ["json"] }
tokio = "0.2.21"
once_cell = "1.4.0"
native-runner = { path = "../native" }
wasi-process = "0.1"
wasmer-runtime = "0.17.0"
wasmer-wasi = "0.17.0"

mimalloc = "0.1.19"

# https://github.com/sfackler/rust-openssl/issues/980#issuecomment-415757400
# Add openssl-sys as a direct dependency so it can be cross compiled to
# x86_64-unknown-linux-musl using the "vendored" feature below
openssl-sys = "0.9.58"
tempfile = "3.1.0"

[features]
# Force openssl-sys to staticly link in the openssl library. Necessary when
# cross compiling to x86_64-unknown-linux-musl.
vendored = ["openssl-sys/vendored"]

These errors look like a standard library linking issue, but unfortunately I am no CMAKE expert. Any ideas would be much appreciated :)

[octavonce]

@outkine My guess is that cmake does not receive cross-compilation flags during the build and uses your OS's libc instead of the musl-libc.

We also cross-compile on musl but we have opted to build in the same environment using docker. You can try to hack around cmake but using docker is the only reliable way I know to cross-compile rust projects with c/c++ dependencies.

I hope this helps you!

[outkine]

@octavonce Thank you for the recommendation! Building inside of Docker yields yet another strange error:

/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/libmimalloc-sys-0.1.15/c_src/mimalloc/src/os.c:26:10: fatal error: linux/mman.h: No such file or directory
 #include <linux/mman.h> // linux mmap flags
          ^~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [CMakeFiles/mimalloc-static.dir/src/os.c.o] Error 1
make[2]: *** [CMakeFiles/mimalloc-static.dir/all] Error 2
make[1]: *** [CMakeFiles/mimalloc-static.dir/rule] Error 2
make: *** [mimalloc-static] Error 2

I'm using the rust-musl-builder configuration here.

[outkine]

I've tried creating a custom docker file that also installs linux-headers-generic (per here) and libc6-dev (per here), but nothing has helped.

[outkine]

Setting some Rust compiler flags solved the problem. Full command:

RUSTFLAGS='-Clink-arg=-Wl,-Bstatic -Clink-arg=-lc' cross build --target=x86_64-unknown-linux-musl --release --all-features

In case anyone is having a related issue: these flags are supposed to be a good general toolkit for musl builds, so try them out!