Cross-compiling with clang failed because cargo doesn't specify --target in the linking command

[inflation]

Problem

Issues first filed here: rui314/mold#564

I'd like to cross-compile a rust project from aarch64-apple-darwin to x86_64-unknown-linux-gnu with these settings:

[target.x86_64-unknown-linux-gnu]
rustflags = [
  "-Clink-arg=-fuse-ld=/opt/homebrew/bin/ld.mold",
  "-Clink-arg=--sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot"]

But if failed with unrecognized option:

unrecognized option '-dynamic'

However, the linker command emitted doesn't show -dynamic but -Wl,-Bdynamic:

 "cc" "-m64"
 ...
"-Wl,--as-needed" 
"-L" "/private/tmp/testing/target/x86_64-unknown-linux-gnu/debug/deps"
"-L" "/private/tmp/testing/target/debug/deps" 
"-L" "/Users/inflation/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" 
"-Wl,--start-group" "-Wl,-Bstatic" 
...
"-Wl,--end-group" 
"/Users/inflation/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-377835cfab8dae0d.rlib" 
"-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" 
"-L" "/Users/inflation/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" 
"-o" "/private/tmp/testing/target/x86_64-unknown-linux-gnu/debug/deps/testing-c379b9730434cd22" 
"-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs" 
"-fuse-ld=/opt/homebrew/bin/ld.mold" "--sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot"

Further investigation shows that it's not only affecting mold, but also ld.lld from llvm and x86_64-unknown-linux-gnu-ld from the cross-compilation toolchain.

Steps

0. Download a cross-compiling toolchain or use crosstool-ng
1. cargo new testing
2. cd testing && mkdir .cargo
3. For mold:

[target.x86_64-unknown-linux-gnu]
rustflags = [
  "-Clink-arg=-fuse-ld=/opt/homebrew/bin/ld.mold",
  "-Clink-arg=--sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot"]

For x86_64-unknown-linux-gnu-ld:

[target.x86_64-unknown-linux-gnu]
rustflags = [
  "-Clink-arg=-fuse-ld=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-ld",
  "-Clink-arg=--sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot"]

For ld.lld:

[target.x86_64-unknown-linux-gnu]
rustflags = [
  "-Clink-arg=-fuse-ld=/opt/homebrew/opt/llvm/bin/ld.lld",
  "-Clink-arg=--sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot"]

4. cargo build --target=x86_64-unknown-linux-gnu and fail

Possible Solution(s)

So, I guess if the linker is clang on macOS, regardless of what target it is, it always passes linker options in ld64 forms somehow.

Notes

Setting x86_64-unknown-linux-gnu-gcc as the linker and using mold or lld is fine:

ln -s /opt/homebrew/bin/ld.mold /Volumes/CSP/x-tools/aarch64-linux-gnu/aarch64-linux-gnu/bin/ld

[target.aarch64-unknown-linux-gnu]
linker = "/Volumes/CSP/x-tools/aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc"

Version

cargo 1.62.0 (a748cf5a3 2022-06-08)
release: 1.62.0
commit-hash: a748cf5a3e666bc2dcdf54f37adef8ef22196452
commit-date: 2022-06-08
host: aarch64-apple-darwin
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.79.1 (sys:0.4.51+curl-7.80.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
os: Mac OS 12.4.0 [64-bit]

[ehuss]

rustc generally uses cc to perform linking unless you tell it otherwise (with the linker config option in cargo). The C-compiler will be responsible for finding system libs and such (which a bare linker won't do). macOS clang won't work for cross-compiling to other targets. As you've discovered, you'll need a gcc built for x86_64-unknown-linux-gnu.

Closing as this isn't an issue with Cargo. If you need more help with cross-compiling, I suggest trying one of the user forums such as https://users.rust-lang.org/.

[inflation]

@ehuss, But it's perfectly fine if I invoke clang directly:

clang -target x86_64-unknown-linux-gnu --sysroot=/Volumes/CSP/x-tools/x86_64-unknown-linux-gnu/sysroot -fuse-ld=/opt/homebrew/bin/ld.mold test.c

This produces a valid ELF executable.

EDIT: Also tried with “real” clang from LLVM, and it also failed in Rust but succeed in C.

[inflation]

So, because clang is a cross-compiler by itself, it needs to use a target options to do that. And the emitted linking command doesn't specify --target or -target. This means that clang would choose options regardless of target settings. Manually adding those in build.rs solves the problem:

fn main() {
    let target = std::env::var("TARGET").unwrap();
    println!(
        "cargo:rustc-link-arg=--target={}", target
    );
    println!(
        "cargo:rustc-link-arg=-fuse-ld=/opt/homebrew/bin/ld.mold"
    );
    println!(
        "cargo:rustc-link-arg=--gcc-toolchain=/Volumes/CSP/x-tools/{}", target
    );
    println!(
        "cargo:rustc-link-arg=--sysroot=/Volumes/CSP/x-tools/{}/{}/sysroot", target, target
    );
}

I think cargo should explicitly output --target in the linking command if it's using clang. But I'm aware it's just passing options to cc so I won't push any further. I'll save this workaround for anyone who wonders it later.