Regression when using linker=arm-none-eabi-gcc

[adamgreig]

The cortex-m CI tests building with all our supported linkers: Rust's provided lld, ARM GCC's arm-none-eabi-ld, and ARM GCC's arm-none-eabi-gcc. It recently encountered an error when using arm-none-eabi-gcc in nightly builds since #112910.

The most likely culprit is that the failing version has started added "-fuse-ld=lld" to the linker arguments, which was not present in previous working versions. @lqd, I wonder if we're meant to be adding some other argument to disable adding -fuse-ld=lld?

Code

The error occurs with the cortex-m-rt CI and can be reproduced like so:

git clone https://github.com/rust-embedded/cortex-m
cd cortex-m/cortex-m-rt
rustup toolchain install nightly-2023-07-03
rustup target add thumbv6m-none-eabi --toolchain nightly-2023-07-03
cargo +nightly-2023-07-03 rustc --target thumbv6m-none-eabi --example alignment --features cortex-m/critical-section-single-core -- -C linker=arm-none-eabi-gcc -C link-arg=-nostartfiles

Or viewed in CI here.

The error is:

error: linking with `arm-none-eabi-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="<snip>" VSLANG="1033" "arm-none-eabi-gcc" "/tmp/rustcFHUl0t/symbols.o" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/examples/alignment-c363c9558a2fe06e.15yn3mrxz53m7uw7.rcgu.o" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/examples/alignment-c363c9558a2fe06e.6wqau5vl49hu4fi.rcgu.o" "-Wl,--as-needed" "-L" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/deps" "-L" "/home/adam/Projects/re/cortex-m/target/debug/deps" "-L" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/build/cortex-m-rt-69b7ab10efe30aa1/out" "-L" "/home/adam/.rustup/toolchains/nightly-2023-07-03-x86_64-unknown-linux-gnu/lib/rustlib/thumbv6m-none-eabi/lib" "-Wl,-Bstatic" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/deps/libpanic_halt-f60a67ec71296eeb.rlib" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/deps/libcortex_m_rt-9a1a89eba3425cf7.rlib" "/home/adam/.rustup/toolchains/nightly-2023-07-03-x86_64-unknown-linux-gnu/lib/rustlib/thumbv6m-none-eabi/lib/librustc_std_workspace_core-cc16e9be0f247487.rlib" "/home/adam/.rustup/toolchains/nightly-2023-07-03-x86_64-unknown-linux-gnu/lib/rustlib/thumbv6m-none-eabi/lib/libcore-b9a65150d0351313.rlib" "/home/adam/.rustup/toolchains/nightly-2023-07-03-x86_64-unknown-linux-gnu/lib/rustlib/thumbv6m-none-eabi/lib/libcompiler_builtins-2769d4646d3d396e.rlib" "-Wl,-Bdynamic" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/adam/.rustup/toolchains/nightly-2023-07-03-x86_64-unknown-linux-gnu/lib/rustlib/thumbv6m-none-eabi/lib" "-o" "/home/adam/Projects/re/cortex-m/target/thumbv6m-none-eabi/debug/examples/alignment-c363c9558a2fe06e" "-Wl,--gc-sections" "-no-pie" "-nodefaultlibs" "-nostartfiles" "-Tlink.x"
  = note: collect2: fatal error: cannot find 'ld'
          compilation terminated.

Version it worked on

nightly-2023-07-02

Version with regression

nightly-2023-07-03

bisected to 8e2d5e3

from PR #112910

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

[jyn514]

rust/compiler/rustc_target/src/spec/thumb_base.rs

Line 37 in 993deaa

linker: Some("rust-lld".into()),

shows that the default for the target is rust-lld. i wonder if the new code uses the default instead of the linker in -C linker for some reason?

[lqd]

I wonder if we're meant to be adding some other argument to disable adding -fuse-ld=lld?

You were not meant to do anything, sorry about that. I'll fix it.

[lqd]

I can't build a thumbv6m-none-eabi toolchain locally, via ./x build library/ --target thumbv6m-none-eabi, which would be complete enough for this cortex-m test to build. There are issues about missing core/std.

I tried on a no_std example, and can also reproduce the arm-none-eabi-gcc issue and investigate more there, and that rust-lld and arm-none-eabi-ld work.

The code I added in #112910 is asked to add CLI arguments for a Gnu(Cc::Yes, Lld::Yes) flavor: this asks to link using lld via a c/c++ compiler, and why -fuse-ld=lld is added. I assume that flavor is incorrect when using -C linker=arm-none-eabi-gcc -C link-arg=-nostartfiles @adamgreig ? (I don't know who the target maintainers are, so I'll also ping the other 2 people involved in the discovery: @hannobraun @thejpster)

-C linker=arm-none-eabi-ld and -C linker=rust-lld still work because they result in a non-cc request, and we don't do anything there as expected.

For my own understanding: does arm-none-eabi-gcc use arm-none-eabi-ld by default maybe ? And if so, I'd expect that explicitly specifying the linker would also fix the issue, à la -C linker=arm-none-eabi-gcc -C link-arg=-nostartfiles -C link-arg=-fuse-ld=bfd which seems to work for me ?

I'll look into why this surprising flavor is used in this situation, as it seems to be the pre-existing issue here uncovered by the PR, and report back. It could be an issue in the flavor inference, or some information missing in the targets' definitions, and so on.

[workingjubilee]

Technically, the target builds.
As a tier 2 target, we would strongly prefer it to not require additional antics to build.

There was some discussion as to whether this should be P-high but I have taken lqd's remarks that this may be just exposed preexisting problems as a reason to set it slightly lower.

[lqd]

The linker flavor used is a mix between what can be inferred from the CLI (-C linker) and the target's default linker flavor:

• there is no linker flavor on the CLI (and that also offers another workaround on nightly: -C linker-flavor=gnu-cc -Zunstable-options), so it will have to be inferred from here to here.
• in infer_linker_hints -C linker=arm-none-eabi-gcc infers a Some(Cc::Yes) cc hint, and no hint about lld.
• the target's linker_flavor is combined in with_cli_hints with these hints. We have our Cc::Yes, but there is no hint about lld, so the target's flavor lld component is used. It's Gnu(Cc::No, Lld::Yes).
• so we now have our Gnu(Cc::Yes, Lld::Yes) flavor

@adamgreig @hannobraun @thejpster: is the linker people are expected to use on thumbv6m-none-eabi supposed to be some lld ? Is the default linker and flavor in thumb_base.rs correct ?

In the meantime, I'll see to change this so that the lld flags are only added when the unstable MCP510 options are used, to keep this stable and unfortunate behavior unchanged when the CLI and target conflict instead of compose.

[adamgreig]

Thanks for looking into this quickly!

Most people building for thumbv6m-none-eabi will use the default rust-lld linker today. It works fine for (I believe) all Rust-only projects. However, it's not uncommon to use arm-none-eabi-ld, maybe to use specific GCC linker script directives, or just when hunting down linker-related bugs. For both of those use cases there's no problem with nightly, and I think the default in thumb_base is therefore fine.

Some people do need to use arm-none-eabi-gcc when linking with pre-compiled C libraries - I understand that using gcc pulls in a target specific libc, and a lot of pre-compiled C libraries will be trying to link against symbols it exports which Rust doesn't. I believe arm-none-eabi-gcc does delegate to arm-none-eabi-ld.

Passing either -C linker-flavor=gnu-cc -Zunstable-options or -C link-arg=-fuse-ld=bfd does work to get a build without errors, though I don't have a test case handy to check it's actually getting what people need out of arm-none-eabi-gcc. I'll try and dig around a bit more for the specific use cases where it's required over arm-none-eabi-ld.

[thejpster]

My main recollection is that gcc requires you to put -Wl, in front of your linker arguments and using ld directly you don't 🤣

But I think -nodefaultlibs and friends are gcc options not ld options so maybe ld won't bring in libc and the C startup automatically. I'll test it out shortly.

[thejpster]

main.c:

int main() {
    puts("Hello, world");
    return 0;
}

Compile object code:

$ arm-none-eabi-gcc -o main.o -c main.c -mcpu=cortex-m3

Link with GCC:

$ arm-none-eabi-gcc -o main main.o -mcpu=cortex-m3
<bunch of errors about missing _close_r, _lseek_r and other syscalls that the newlib C library requires>

Link with ld:

$ arm-none-eabi-ld -o main main.o -mcpu=cortex-m3
warning: cannot find entry symbol _start
main.o: in function `main` main.c: undefined reference to `puts`

This does imply that for trivial invocations arm-none-eabi-gcc will include both the default start code (which provides _start) and newlib (the C library), whilst arm-none-eabi-ld will not.

[thejpster]

See also #113535, where I set the default target for bare-metal SPARC to be gcc, precisely because the default libraries it links in are so useful - they include not only the start-up code (no idea how that works, didn't need to find out) but also a C library for talking to the stdout of a popular SPARC CPU emulator.

So, yeah, GCC is a really useful linker driver in some scenarios and we don't want to stop that from working.

[lqd]

Thanks for the information, it's very helpful.

The good news is that using GCC as a linker is still working 😅. In the future where we switch to lld, maybe rustc will also need an explicit linker flavor here, to make sure the flavor it guesses actually matches your expectations. But that's a topic for another day/stabilization: in the more immediate future, things will return to normal for the target and cortex-m's tests in a day or two.

[thejpster]

Thank you!

Just to come back to this point:

I don't know who the target maintainers are

I was under the impression that Arm Ltd were maintaining the thumbv*m-none-eabi* targets. However, they don't appear to have the platform-support documentation that is required for more recently added targets. Maybe that could be addressed.

[lqd]

This is now on beta, reopening to track backport.

[workingjubilee]

@thejpster Pursuant to your concern I have opened #113739 to summarize the issue that affects indeed all our tier 2 targets.

[lqd]

The backport of #113631 has landed on beta in #113938. We can close this issue as fixed.