Mold causes Rust to lose backtraces in certain cases.

[itsfarseen]

Minimal repro:
https://github.com/itsfarseen/repro-sqlx-rocket-anyhow-no-backtrace

A notable thing is, if I comment out use sqlx; import, it works fine. If I don't import sqlx, I think Rust compiler is smart enough to skip compiling it.
I don't have to use anything inside sqlx, just importing it causes backtraces to be lost.

Cargo.toml

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
# rustflags = ["-Clink-arg=-fuse-ld=lld"]
rustflags = ["-Clink-arg=-fuse-ld=/usr/local/bin/mold"]

Rust version: rustc 1.53.0-nightly (ca075d268 2021-04-28)

[rui314]

Thank you for your report! Do I need to install an unofficial version of Rust to build your code? It looks like the stable version of Rust doesn't support the #![feature(backtrace)] directive.

[itsfarseen]

@rui314 yes you need to select nightly channel of Rust.

rustup toolchain install nightly
rustup default nightly

should do the trick.
This sets nightly as default globally. Alternatively, you can do rustup override set nightly to override it just for pwd

[ssokolow]

...or you can use cargo +nightly build or cargo +nightly run to use nightly for a single operation without changing the default toolchain.

(Which is very useful for things like using the stable toolchain, but using cargo +nightly fmt to gain access to unstable rustfmt features or cargo +nightly miri to run unsafe Rust's analogue to LLVM's UBSan, Valgrind's memcheck, and Go's data race detector which isn't yet part of the stable toolchain.)

[rui314]

@itsfarseen Thank you for the instruction. I think I successfully build your code, but it looks like a backtrace was displayed correctly as shown below.

$ ~/mold/mold -run cargo run
warning: unused manifest key: target.x86_64-unknown-linux-gnu.linker
warning: unused manifest key: target.x86_64-unknown-linux-gnu.rustflags
   Compiling sqlx-no-backtrace v0.1.0 (/tmp/repro-sqlx-rocket-anyhow-no-backtrace)
warning: unused import: `sqlx`
 --> src/main.rs:4:5
  |
4 | use sqlx;
  |     ^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

    Finished dev [unoptimized + debuginfo] target(s) in 0.29s
     Running `target/debug/sqlx-no-backtrace`
   0: sqlx_no_backtrace::main
             at ./src/main.rs:7:20
   1: core::ops::function::FnOnce::call_once
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/core/src/ops/function.rs:227:5
   2: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/sys_common/backtrace.rs:125:18
   3: std::rt::lang_start::{{closure}}
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/rt.rs:66:18
   4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/panicking.rs:379:40
      std::panicking::try
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/panicking.rs:343:19
      std::panic::catch_unwind
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/panic.rs:431:14
      std::rt::lang_start_internal
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/rt.rs:51:25
   5: std::rt::lang_start
             at /rustc/42816d61ead7e46d462df997958ccfd514f8c21c/library/std/src/rt.rs:65:5
   6: main
   7: __libc_start_main
   8: _start

I'm sure that the binary has been built by lld:

$ readelf -p .comment target/debug/sqlx-no-backtrace
String dump of section '.comment':
  [     1]  mold command line: <omit>
  [  2fde]  GCC: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
  [  3008]  mold 5d99b41328af5dae4654d778fa032834cf03b7f0

What am I missing?

[itsfarseen]

This is the output I'm getting:

   Compiling sqlx-no-backtrace v0.1.0 (/media/linux2/Projects/97.Temp/repro-sqlx-rocket-anyhow-no-backtrace)
warning: unused import: `sqlx`
 --> src/main.rs:4:5
  |
4 | use sqlx;
  |     ^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

    Finished dev [unoptimized + debuginfo] target(s) in 1m 31s
     Running `target/debug/sqlx-no-backtrace`
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: __libc_start_main
   8: <unknown>

[itsfarseen]

Oh my mold and gcc versions seem to be different:

$ readelf -p .comment target/debug/sqlx-no-backtrace

String dump of section '.comment':
  [     0]  GCC: (GNU) 10.2.0
  [    13]  mold a546295e86ccfb5b5fd0f8f1d0dedb65df4d4247

Let me try again with latest master.

Also my rust version is:

$ rustc --version
rustc 1.53.0-nightly (42816d61e 2021-04-24)

[itsfarseen]

Same with latest master:

$ readelf -p .comment target/debug/sqlx-no-backtrace

String dump of section '.comment':
  [     0]  mold 5ec4ffced47f185b05ae4325acac6eca78dc6b13
  [    2e]  GCC: (GNU) 10.2.0

[itsfarseen]

Possible points of differences:

• GCC Version.
• OS - I'm using Arch Linux.
• Maybe different build configuration of mold, because we are building from source? Could you send me the mold executable you are using?

[itsfarseen]

For me mold -run cargo run seems to be not working.
I have to put the following flags in ~/.cargo/config, and run just cargo run.

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-Clink-arg=-fuse-ld=/usr/local/bin/mold"]

Otherwise it gets built by LLD:

$ readelf -p .comment target/debug/sqlx-no-backtrace

String dump of section '.comment':
  [     0]  Linker: LLD 11.1.0
  [    13]  GCC: (GNU) 10.2.0

PS: I had copied mold and mold-wrapper.so to /usr/local/bin for convenience

[rui314]

I believe in your environment ld.lld is used by default, and that's why mold -run doesn't work. mold -run intercepts all calls to /usr/bin/ld and doesn't do anything with /usr/bin/ld.lld or something. (I probably should make a change to intercept not only ld but also ld.lld and ld.gold, but that's another feature.)

My rustc is rustc 1.53.0-nightly (42816d61e 2021-04-24), which is exactly the same as yours.

mold doesn't have a compile-time options. However you build it, the linker should behave the same.

I added a new feature so that it is easy to collect input files from mold users. Can you re-sync your local mold repo, rebuild mold and then re-run cargo build as MOLD_REPRO=1 cargo build? With this environment variable set, mold embeds all input files to .repro section as a tar file in an output file. You can see what is inside the tar file using the following commands:

$ objcopy --dump-section .repro=repro.tar target/debug/sqlx-no-backtrace
$ tar tvf repro.tar

Then please share the tar file (or the exectuable file) with me, so that I can run mold with the exact same input files as yours.

[itsfarseen]

Here's repro.tar.xz: https://file.io/EGi5GrZBrtFS

[rui314]

It looks like mold-linked debug info is broken. If I remove debug info from an executable with strip --strip-debug, it prints out a stack trace. dwarfdump says that mold-linked debug info is malformed. I'll investigate it further.

[rui314]

It looks like there's a bug in the code to handle compressed debug info. I'm preparing a patch.

[itsfarseen]

I can confirm that this fixes the backtrace issue in Rust 👍

[adsick]

Hello, what does unused manifest key: target.x86_64-unknown-linux-gnu.linker mean? I'm trying to speed up the compilation of my project, but it seems it is the same as before. Maybe this message indicates that I am doing something wrong?

[itsfarseen]

@adsick It probably means there is a typo in the key and the linker is not being set.

@rui314 Could you enable discussion feature for this repo? It would be better to discuss this over there.

[adsick]

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/mold"]

[itsfarseen]

@adsick Are you on windows or on 32bit or an ARM Linux?

[adsick]

I am on Fedora Linux 35 (64 bit)

[itsfarseen]

@adsick did you put this in Cargo.toml?

[adsick]

Of course I did

[itsfarseen]

@adsick it was supposed to go in ~/.cargo/config.toml

[adsick]

Ah, yes, I'll fix that, thanks a lot!

[rui314]

@itsfarseen I've enabled the discussion forum feature now.