Backtrace does not include file and line number on non-Linux platforms

[kornelski]

When a Rust program run with RUST_BACKTRACE=1 panics it outputs backtrace which only has function names (with noisy hashes), but doesn't include file names and line numbers.

 RUST_BACKTRACE=1 cargo run --verbose --example test

12: 0x1088db5d9 - main::hed092b4e13c9cca7Iaa

During development (not necessarily in release builds), I'd prefer something like:

12: 0x1088db5d9 - main() at test.rs:10

Sometimes line numbers are very important, e.g. if the panic is some generic out-of-bounds assertion, and the function uses vectors and slices in many places, just the name is not enough to pinpoint the offending expression.

[huonw]

This does work on some platforms for binaries compiled with debug-info (which cargo run does by default, unless you've edited profiles in the Cargo.toml); what are you on?

E.g. on x86-64 linux, I get:

fn foo() {
    panic!()
}
fn bar() {
    foo()
}
fn main() {
    bar()
}

thread '<main>' panicked at 'explicit panic', panic.rs:2
stack backtrace:
   1:     0x7fbdbdb2d5f9 - sys::backtrace::write::h6bf89e75ba594056cRC
   2:     0x7fbdbdb308f9 - panicking::on_panic::hbf9e41a2aefa0bc6ghJ
   3:     0x7fbdbdb2ad62 - rt::unwind::begin_unwind_inner::h41f8448b15c8a007oWI
   4:     0x7fbdbdb2a16e - rt::unwind::begin_unwind::h17882462255754141092
                        at /home/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/rt/unwind.rs:522
   5:     0x7fbdbdb2a0ba - foo::h846a49565d377979eaa
                        at /home/huon/projects/test-rust/<std macros>:3
   6:     0x7fbdbdb2a4bd - bar::h20543e655da0f511Eaa
                        at /home/huon/projects/test-rust/panic.rs:5
   7:     0x7fbdbdb2a4ed - main::h32ccc23e635fe73eJaa
                        at /home/huon/projects/test-rust/panic.rs:8
   8:     0x7fbdbdb34148 - rust_try_inner
   9:     0x7fbdbdb34135 - rust_try
  10:     0x7fbdbdb31f9b - rt::lang_start::hba3ed105db79dd3fJbJ
  11:     0x7fbdbdb2a634 - main
  12:     0x7fbdbcd3bb44 - __libc_start_main
  13:     0x7fbdbdb29f58 - <unknown>
  14:                0x0 - <unknown>

[kornelski]

I'm using rustc 1.0.0-nightly (6436e34 2015-04-08) (built 2015-04-08) on OS X 10.10.3.

[nareix]

rustc 1.2.0-nightly (0cc99f9 2015-05-17) (built 2015-05-17) in OS X 10.9.4 won't print line number.
But same version on Linux x86_64 can.

[FredrikNoren]

👍 on this one, sorely needed!

[alexbool]

Any news on this?

[FredrikNoren]

Looks like this has gotten even worse lately; type names seem to get mangled now. For instance I get this:

9:        0x10d924b37 - _<impl>::on_cycle::h0c7dcf14c18c4ee9Jpa

for a regular struct with an impl (not implementing any trait or anything).

[coriolinus]

The results right now in 1.5 are utterly useless:

> cargo run
     Running `target\debug\day21bin.exe`
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore\option.rs:366
stack backtrace:
   0:           0x4517a7 - <unknown>
   1:           0x45915e - <unknown>
   2:           0x415d1c - <unknown>
   3:           0x41648b - <unknown>
   4:           0x44e9c2 - <unknown>
   5:           0x473cd8 - <unknown>
   6:           0x46e130 - <unknown>
   7:           0x410530 - <unknown>
   8:           0x4107df - <unknown>
   9:           0x410a9a - <unknown>
  10:           0x410dd4 - <unknown>
  11:           0x40153e - <unknown>
  12:           0x4587e8 - <unknown>
  13:           0x441da1 - <unknown>
  14:           0x4586f7 - <unknown>
  15:           0x402a0a - <unknown>
  16:           0x4013b4 - <unknown>
  17:           0x4014e7 - <unknown>
  18:     0x7ffaf70a8101 - <unknown>
Process didn't exit successfully: `target\debug\day21bin.exe` (exit code: 101)

This is on Windows, but both Powershell and Cygwin64 produce identical results.

[OllyGinger]

Can confirm on macos 10.10.5

$ rustc --version
rustc 1.5.0 (3d7cd77e4 2015-12-04)

$ rustc -g file_embed.rs 

$ RUST_BACKTRACE=1 ./file_embed
thread '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 9, message: "Bad file descriptor" } }', ../src/libcore/result.rs:738
stack backtrace:
   1:        0x100ff9b88 - sys::backtrace::tracing::imp::write::h06df28c2a8d245f9t9s
   2:        0x100ffb1ef - panicking::log_panic::_<closure>::closure.39451
   3:        0x100ffac92 - panicking::log_panic::h9348b1b0af36ae6aCYw
   4:        0x100ff52c6 - sys_common::unwind::begin_unwind_inner::h005d402245ebd8560cs
   5:        0x100ff53fe - sys_common::unwind::begin_unwind_fmt::h440d2eb18c439fa86bs
   6:        0x100ff9217 - rust_begin_unwind
   7:        0x1010166b0 - panicking::panic_fmt::h4c8d12e3c05f3b8cZEK
   8:        0x100ff3566 - result::_<impl>::unwrap::unwrap::h5984419141872088150
   9:        0x100ff2a3d - write_cpp::hf673fd381a899287Gca
  10:        0x100ff4340 - main::h35317f0a7a25286cpia
  11:        0x100ffaa12 - sys_common::unwind::try::try_fn::h4733637608193038088
  12:        0x100ff9058 - __rust_try
  13:        0x100ffa8b9 - rt::lang_start::h44a8548d0ff91511KVw
  14:        0x100ff4789 - main

[TimNN]

cc @alexcrichton @lifthrasiir

From what I can tell you are the original authors of the line number in backtraces functionality (or at least familiar with the code) so I was wondering if one of you could chime in?

I'm most curios about whether the current situation is expected (ie. it is a known deficit of the current implementation) or if things should be working on os x (and thus there may be bug in the implementation)?

[salty-horse]

I can confirm on Linux, too, so it's not exclusive to mac.
rustc 1.7.0-nightly (0c7c94e 2015-12-26)

Correction: I do get line numbers on Linux. I no longer have the exact test-case I had before, but an attempt at reproducing it does show line numbers.

[sfackler]

@TimNN OSX support for filename and line numbers is unimplemented - the dladdr-based implementation here calls to output but not output_fileline which includes that information.

[shssoichiro]

Bump, also not working for me on OS X 10.11 with yesterday's nightly. The nightly on Gentoo Linux properly displays the line numbers however. Unfortunately I do most of my dev work on my Macbook so this makes debugging very difficult.

[shssoichiro]

After some investigating this seems to be a limitation of dladdr. dladdr is, as far as I can tell, unable to retrieve the source filename or line number (the filename it gets is the name of the compiled binary, which is not what we're looking for). And, since OS X uses clang by default, GCC's libbacktrace is unavailable on that platform.

I'm not sure if there is an implementation similar to libbacktrace that could be used to accomplish this on OS X, but I think that would most likely need to be the approach taken here.

[alexcrichton]

@TimNN sure, here's a summary of what I know:

• Linux - should have line numbers and filenames so long as debug info is emitted (e.g. the -g flag to the compiler). If no debug info is present they won't be printed. This information is somewhat resilient to inlining as well.
• OSX - no line numbers or filenames are printed anywhere. This is because a different method is used to acquire a backtrace. Symbols should always be available, however.
• Windows - no line numbers or filenames, and symbols are only sporadically available. The MSVC toolchain I think basically never has symbols, and I believe the GNU toolchain only sometimes does (but my memory may be hazy)
• All other platforms - most just have backtraces disabled or not working, the support is very spotty away from these three main ones.

@shssoichiro

Yes it's intentional there is no file/line information on OSX. If there's a library we could use, however, we could consider linking to it!

[emoon]

I have the same issue on Mac. My work-around is like that this:

1. Start your Rust executable with lldb lldb <executable>
2. Set a breakpoint in rust_begin_unwind breakpoint set -b rust_begin_unwind
3. Type run to start the executable.
4. When panic happens typ bt to see a backtrace with filenames

Not as nice as to see the filenames directly but better than nothing.

[korczis]

So what is the plan with this? Is there any ETA?

[sfackler]

@korczis Someone who feels sufficiently motivated will have to track down the appropriate libraries/APIs on Windows, OSX, BSD, etc and integrate them.

[wuranbo]

As @emoon saying, lldb can do it. Thanks. rust-lang/rfcs#1669 is relatived.

[alexcrichton]

So I think that at this point this bug is basically about OSX. I've seen really good backtrace line numbers on MSVC and Linux, so OSX is the only major desktop platform at least left that needs this.

@sujayakar has done some awesome work and discovered that the atos tool shipped on OSX actually already can do the conversion from address to filename/line number, we just need to call it! Unfortunately we probably can't shell out during a backtrace in the standard library, so we'll probably need to figure out how to do the parsing ourselves to figure out this kind of information.

The atos tool may be closed source, but Facebook apparently has an implementation for ARM which may be a great place to start with this perhaps? I'd also be quite interested in adding support immediately to the backtrace crate which shells out to a tool, as it could easily go behind a feature flag or something like that.

[alexcrichton]

@JohnColanduoni holy cow that's awesome!

I think we'd definitely be up to merge that as soon as it's ready!

Also if you're feeling inspired you could try upstreaming a local soundness fix as well :)

[petrochenkov]

upstreaming a local soundness fix

And this one as well 55e2b7e#diff-29b9b4427b4e6fd7450b277c2bc4fed5

libbacktrace accepting GitHub pull requests is something new (?)
I believe they required sending patches through GCC mailing lists previously.

[beatgammit]

This discussion has mostly been about MacOSX, but does anyone know where to start looking on FreeBSD (and probably other BSDs)? I use FreeBSD for my production servers, and it would be very nice to have this working on that platform.

I'll poke around, but it would be nice if someone could give mea hint as to where to start.

[JohnColanduoni]

@beatgammit

IIRC the issue with the BSDs is that there are some memory safety concerns about how libbacktrace handles malformed DWARF data, and the ability for an attacker to manipulate current_exe on those platforms (see here). libbacktrace supports ELF already, so it's not an issue on that side of things.

[lilith]

Line numbers are important to my development process. If I didn't have a linux box, I'd be dead in the water with Rust right now (particularly given my use of closures).

I'm confused as to why this is labeled I-papercut. Is anyone here still using OS X, and if so, what tools are you using to mitigate this? How do you debug? I must be missing something.

[sfackler]

I use a debugger to debug. The third party backtrace crate can produce line numbers on OSX, though it seems to be a bit unreliable on whether that actually ends up working.

[fitzgen]

Specifically, break on rust_panic.

Also, prefer expect("useful message here") to unwrap() (although that is just one specific kind of panic).

That's not to say that this isn't a large drain on productivity.

[lilith]

Thanks! I stopped trying to make lldb work with Rust back in October... but

lldb stack traces via b rust_panic and bt are working again :)

I'm shipping rust binaries to win/max/linux, so bug reports need good stack traces. Do you mean I shoul use backtrace-rs from std::panic::set_hook?

[sfackler]

Yep - here's a hook that should be equivalent to the standard library's format-wise for reference: https://github.com/sfackler/rust-log-panics/blob/master/src/lib.rs#L51

[lilith]

@sfackler backtrace-rs (after a few seconds) adds line numbers for the outer crate, but can't seem to resolve line numbers for other (local, debug) crates where most frames are at. Using OS X, nightly-2017-02-04, backtrace 0.3.0.

[jrmuizel]

To find the dSYM files you can use the DebugSymbols framework. That's what gdb/lldb use. You can find some information about using it here: https://github.com/jrmuizel/dsym-find/blob/master/dsym-find.c

[lilith]

Is it necessary that we wait for upstream acceptance of libbacktrace patches? OS X has been waiting a while to get good runtime errors, and the workarounds are unreliable.

[JohnColanduoni]

@jrmuizel DebugSymbols is a private framework. At that point, we might as well use the private CoreSymbolication framework which will handle the whole process.

@nathanaeljones

I've fixed the issues for that pull request. I'd be happy to put together one for the Rust codebase (should be pretty minimal, there's no API changes) if that's acceptable.

[lilith]

@JohnColanduoni I think such a PR would be great.

I'd also love to see this be upgraded from "papercut". I ship server software that other people run; backtraces are crucial.

[adaszko]

I'd also love to see this be upgraded from "papercut".

+1

---

Another workaround I'm currently using is the trace-error crate. There are caveats though:

• Pretty heavy modifications to the code base are required (special macros instead of the ? operator)
• Works only in debug mode
• These macros have a significant presence in the profiling output

[steveklabnik]

Triage: there hasn't been any comments here in a long time, but there has been a ton of changes in the past 18 months. Is this still an issue?

[nbigaouette-eai]

I've haven't been hit by this since a long time on two macos machines.

 -> cat src/main.rs
fn main() {
    panic!("Hello, world!");
}

 -> cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.21s
     Running `target/debug/issue24346`
thread 'main' panicked at 'Hello, world!', src/main.rs:2:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at libstd/panicking.rs:476
   5: std::panicking::begin_panic
             at libstd/panicking.rs:410
   6: issue24346::main
             at src/main.rs:2
   7: std::rt::lang_start::{{closure}}
             at libstd/rt.rs:74
   8: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
   9: macho_symbol_search
             at libpanic_unwind/lib.rs:102
  10: std::alloc::default_alloc_error_hook
             at libstd/panicking.rs:289
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  11: std::rt::lang_start
             at libstd/rt.rs:74
  12: issue24346::main

[djc]

Yeah, I think this has been fixed for a while.