Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc(with --enable-debuginfo) hangs for 20-45 seconds on errors(of any kind) when RUST_BACKTRACE is set #29293

Closed
ghost opened this issue Oct 25, 2015 · 27 comments
Labels
T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@ghost
Copy link

ghost commented Oct 25, 2015

EDIT: when env var RUST_BACKTRACE is set to any value (even empty) [it was set to 1 for me here]
EDIT2: I had rust compiled with --enable-debuginfo; when compiled with --disable-debuginfo then it only takes 10 times more time than normal: that is, 0.6sec instead of 0.05 sec

$ time rustc -VV
error: Option 'version' given more than once.

real    0m47.717s
user    0m47.273s
sys 0m0.250s
$ time rustc -Vv
rustc 1.5.0-dev (04e497c00 2015-10-24)
binary: rustc
commit-hash: 04e497c0056aed899cd6edbc98e7a68a9b391c5c
commit-date: 2015-10-24
host: x86_64-unknown-linux-gnu
release: 1.5.0-dev

real    0m0.068s
user    0m0.017s
sys 0m0.030s

I thought the culprit was cargo at first: rust-lang/cargo#2079 but nope.

@ghost
Copy link
Author

ghost commented Oct 25, 2015

I forgot to mention that this may be related: #11493

EDIT: however it hangs on the next line for me, next (empty) line after error: Option 'version' given more than once.

@achanda
Copy link
Contributor

achanda commented Oct 25, 2015

Can't seem to repro this on OSX

:~/src/rust (master=)$ time rustc -VV
error: Option 'version' given more than once.

real    0m0.106s
user    0m0.054s
sys 0m0.055s
:~/src/rust(master=)$ time rustc -V
rustc 1.5.0-nightly (1210fb9bc 2015-10-23)

real    0m0.105s
user    0m0.053s
sys 0m0.053s

@dylanmckay
Copy link
Contributor

Can't reproduce it on Linux.

@ghost
Copy link
Author

ghost commented Oct 25, 2015

Thanks for testing guys.
I just notice that while it's hanged, it uses 100% cpu

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
16019 zazdxscf  20   0  371032  87312  47316 S 100.0  0.6   0:09.06 rustc       
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
16038 zazdxscf  20   0  395396 116692  52220 S 100.0  0.7   0:42.02 rustc       

@ghost
Copy link
Author

ghost commented Oct 25, 2015

It works fine like this:

$ time sudo --user=zazdxscf /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m0.109s
user    0m0.047s
sys 0m0.057s

but not like this:

$ time /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m44.118s
user    0m43.787s
sys 0m0.313s

Here's a strace:

$ time strace /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV 2>/tmp/strace3.log 

real    0m44.451s
user    0m44.073s
sys 0m0.360s
$ cat /tmp/strace3.log

EDIT: moved here to make scrolling here on this page easier.

@ghost
Copy link
Author

ghost commented Oct 25, 2015

Here's a crazier strace, at the end we see most of the time is spend in futex stuff
https://gist.githubusercontent.com/zazdxscf/7768f1a4679ad2d60b7e/raw/0309272e682c93507321825e97e8d4cbe43abccd/strace.log

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.60    1.126667      375556         3           futex
  0.68    0.007794          14       558           read
  0.41    0.004738           0     11064           mmap
  0.24    0.002716          17       159           munmap
  0.05    0.000552           7        81        13 open
  0.02    0.000235           3        68           close
  0.00    0.000000           0         8           write
  0.00    0.000000           0        10         8 stat
  0.00    0.000000           0        33           fstat
  0.00    0.000000           0        24           lseek
  0.00    0.000000           0        66           mprotect
  0.00    0.000000           0         4           brk
  0.00    0.000000           0         5           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0        25           madvise
  0.00    0.000000           0         1           clone
  0.00    0.000000           0         1           execve
  0.00    0.000000           0        33           fcntl
  0.00    0.000000           0         2         1 readlink
  0.00    0.000000           0         2           getrlimit
  0.00    0.000000           0         2           sigaltstack
  0.00    0.000000           0         1           prctl
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         2           sched_getaffinity
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         2           set_robust_list
  0.00    0.000000           0         2           getrandom
------ ----------- ----------- --------- --------- ----------------
100.00    1.142702                 12162        23 total

EDIT: for comparison, here's when it works(doesn't hang)
https://gist.github.com/zazdxscf/f30352d3deffce5c1439

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 77.45    0.026666        8889         3           futex
 19.98    0.006879          12       558           read
  1.38    0.000475           7        66           mprotect
  0.58    0.000201           4        47        12 open
  0.38    0.000131           1        99           mmap
  0.23    0.000078           2        33           fstat
  0.00    0.000000           0         8           write
  0.00    0.000000           0        35           close
  0.00    0.000000           0        10         8 stat
  0.00    0.000000           0        24           lseek
  0.00    0.000000           0        12           munmap
  0.00    0.000000           0         4           brk
  0.00    0.000000           0         5           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0        23           madvise
  0.00    0.000000           0         1           clone
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         2         1 readlink
  0.00    0.000000           0         2           getrlimit
  0.00    0.000000           0         2           sigaltstack
  0.00    0.000000           0         1           prctl
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         2           sched_getaffinity
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         2           set_robust_list
  0.00    0.000000           0         2           getrandom
------ ----------- ----------- --------- --------- ----------------
100.00    0.034430                   948        22 total

@ghost
Copy link
Author

ghost commented Oct 25, 2015

Oh figured it out: it's because RUST_BACKTRACE was set in my environment (set to 1)

$ unset RUST_BACKTRACE
time /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m0.106s
user    0m0.053s
sys 0m0.050s

even if I set it to 0 or empty value, the issue remains:

$ time RUST_BACKTRACE=0 /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m44.758s
user    0m44.310s
sys 0m0.390s

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m44.767s
user    0m44.460s
sys 0m0.283s

EDIT: This explains why rust tests were taking ages to complete... (half a day at least, but I stopped them)
EDIT2: Also realized that the value likely means nothing, and just the fact that the var is set is considered enabled.

@ghost ghost changed the title rustc hangs for 20-45 seconds on errors(of any kind) rustc hangs for 20-45 seconds on errors(of any kind) when RUST_BACKTRACE is set Oct 25, 2015
ghost referenced this issue in alexcrichton/cargo Oct 25, 2015
This commit removes the ndebug support from Cargo and also adds a new
configuration option for profiles, `debug-assertions`, which controls whether
debug assertions in the compiler are turned on or not.

Closes rust-lang#1398
@nagisa
Copy link
Member

nagisa commented Oct 25, 2015

futex taking so long seems to only be indicative of some mutex waiting to get a lock for a long time.

What seems to be troubling is count of mmap calls. Even if their time/call seems to be recorded as 0, there’s syscall overhead to think about (the overhead is not big enough to cause 40 seconds though).

That being said, I can’t reproduce this (nor increase in syscall count/duration) either.

@alexcrichton
Copy link
Member

@zazdxscf like others, I unfortunately can't reproduce this (even with RUST_BACKTRACE being set). Lots of mmap syscalls may be indicative of something just going into an infinite loop of allocating, so could you also try to use gdb to capture some stack traces? That may help us pin down where this is coming from.

@ghost
Copy link
Author

ghost commented Oct 26, 2015

tl;dr: hang time reduced by 20 times(now at 2.6 sec instead of 40 sec) after recompiling rust(with newer latest master and unoptimized llvm) and someone please tell me how to make gdb show symbols-ridden stacktrace. Thanks.

I'm having trouble figuring out how get a stacktrace (with symbols). I need help (maybe someone could point me to an article on how to debug rustc with gdb, but not how to debug rust programs with gdb).

$ PATH="$PATH:/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage0/bin/" RUST_BACKTRACE=1 rust-gdb -- rustc
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rustc...done.
(gdb) run -VV
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages!
[New LWP 7481]
error: Option 'version' given more than once.
^C
Program received signal SIGINT, Interrupt.
0x0000033afe21e9ad in ?? ()
(gdb) bt full
#0  0x0000033afe21e9ad in ?? ()
No symbol table info available.
#1  0x000000000000000e in ?? ()
No symbol table info available.
#2  0x00000000fbc24120 in ?? ()
No symbol table info available.
#3  0x0000033afe21e8d0 in ?? ()
No symbol table info available.
#4  0x0000033afbb6bd28 in ?? ()
No symbol table info available.
#5  0x0000033afbc00000 in ?? ()
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Hmm now that I look at this gdb from 2 days ago, I notice some have symbols and some don't #29270 (comment)
Is it because the source is not c/c++ ?

On another note: I noticed that I cannot reproduce it with the snapshot rust

$ time RUST_BACKTRACE=1 /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage0/bin/rustc -VV
error: Option 'version' given more than once.

real    0m0.401s
user    0m0.360s
sys 0m0.020s

$ time RUST_BACKTRACE=1 /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage0/bin/rustc -Vv
rustc 1.4.0-dev (1af31d497 2015-08-11)
binary: rustc
commit-hash: 1af31d4974e33027a68126fa5a5a3c2c6491824f
commit-date: 2015-08-11
host: x86_64-unknown-linux-gnu
release: 1.4.0-dev

real    0m0.018s
user    0m0.000s
sys 0m0.010s

Although there's 0.4 sec vs 0.02 sec ! I can feel that hang a litle, but let's say it doesn't at all.

Now with non-optimized debug-enabled llvm (this --enable-debug-symbols --disable-optimized --enable-debug-runtime to llvm, and this --disable-optimize-llvm to rust's ./configure; instead of this --enable-optimized to llvm(which implies no debug symbols) because of this --enable-optimize-llvm to rust's ./configure), the time spent is 10-20 times less:
(I swear I had output that was consistently showing 4.3 seconds(like 10 times less), but now consistently 2.6 sec (like 20 times less, than I previously reported))

stage0 rust (not snapshot stage0)

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/bin/rustc -VV 
error: Option 'version' given more than once.

real    0m2.671s
user    0m2.407s
sys 0m0.070s

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/bin/rustc -Vv
rustc 1.5.0-dev (2a418216f 2015-10-25)
binary: rustc
commit-hash: 2a418216feeb83fd3f68c725c0e5577beacff59b
commit-date: 2015-10-25
host: x86_64-unknown-linux-gnu
release: 1.5.0-dev

real    0m0.123s
user    0m0.080s
sys 0m0.030s

stage1 rust

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
error: Option 'version' given more than once.

real    0m2.654s
user    0m2.410s
sys 0m0.090s

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -Vv
rustc 1.5.0-dev (2a418216f 2015-10-25)
binary: rustc
commit-hash: 2a418216feeb83fd3f68c725c0e5577beacff59b
commit-date: 2015-10-25
host: x86_64-unknown-linux-gnu
release: 1.5.0-dev

real    0m0.150s
user    0m0.070s
sys 0m0.037s

stage2 is still compiling... Good god! :)) (but I don't want go, I want rust) EDIT: my bad for having set cpu governor to conservative!! (it was stuck at 800Mhz even thought rustc was using 100% cpu, aka 1 thread, instead of 4) switched back to ondemand. cpupower frequency-set -g ondemand (which was is the default). The thing is, it was actually pretty fast until after stage1 got compiled; but then, it's been some serious hours since! (likely because of that 800Mhz)
EDIT2: here's stage2 (which takes normal time, compared to stage1)

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m46.847s
user    0m45.843s
sys 0m0.313s

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -Vv
rustc 1.5.0-dev (2a418216f 2015-10-25)
binary: rustc
commit-hash: 2a418216feeb83fd3f68c725c0e5577beacff59b
commit-date: 2015-10-25
host: x86_64-unknown-linux-gnu
release: 1.5.0-dev

real    0m0.181s
user    0m0.080s
sys 0m0.030s

$ time /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.

real    0m0.116s
user    0m0.083s
sys 0m0.033s

(I did rerun stage1 after this, and it's still 2.5 sec)

$ RUST_BACKTRACE= gdb -- /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc...done.
(gdb) run -VV
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages!
[New LWP 31140]
error: Option 'version' given more than once.
^C
Program received signal SIGINT, Interrupt.
0x0000035c616709ad in ?? ()
(gdb) run -VVQuit
(gdb) thread apply all bt full

Thread 2 (LWP 31140):
#0  0x0000035c694cc986 in ?? ()
No symbol table info available.
#1  0x0000035c5fb20ca0 in ?? ()
No symbol table info available.
#2  0x0000035c694cc461 in ?? ()
No symbol table info available.
#3  0x0000035c694cc95e in ?? ()
No symbol table info available.
#4  0x0000000000000018 in ?? ()
No symbol table info available.
#5  0x0000000000000685 in ?? ()
No symbol table info available.
#6  0x0000035c5ce548e8 in ?? ()
No symbol table info available.
#7  0x0000035c5ce548e8 in ?? ()
No symbol table info available.
#8  0x000000000000011b in ?? ()
No symbol table info available.
#9  0x000000000000031f in ?? ()
No symbol table info available.
#10 0x0000035c5ce548e8 in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x0000035c5fb20cf0 in ?? ()
No symbol table info available.
#12 0x0000035c694cc56a in ?? ()
No symbol table info available.
#13 0x0000035c694cc95e in ?? ()
No symbol table info available.
#14 0x0000000000000018 in ?? ()
No symbol table info available.
#15 0x00000000000016ee in ?? ()
No symbol table info available.
#16 0x0000035c5ce3c708 in ?? ()
No symbol table info available.
#17 0x0000035c5ce64830 in ?? ()
No symbol table info available.
#18 0x0000000000001013 in ?? ()
No symbol table info available.
#19 0x00000000000016ee in ?? ()
No symbol table info available.
#20 0x0000035c5ce3c708 in ?? ()
No symbol table info available.
#21 0x0000035c5fb20d40 in ?? ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#22 0x0000035c694cc50d in ?? ()
No symbol table info available.
#23 0x0000035c694cc95e in ?? ()
No symbol table info available.
#24 0x0000000000000018 in ?? ()
No symbol table info available.
#25 0x0000000000004c32 in ?? ()
No symbol table info available.
#26 0x0000035c5cdb7e80 in ?? ()
No symbol table info available.
#27 0x0000035c5cdb7e80 in ?? ()
No symbol table info available.
#28 0x0000000000001ab6 in ?? ()
No symbol table info available.
#29 0x0000000000004c32 in ?? ()
No symbol table info available.
#30 0x0000035c5ce3c708 in ?? ()
No symbol table info available.
#31 0x0000035c5fb20d90 in ?? ()
No symbol table info available.
#32 0x0000035c694cc56a in ?? ()
No symbol table info available.
#33 0x0000035c694cc95e in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#34 0x0000000000000018 in ?? ()
No symbol table info available.
#35 0x000000000001d455 in ?? ()
No symbol table info available.
#36 0x0000035c5c96b000 in ?? ()
No symbol table info available.
#37 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (LWP 31020):
#0  0x0000035c616709ad in ?? ()
No symbol table info available.
#1  0x00000000697213a8 in ?? ()
No symbol table info available.
#2  0x0000000000000010 in ?? ()
No symbol table info available.
#3  0x0000035c616708d0 in ?? ()
No symbol table info available.
#4  0x0000035c5fb26d28 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) quit
A debugging session is active.

    Inferior 1 [process 31020] will be killed.

Quit anyway? (y or n) y

And dmesg shows:

[282001.448603] grsec: exec of /bin/bash (/bin/bash -c exec /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /bin/bash[gdb:31020] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:30613] uid/euid:1000/1000 gid/egid:1000/1000

[282001.454206] grsec: exec of /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc (/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:31020] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:30613] uid/euid:1000/1000 gid/egid:1000/1000

[282001.472263] PAX: execution attempt in: <anonymous mapping>, 351e56b2000-351e56b3000 351e56b2000
[282001.472282] PAX: terminating task: /usr/bin/gdb(gdb):31031, uid/euid: 1000/1000, PC: 00000351e56b2000, SP: 0000038f2575db10
[282001.472294] PAX: bytes at PC: cc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
[282001.472322] PAX: bytes at SP-8: 00000351e56b2000 0000038f2575db50 62869905ee6e2300 0000000000000000 0000006fd7b21760 0000006fd74918a0 000000000000792c 0000006fd8e80270 0000006fd8dce0d0 0000038f2575dbe0 0000006fd72f4409 

(NOTE however that there are no errors of any kind on dmesg while executing rust -VV normally, only through gdb)
The above pax terminating task may be explained by this: https://forums.gentoo.org/viewtopic-p-7706666.html#7706666
In other words, it's just a test by gdb to detect stuff, after which I presume it displays that warning.
Perhaps gdb needs this in order to show function names and such? in the stacktrace. I don't see a way to disable this temporarily(tried some paxctl-ng flags on gdb with no effect)

Hmm that must be it, I mean, the No symbol table info available. may always be there regardless, but the ?? () should show the right things, if I manage to relax PaX so that gdb doesn't output that warning. Right? It seems to make sense.

Oh wait, what did I just find? https://wiki.gentoo.org/wiki/Hardened/Debugging

Ok this is stupid:)

/home/zazdxscf $ sudo paxctl -v /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
Password: 
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc]
    RANDEXEC is disabled
    EMUTRAMP is disabled
-----------
zazdxscf@gobaby 2015/10/26 01:12:24 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12306 8 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl -vr /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>

/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc: Text file busy
-----------
zazdxscf@gobaby 2015/10/26 01:12:46 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12307 9 1  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl -v /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc]
    RANDEXEC is disabled
    EMUTRAMP is disabled
-----------
zazdxscf@gobaby 2015/10/26 01:12:53 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12308 10 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl -vr /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>

/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc: Text file busy
-----------
zazdxscf@gobaby 2015/10/26 01:12:56 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12309 11 1  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl -v /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc]
    RANDEXEC is disabled
    EMUTRAMP is disabled
-----------
zazdxscf@gobaby 2015/10/26 01:13:11 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12310 12 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -v /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : not found

-----------
zazdxscf@gobaby 2015/10/26 01:13:21 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12311 13 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -v -l -r /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e-r-

-----------
zazdxscf@gobaby 2015/10/26 01:13:55 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12312 14 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -v -rl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e-r-

-----------
zazdxscf@gobaby 2015/10/26 01:14:23 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12313 15 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vrl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e-r-

-----------
zazdxscf@gobaby 2015/10/26 01:14:27 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12314 16 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vrl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e-r-

-----------
zazdxscf@gobaby 2015/10/26 01:14:41 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12315 17 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ time RUST_BACKTRACE= gdb --args /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc...done.
(gdb) r
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) quit

real    0m6.921s
user    0m0.090s
sys 0m0.077s
-----------
zazdxscf@gobaby 2015/10/26 01:14:58 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12316 18 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vRl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e-R-

-----------
zazdxscf@gobaby 2015/10/26 01:15:03 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12317 19 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ time RUST_BACKTRACE= gdb --args /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc...done.
(gdb) r
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
warning: Cannot call inferior functions, Linux kernel PaX protection forbids return to non-executable pages!
[New LWP 20518]
error: Option 'version' given more than once.
^C
Program received signal SIGINT, Interrupt.
0x000003bd5507b9ad in ?? ()
(gdb) bt full
#0  0x000003bd5507b9ad in ?? ()
No symbol table info available.
#1  0x000000005d12c3a8 in ?? ()
No symbol table info available.
#2  0x0000000000000010 in ?? ()
No symbol table info available.
#3  0x000003bd5507b8d0 in ?? ()
No symbol table info available.
#4  0x000003bd53512d28 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) quit
A debugging session is active.

    Inferior 1 [process 20514] will be killed.

Quit anyway? (y or n) EOF [assumed Y]

real    0m10.945s
user    0m1.257s
sys 0m0.860s
-----------
zazdxscf@gobaby 2015/10/26 01:15:16 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12318 20 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vrsmpl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : pemrs

-----------
zazdxscf@gobaby 2015/10/26 01:15:38 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12319 21 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ time RUST_BACKTRACE= gdb --args /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc...done.
(gdb) r
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) quit

real    0m3.143s
user    0m0.093s
sys 0m0.070s
-----------
zazdxscf@gobaby 2015/10/26 01:15:44 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12320 22 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vRsmpl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : pemRs

-----------
zazdxscf@gobaby 2015/10/26 01:15:50 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12321 23 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ time RUST_BACKTRACE= gdb --args /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
GNU gdb (Gentoo 7.10 vanilla) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc...done.
(gdb) r
Starting program: /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) quit

real    0m5.499s
user    0m0.107s
sys 0m0.053s
-----------
zazdxscf@gobaby 2015/10/26 01:15:58 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12322 24 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vzl /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -----

-----------
zazdxscf@gobaby 2015/10/26 01:16:10 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12323 25 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vzle /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc

Package Name : elfix 0.9.1
Bug Reports  : http://bugs.gentoo.org/
Program Name : paxctl-ng
Description  : Get or set pax flags on an ELF object

Usage        : paxctl-ng -PpEeMmRrSs|-Z|-z [-L|-l] [-v] ELF
             : paxctl-ng -C|-c|-d [-v] ELF
             : paxctl-ng -F|-f [-v] ELF
             : paxctl-ng -v ELF
             : paxctl-ng -L|-l
             : paxctl-ng [-h]

Options      : -P enable PAGEEXEC   -p disable  PAGEEXEC
             : -E enable EMUTRAMP   -e disable  EMUTRAMP
             : -M enable MPROTECT   -m disable  MPROTECT
             : -R enable RANDMMAP   -r disable  RANDMMAP
             : -S enable SEGMEXEC   -s disable  SEGMEXEC
             : -Z all secure settings   -z all default settings
             : -L set only PT_PAX flags -l set only XATTR_PAX flags
             :
             : -C create XATTR_PAX with most secure setting
             : -c create XATTR_PAX all default settings
             : -d delete XATTR_PAX field
             : -F copy PT_PAX to XATTR_PAX
             : -f copy XATTR_PAX to PT_PAX
             : -L when given alone, EXIT_SUCCESS (PT_PAX is supported)
             : -l when given alone, EXIT_SUCCESS (XATTR_PAX is supported)
             : -v view the flags, along with any accompanying operation
             : -h print out this help

Note         :  If both enabling and disabling flags are set, the default - is used

-----------
zazdxscf@gobaby 2015/10/26 01:16:14 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12324 26 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vel /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e---

-----------
zazdxscf@gobaby 2015/10/26 01:16:19 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12325 27 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vc /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : -e---

-----------
zazdxscf@gobaby 2015/10/26 01:16:26 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12326 28 1  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ sudo paxctl-ng -vd /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc:
    open(O_RDWR) failed: cannot change PT_PAX flags
    PT_PAX    : -e---
    XATTR_PAX : not found

-----------
zazdxscf@gobaby 2015/10/26 01:16:28 -bash4.3.42 t:14 j:0 d:5 pp:9467 p:30555
!12327 29 0  4.2.3-hardened-r4-g45b4b78 #2 SMP Tue Oct 20 14:39:20 CEST 2015
/home/zazdxscf
/home/zazdxscf $ 

dmesg:

[284431.059807] grsec: exec of /usr/bin/gdb (gdb --args /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /usr/bin/gdb[bash:20489] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:30555] uid/euid:1000/1000 gid/egid:1000/1000
[284431.081685] grsec: exec of /usr/bin/iconv (iconv -l ) by /usr/bin/iconv[gdb:20490] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20489] uid/euid:1000/1000 gid/egid:1000/1000
[284431.901487] grsec: exec of /bin/bash (/bin/bash -c exec /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /bin/bash[gdb:20491] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20489] uid/euid:1000/1000 gid/egid:1000/1000
[284431.909694] grsec: exec of /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc (/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:20491] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20489] uid/euid:1000/1000 gid/egid:1000/1000
[284431.911916] grsec: Segmentation fault occurred at            (nil) in /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:20491] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20489] uid/euid:1000/1000 gid/egid:1000/1000

...
[284447.041691] grsec: exec of /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc (/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc -VV ) by /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:20514] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20512] uid/euid:1000/1000 gid/egid:1000/1000
[284447.065900] PAX: execution attempt in: <anonymous mapping>, 383691f5000-383691f6000 383691f5000
[284447.065917] PAX: terminating task: /usr/bin/gdb(gdb):20517, uid/euid: 1000/1000, PC: 00000383691f5000, SP: 000003dc07db2ce0
[284447.065927] PAX: bytes at PC: cc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
[284447.065955] PAX: bytes at SP-8: 00000383691f5000 000003dc07db2d20 bbee2a0d7dc77d00 0000000000000000 0000007af2533760 0000007af1ea38a0 0000000000005022 0000007af5893a80 0000007af57d31e0 000003dc07db2db0 0000007af1d06409 
...

[284483.000550] grsec: Segmentation fault occurred at            (nil) in /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:20543] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20541] uid/euid:1000/1000 gid/egid:1000/1000
...

[284496.186188] grsec: Segmentation fault occurred at            (nil) in /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage1/bin/rustc[bash:20566] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/gdb[gdb:20564] uid/euid:1000/1000 gid/egid:1000/1000
...

@ghost
Copy link
Author

ghost commented Oct 26, 2015

I forced it to dump core (by sending it signal 11, twice! apparently once had no effect!)

$ time RUST_BACKTRACE= /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc -VV
error: Option 'version' given more than once.
Segmentation fault (core dumped)

real    0m15.874s
user    0m13.973s
sys 0m1.537s

$ time LD_LIBRARY_PATH=/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/lib coredump_to_backtrace   /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc ./core all >~/logs/rustc2_crash_backtrace_full.log 2>&1 

real    0m26.339s
user    0m24.480s
sys 0m1.377s

EDIT: Skip to rust-gdb output: #29293 (comment)
The following is with plain gdb (as seen with the above command):

2 -ex thread apply all bt full
3 -ex quit --batch
[New LWP 16104]
[New LWP 16105]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000003761d0289ad in pthread_join (threadid=3805789603584, thread_return=0x0) at pthread_join.c:90
90      lll_wait_tid (pd->tid);
[Current thread is 1 (Thread 0x37625654a00 (LWP 16104))]
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc.
Use `info auto-load python-scripts [REGEXP]' to list them.

Thread 2 (Thread 0x3761abcc700 (LWP 16105)):
#0  0x0000037624c3724b in backtrace_alloc (state=0x3762567a000, size=384, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/mmap.c:110
        ret = 0x0
        locked = 1
        pp = 0x375fc349da8
        pagesize = 3805957782718
        asksize = 3805789579536
        page = 0x3761abc70d0
#1  0x0000037624c37558 in backtrace_vector_grow (state=0x3762567a000, size=24, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/mmap.c:236
        pagesize = 4096
        alc = 384
        base = 0x3761abc6a20
        ret = 0x8
#2  0x0000037624c3386c in add_function_range (state=0x3762567a000, ddata=0x37618082970, function=0x375fad7e230, lowpc=3805919552193, highpc=3805919552197, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2175
        p = 0x37624c30173 <read_address+126>
#3  0x0000037624c33a26 in add_function_ranges (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, function=0x375fad7e230, ranges=64, base=0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2233
        low = 553665
        high = 553669
        ranges_buf = {name = 0x37624cc5999 ".debug_ranges", start = 0x3761288e030 <error: Cannot access memory at address 0x3761288e030>, buf = 0x3761288e080 <error: Cannot access memory at address 0x3761288e080>, left = 5360352, is_bigendian = 0, error_callback = 0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data = 0x3761abc75c0, reported_underflow = 0}
#4  0x0000037624c33fd0 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2443
        is_function = 1
        vec = 0x3761abc6bd0
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 1
        function = 0x375fad7e230
        have_lowpc = 0
        ranges = 64
        code = 10
        abbrev = 0x3761a125de8
        i = 4
        have_highpc = 0
#5  0x0000037624c34136 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2481
        fvec = {vec = {base = 0x0, size = 0, alc = 0}, count = 0}
        is_function = 1
        vec = 0x376180829d8
        lowpc = 553600
        highpc = 553957
        highpc_is_relative = 1
        have_ranges = 0
        function = 0x375fad7e208
        have_lowpc = 1
        ranges = 0
        code = 46
        abbrev = 0x3761a126268
        i = 9
        have_highpc = 1
#6  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#7  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#8  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#9  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 1
        abbrev = 0x3761a125cc8
        i = 8
        have_highpc = 0
#10 0x0000037624c342fa in read_function_info (state=0x3762567a000, ddata=0x37618082970, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, u=0x3761a110998, fvec=0x376180829d8, ret_addrs=0x3761abc7210, ret_addrs_count=0x3761abc71f8) at ../../../src/libbacktrace/dwarf.c:2546
        lvec = {vec = {base = 0x3761abc75c0, size = 3805341024256, alc = 3805261574144}, count = 6350376}
        pfvec = 0x376180829d8
        unit_buf = {name = 0x37624cc59a7 ".debug_info", start = 0x376116e0fff <error: Cannot access memory at address 0x376116e0fff>, buf = 0x376119642ff <error: Cannot access memory at address 0x376119642ff>, left = 4604877, is_bigendian = 0, error_callback = 0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data = 0x3761abc75c0, reported_underflow = 0}
        addrs = 0x37624f2d401 <panic_loc53072+33>
        addrs_count = 264598
#11 0x0000037624c347f5 in dwarf_lookup_pc (state=0x3762567a000, ddata=0x37618082970, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, found=0x3761abc72d4) at ../../../src/libbacktrace/dwarf.c:2731
        pfvec = 0x376180829d8
        function_addrs_count = 0
        lhdr = {version = 2, min_insn_len = 1, max_ops_per_insn = 1, line_base = -5, line_range = 14, opcode_base = 13, opcode_lengths = 0x37611dc8e84 <error: Cannot access memory at address 0x37611dc8e84>, dirs_count = 52, dirs = 0x375fed75d20, filenames_count = 225, filenames = 0x375fc430000}
        count = 264598
        entry = 0x3761807f5e0
        u = 0x3761a110998
        new_data = 0
        lines = 0x375fb43b000
        ln = 0x0
        function_addrs = 0x0
        function = 0x37625449048 <vtable34401>
        filename = 0x37612e05a82 <error: Cannot access memory at address 0x37612e05a82>
        lineno = 0
        ret = 886
#12 0x0000037624c34cbf in dwarf_fileline (state=0x3762567a000, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/dwarf.c:2876
        ddata = 0x37618082970
        found = 1
        ret = 0
#13 0x0000037624c35269 in backtrace_pcinfo (state=0x3762567a000, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/fileline.c:176
No locals.
#14 0x0000037624c267c8 in sys::backtrace::tracing::imp::write::trace_fn::he8e626ec33150cf8Npt () at src/libstd/sys/common/gnu/libbacktrace.rs:191
No symbol table info available.
#15 0x000003761cb1b769 in _Unwind_Backtrace (trace=0x37624c26300 <sys::backtrace::tracing::imp::write::trace_fn::he8e626ec33150cf8Npt>, trace_argument=0x3761abc79c8) at /usr/src/debug/sys-devel/gcc-5.2.0/gcc-5.2.0/libgcc/unwind.inc:295
        fs = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 18446744073709551600, offset = -16, exp = 0xfffffffffffffff0 <error: Cannot access memory at address 0xfffffffffffffff0>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}, {loc = {reg = 18446744073709551608, offset = -8, exp = 0xfffffffffffffff8 <error: Cannot access memory at address 0xfffffffffffffff8>}, how = REG_SAVED_OFFSET}, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}}, prev = 0x0, cfa_offset = 16, cfa_reg = 6, cfa_exp = 0x0, cfa_how = CFA_REG_OFFSET}, pc = 0x37622b6c0b4 <rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError>+4>, personality = 0x0, data_align = -8, code_align = 1, retaddr_column = 16, fde_encoding = 27 '\033', lsda_encoding = 255 '\377', saw_z = 1 '\001', signal_frame = 0 '\000', eh_ptr = 0x0}
        context = {reg = {0x0, 0x0, 0x0, 0x3761abc7d58, 0x0, 0x0, 0x3761abc7d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3761abc7d60, 0x3761abc7d68, 0x3761abc7d70, 0x3761abc7d78, 0x3761abc7d88, 0x0}, cfa = 0x3761abc7d90, ra = 0x37622b6c0cb, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0, func = 0x37622b6c0b0 <rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError>>}, flags = 4611686018427387904, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>}
        code = _URC_NO_REASON
#16 0x0000037624c26017 in std::sys::backtrace::tracing::imp::write (w=...) at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:40
No locals.
#17 0x0000037624c2e0dc in fnfn (err=...) at src/libstd/panicking.rs:48
No locals.
#18 std::panicking::log_panic (obj=..., file=..., line=<optimized out>, log_backtrace=true) at src/libstd/panicking.rs:55
        thread = <optimized out>
#19 0x0000037624be445e in std::panicking::on_panic (file=..., line=486, obj=...) at src/libstd/panicking.rs:87
No locals.
#20 std::sys_common::unwind::begin_unwind_inner (msg=..., file_line=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:242
        file = {data_ptr = 0x37622c0f840 <str110789> "src/librustc/session/mod.rs", length = 27}
        line = 486
#21 0x0000037622b6c0cb in rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError> (msg=..., file_line=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:224
No locals.
#22 0x0000037622b9853b in rustc::session::early_error (color=<optimized out>, msg=...) at <std macros>:3
        emitter = <optimized out>
#23 0x00000376251f9a5b in rustc_driver::handle_options::parse_all_options (args=<optimized out>) at src/librustc_driver/lib.rs:724
        all_groups = <optimized out>
#24 0x00000376251ec738 in rustc_driver::handle_options (args=...) at src/librustc_driver/lib.rs:742
        matches = <optimized out>
#25 0x00000376251e9f84 in rustc_driver::run_compiler (args=..., callbacks=...) at src/librustc_driver/lib.rs:117
        sess = <optimized out>
        control = <optimized out>
        sopts = <optimized out>
        matches = <optimized out>
        cfg = <optimized out>
#26 0x00000376251e7d5b in fnfn () at src/librustc_driver/lib.rs:102
No locals.
#27 fnfn () at src/librustc_driver/lib.rs:824
No locals.
#28 fnfn () at src/libstd/thread/mod.rs:271
No locals.
#29 rustc_driver::sys_common::unwind::try::try_fn<closure> (opt_closure=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:156
No locals.
#30 0x0000037624c23529 in __rust_try () at src/libcore/macros.rs:20
No symbol table info available.
#31 0x0000037624c13671 in fnfn (s=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:144
        ep = <optimized out>
        prev = <optimized out>
#32 std::thread::local::LocalKey<T>::with<closure,core::result::Result<(), Box<Any>>> (self=<optimized out>, f=...) at src/libstd/thread/local.rs:217
No locals.
#33 std::sys_common::unwind::try::inner_try (f=0x376251e7b80 <rustc_driver::sys_common::unwind::try::try_fn<closure>>, data=0x3761abcbb70 "") at src/libstd/sys/common/unwind/mod.rs:141
No locals.
#34 0x00000376251e822d in rustc_driver::sys_common::unwind::try<closure> (f=...) at src/libstd/sys/common/unwind/mod.rs:126
No locals.
#35 fnfn () at src/libstd/thread/mod.rs:271
        ptr = <optimized out>
        output = <optimized out>
        their_packet = <optimized out>
#36 rustc_driver::boxed::F.FnBox<A>::call_box (self=0x3761ac24120, args=<optimized out>) at src/liballoc/boxed.rs:521
No locals.
#37 0x0000037624c2ca56 in sys::thread::_$LT$impl$GT$::new::thread_start::h29e6fdb9ec3a6fccfGw () at src/liballoc/boxed.rs:529
No symbol table info available.
#38 0x000003761d02764c in start_thread (arg=0x3761abcc700) at pthread_create.c:334
        __res = <optimized out>
        pd = 0x3761abcc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {3805789603584, 300255978620416884, 4161273009680, 4161273009135, 0, 4161273009680, 300255978599445364, 300252734095455092}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#39 0x00000376248a59bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

Thread 1 (Thread 0x37625654a00 (LWP 16104)):
#0  0x000003761d0289ad in pthread_join (threadid=3805789603584, thread_return=0x0) at pthread_join.c:90
        __tid = 16105
        _buffer = {__routine = 0x3761d0288d0 <cleanup>, __arg = 0x3761abccd28, __canceltype = 448790528, __prev = 0x0}
        oldtype = 0
        pd = 0x3761abcc700
        self = 0x37625654a00
        result = 0
#1  0x0000037624c2cb3f in std::sys::thread::Thread::join (self=...) at src/libstd/sys/unix/thread.rs:160
        ret = <optimized out>
#2  0x00000376251e6bfb in rustc_driver::thread::JoinInner<T>::join (self=<optimized out>) at src/libstd/thread/mod.rs:586
No locals.
#3  rustc_driver::thread::JoinHandle<T>::join (self=...) at src/libstd/thread/mod.rs:616
No locals.
#4  rustc_driver::monitor<closure> (f=...) at src/librustc_driver/lib.rs:824
        cfg = {name = {RUST$ENCODED$ENUM$0$0$0$0$0$0$None = {__0 = {vec = {buf = {ptr = {pointer = {__0 = 0x3c8df331e60 '\035' <repeats 40 times>, "\001"}, _marker = {<No data fields>}}, cap = 8388608}, len = 2097865012304223517}}}}, stack_size = {{RUST$ENUM$DISR = (Some | unknown: 2097865012304223516)}, {RUST$ENUM$DISR = (Some | unknown: 2097865012304223516), __0 = 36}}}
        err = <optimized out>
#5  rustc_driver::run (args=...) at src/librustc_driver/lib.rs:102
No locals.
#6  0x00000376251fa55b in rustc_driver::main () at src/librustc_driver/lib.rs:879
No locals.
#7  0x0000037624c2dc0a in fnfn () at src/libstd/thread/mod.rs:381
No locals.
#8  std::sys_common::unwind::try::try_fn<closure> (opt_closure=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:156
No locals.
#9  0x0000037624c23529 in __rust_try () at src/libcore/macros.rs:20
No symbol table info available.
#10 0x0000037624c2da0b in fnfn (s=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:144
        ep = <optimized out>
        prev = <optimized out>
#11 std::thread::local::LocalKey<T>::with<closure,core::result::Result<(), Box<Any>>> (self=<optimized out>, f=...) at src/libstd/thread/local.rs:217
No locals.
#12 std::sys_common::unwind::try::inner_try (f=0x3761abcc9d0, data=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:141
No locals.
#13 std::sys_common::unwind::try<closure> (f=...) at src/libstd/sys/common/unwind/mod.rs:126
No locals.
#14 std::thread::catch_panic<fn(),()> (f=<optimized out>) at src/libstd/thread/mod.rs:381
        result = <optimized out>
        result = <optimized out>
#15 std::rt::lang_start (main=0x7f7cc32910 <driver::main> "UH\211\345]\351v\376\377\377f\017\037D", argc=<optimized out>, argv=<optimized out>) at src/libstd/rt.rs:55
No locals.
#16 0x00000376247d06a4 in __libc_start_main (main=0x7f7cc32920 <main>, argc=2, argv=0x3c8df332838, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x3c8df332828) at libc-start.c:289
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 384831440184636276, 547554011472, 4161273014320, 0, 0, 384831440344019828, 300268635666477940}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x37625687148, 0x3c8df332820}, data = {prev = 0x0, cleanup = 0x0, canceltype = 627601736}}}
        not_first_call = <optimized out>
#17 0x0000007f7cc32809 in _start () at ../sysdeps/x86_64/start.S:108
No locals.

@ghost
Copy link
Author

ghost commented Oct 26, 2015

Here's with rust-gdb

EDIT: Here's raw (since the below is hard to horizontally scroll through): https://gist.githubusercontent.com/zazdxscf/56623981eb212b4066e5/raw/4fdfaa3acafbfc87568f3a554b930c64c687b258/gistfile1.txt

$ time rust-gdb /home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc --core ./core --quiet -ex 'thread apply all bt full' -ex quit --batch >~/logs/rustc2_crash_backtrace_via_rust-gdb_full.log 2>&1 

real    0m25.916s
user    0m24.437s
sys 0m1.357s

$ cat ~/logs/rustc2_crash_backtrace_via_rust-gdb_full.log

[New LWP 16104]
[New LWP 16105]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000003761d0289ad in pthread_join (threadid=3805789603584, thread_return=0x0) at pthread_join.c:90
90      lll_wait_tid (pd->tid);
[Current thread is 1 (Thread 0x37625654a00 (LWP 16104))]

Thread 2 (Thread 0x3761abcc700 (LWP 16105)):
#0  0x0000037624c3724b in backtrace_alloc (state=0x3762567a000, size=384, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/mmap.c:110
        ret = 0x0
        locked = 1
        pp = 0x375fc349da8
        pagesize = 3805957782718
        asksize = 3805789579536
        page = 0x3761abc70d0
#1  0x0000037624c37558 in backtrace_vector_grow (state=0x3762567a000, size=24, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/mmap.c:236
        pagesize = 4096
        alc = 384
        base = 0x3761abc6a20
        ret = 0x8
#2  0x0000037624c3386c in add_function_range (state=0x3762567a000, ddata=0x37618082970, function=0x375fad7e230, lowpc=3805919552193, highpc=3805919552197, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2175
        p = 0x37624c30173 <read_address+126>
#3  0x0000037624c33a26 in add_function_ranges (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, function=0x375fad7e230, ranges=64, base=0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2233
        low = 553665
        high = 553669
        ranges_buf = dwarf_buf = {name = 0x37624cc5999 ".debug_ranges", start = 0x3761288e030 <error: Cannot access memory at address 0x3761288e030>, buf = 0x3761288e080 <error: Cannot access memory at address 0x3761288e080>, left = 5360352, is_bigendian = 0, error_callback = 0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data = 0x3761abc75c0, reported_underflow = 0}
#4  0x0000037624c33fd0 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x3761abc6bd0) at ../../../src/libbacktrace/dwarf.c:2443
        is_function = 1
        vec = 0x3761abc6bd0
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 1
        function = 0x375fad7e230
        have_lowpc = 0
        ranges = 64
        code = 10
        abbrev = 0x3761a125de8
        i = 4
        have_highpc = 0
#5  0x0000037624c34136 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2481
        fvec = function_vector = {vec = backtrace_vector = {base = 0x0, size = 0, alc = 0}, count = 0}
        is_function = 1
        vec = 0x376180829d8
        lowpc = 553600
        highpc = 553957
        highpc_is_relative = 1
        have_ranges = 0
        function = 0x375fad7e208
        have_lowpc = 1
        ranges = 0
        code = 46
        abbrev = 0x3761a126268
        i = 9
        have_highpc = 1
#6  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#7  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#8  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 2
        abbrev = 0x3761a125ce8
        i = 1
        have_highpc = 0
#9  0x0000037624c340c4 in read_function_entry (state=0x3762567a000, ddata=0x37618082970, u=0x3761a110998, base=0, unit_buf=0x3761abc70d0, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, vec_function=0x376180829d8, vec_inlined=0x376180829d8) at ../../../src/libbacktrace/dwarf.c:2467
        is_function = 0
        vec = 0x376180829d8
        lowpc = 0
        highpc = 0
        highpc_is_relative = 0
        have_ranges = 0
        function = 0x0
        have_lowpc = 0
        ranges = 0
        code = 1
        abbrev = 0x3761a125cc8
        i = 8
        have_highpc = 0
#10 0x0000037624c342fa in read_function_info (state=0x3762567a000, ddata=0x37618082970, lhdr=0x3761abc71b0, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, u=0x3761a110998, fvec=0x376180829d8, ret_addrs=0x3761abc7210, ret_addrs_count=0x3761abc71f8) at ../../../src/libbacktrace/dwarf.c:2546
        lvec = function_vector = {vec = backtrace_vector = {base = 0x3761abc75c0, size = 3805341024256, alc = 3805261574144}, count = 6350376}
        pfvec = 0x376180829d8
        unit_buf = dwarf_buf = {name = 0x37624cc59a7 ".debug_info", start = 0x376116e0fff <error: Cannot access memory at address 0x376116e0fff>, buf = 0x376119642ff <error: Cannot access memory at address 0x376119642ff>, left = 4604877, is_bigendian = 0, error_callback = 0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data = 0x3761abc75c0, reported_underflow = 0}
        addrs = 0x37624f2d401 <panic_loc53072+33>
        addrs_count = 264598
#11 0x0000037624c347f5 in dwarf_lookup_pc (state=0x3762567a000, ddata=0x37618082970, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0, found=0x3761abc72d4) at ../../../src/libbacktrace/dwarf.c:2731
        pfvec = 0x376180829d8
        function_addrs_count = 0
        lhdr = line_header = {version = 2, min_insn_len = 1, max_ops_per_insn = 1, line_base = -5, line_range = 14, opcode_base = 13, opcode_lengths = 0x37611dc8e84 <error: Cannot access memory at address 0x37611dc8e84>, dirs_count = 52, dirs = 0x375fed75d20, filenames_count = 225, filenames = 0x375fc430000}
        count = 264598
        entry = 0x3761807f5e0
        u = 0x3761a110998
        new_data = 0
        lines = 0x375fb43b000
        ln = 0x0
        function_addrs = 0x0
        function = 0x37625449048 <vtable34401>
        filename = 0x37612e05a82 <error: Cannot access memory at address 0x37612e05a82>
        lineno = 0
        ret = 886
#12 0x0000037624c34cbf in dwarf_fileline (state=0x3762567a000, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/dwarf.c:2876
        ddata = 0x37618082970
        found = 1
        ret = 0
#13 0x0000037624c35269 in backtrace_pcinfo (state=0x3762567a000, pc=3805923426506, callback=0x37624c25e70 <sys_common::gnu::libbacktrace::print::pcinfo_cb::ha768a72a949356196et>, error_callback=0x37624c25e60 <sys_common::gnu::libbacktrace::print::error_cb::h2189b2356f8e6008het>, data=0x3761abc75c0) at ../../../src/libbacktrace/fileline.c:176
No locals.
#14 0x0000037624c267c8 in sys::backtrace::tracing::imp::write::trace_fn::he8e626ec33150cf8Npt () at src/libstd/sys/common/gnu/libbacktrace.rs:191
No symbol table info available.
#15 0x000003761cb1b769 in _Unwind_Backtrace (trace=0x37624c26300 <sys::backtrace::tracing::imp::write::trace_fn::he8e626ec33150cf8Npt>, trace_argument=0x3761abc79c8) at /usr/src/debug/sys-devel/gcc-5.2.0/gcc-5.2.0/libgcc/unwind.inc:295
Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
        fs = {regs = frame_state_reg_info = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 18446744073709551600, offset = -16, exp = 0xfffffffffffffff0 <error: Cannot access memory at address 0xfffffffffffffff0>}, how = REG_SAVED_OFFSET}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 18446744073709551608, offset = -8, exp = 0xfffffffffffffff8 <error: Cannot access memory at address 0xfffffffffffffff8>}, how = REG_SAVED_OFFSET}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
Python Exception <type 'exceptions.TypeError'> Type is not a structure, union, or enum type.: 
, {loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED}}, prev = 0x0, cfa_offset = 16, cfa_reg = 6, cfa_exp = 0x0, cfa_how = CFA_REG_OFFSET}, pc = 0x37622b6c0b4 <rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError>+4>, personality = 0x0, data_align = -8, code_align = 1, retaddr_column = 16, fde_encoding = 27 '\033', lsda_encoding = 255 '\377', saw_z = 1 '\001', signal_frame = 0 '\000', eh_ptr = 0x0}
        context = _Unwind_Context = {reg = {0x0, 0x0, 0x0, 0x3761abc7d58, 0x0, 0x0, 0x3761abc7d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3761abc7d60, 0x3761abc7d68, 0x3761abc7d70, 0x3761abc7d78, 0x3761abc7d88, 0x0}, cfa = 0x3761abc7d90, ra = 0x37622b6c0cb, lsda = 0x0, bases = dwarf_eh_bases = {tbase = 0x0, dbase = 0x0, func = 0x37622b6c0b0 <rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError>>}, flags = 4611686018427387904, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>}
        code = _URC_NO_REASON
#16 0x0000037624c26017 in std::sys::backtrace::tracing::imp::write (w=&mut Write) at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:40
No locals.
#17 0x0000037624c2e0dc in fnfn (err=...) at src/libstd/panicking.rs:48
No locals.
#18 std::panicking::log_panic (obj=..., file=..., line=<optimized out>, log_backtrace=true) at src/libstd/panicking.rs:55
        thread = <optimized out>
#19 0x0000037624be445e in std::panicking::on_panic (file="src/librustc/session/mod.rs", line=486, obj=...) at src/libstd/panicking.rs:87
No locals.
#20 std::sys_common::unwind::begin_unwind_inner (msg=..., file_line=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:242
        file = "src/librustc/session/mod.rs"
        line = 486
#21 0x0000037622b6c0cb in rustc::sys_common::unwind::begin_unwind<syntax::diagnostic::FatalError> (msg=..., file_line=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:224
No locals.
#22 0x0000037622b9853b in rustc::session::early_error (color=<optimized out>, msg=...) at <std macros>:3
        emitter = <optimized out>
#23 0x00000376251f9a5b in rustc_driver::handle_options::parse_all_options (args=<optimized out>) at src/librustc_driver/lib.rs:724
        all_groups = <optimized out>
#24 0x00000376251ec738 in rustc_driver::handle_options (args=...) at src/librustc_driver/lib.rs:742
Python Exception <type 'exceptions.MemoryError'> : 
        matches = <optimized out>
#25 0x00000376251e9f84 in rustc_driver::run_compiler (args=&[collections::string::String](len: 3805957742176), callbacks=&mut CompilerCalls) at src/librustc_driver/lib.rs:117
        sess = <optimized out>
        control = <optimized out>
        sopts = <optimized out>
        matches = <optimized out>
        cfg = <optimized out>
#26 0x00000376251e7d5b in fnfn () at src/librustc_driver/lib.rs:102
No locals.
#27 fnfn () at src/librustc_driver/lib.rs:824
No locals.
#28 fnfn () at src/libstd/thread/mod.rs:271
No locals.
#29 rustc_driver::sys_common::unwind::try::try_fn<closure> (opt_closure=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:156
No locals.
#30 0x0000037624c23529 in __rust_try () at src/libcore/macros.rs:20
No symbol table info available.
#31 0x0000037624c13671 in fnfn (s=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:144
        ep = <optimized out>
        prev = <optimized out>
#32 std::thread::local::LocalKey<T>::with<closure,core::result::Result<(), Box<Any>>> (self=<optimized out>, f=...) at src/libstd/thread/local.rs:217
No locals.
#33 std::sys_common::unwind::try::inner_try (f=0x376251e7b80 <rustc_driver::sys_common::unwind::try::try_fn<closure>>, data=0x3761abcbb70 "") at src/libstd/sys/common/unwind/mod.rs:141
No locals.
#34 0x00000376251e822d in rustc_driver::sys_common::unwind::try<closure> (f=...) at src/libstd/sys/common/unwind/mod.rs:126
No locals.
#35 fnfn () at src/libstd/thread/mod.rs:271
        ptr = <optimized out>
        output = <optimized out>
        their_packet = <optimized out>
#36 rustc_driver::boxed::F.FnBox<A>::call_box (self=0x3761ac24120, args=<optimized out>) at src/liballoc/boxed.rs:521
No locals.
#37 0x0000037624c2ca56 in sys::thread::_$LT$impl$GT$::new::thread_start::h29e6fdb9ec3a6fccfGw () at src/liballoc/boxed.rs:529
No symbol table info available.
#38 0x000003761d02764c in start_thread (arg=0x3761abcc700) at pthread_create.c:334
Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
        __res = <optimized out>
        pd = 0x3761abcc700
        now = <optimized out>
        unwind_buf = pthread_unwind_buf = {cancel_jmp_buf = {{jmp_buf = {3805789603584, 300255978620416884, 4161273009680, 4161273009135, 0, 4161273009680, 300255978599445364, 300252734095455092}, mask_was_saved = 0}}Python Exception <type 'exceptions.TypeError'> Invalid lookup for a field not contained in the value.: 
, priv = {pad = {0x0, 0x0, 0x0, 0x0}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#39 0x00000376248a59bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

Thread 1 (Thread 0x37625654a00 (LWP 16104)):
#0  0x000003761d0289ad in pthread_join (threadid=3805789603584, thread_return=0x0) at pthread_join.c:90
        __tid = 16105
        _buffer = _pthread_cleanup_buffer = {__routine = 0x3761d0288d0 <cleanup>, __arg = 0x3761abccd28, __canceltype = 448790528, __prev = 0x0}
        oldtype = 0
        pd = 0x3761abcc700
        self = 0x37625654a00
        result = 0
#1  0x0000037624c2cb3f in std::sys::thread::Thread::join (self=...) at src/libstd/sys/unix/thread.rs:160
        ret = <optimized out>
#2  0x00000376251e6bfb in rustc_driver::thread::JoinInner<T>::join (self=<optimized out>) at src/libstd/thread/mod.rs:586
No locals.
#3  rustc_driver::thread::JoinHandle<T>::join (self=...) at src/libstd/thread/mod.rs:616
No locals.
#4  rustc_driver::monitor<closure> (f=...) at src/librustc_driver/lib.rs:824
Python Exception <class 'gdb.error'> Cannot convert value to int.: 
Python Exception <type 'exceptions.OverflowError'> signed integer is greater than maximum: 
        cfg = Builder = {name = {RUST$ENCODED$ENUM$0$0$0$0$0$0$None = Some = {}}Python Exception <type 'exceptions.TypeError'> Could not convert Python object: None.: 
, stack_size = {(Some | unknown: 2097865012304223516), Some = {36}}}
        err = <optimized out>
#5  rustc_driver::run (args=...) at src/librustc_driver/lib.rs:102
No locals.
#6  0x00000376251fa55b in rustc_driver::main () at src/librustc_driver/lib.rs:879
No locals.
#7  0x0000037624c2dc0a in fnfn () at src/libstd/thread/mod.rs:381
No locals.
#8  std::sys_common::unwind::try::try_fn<closure> (opt_closure=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:156
No locals.
#9  0x0000037624c23529 in __rust_try () at src/libcore/macros.rs:20
No symbol table info available.
#10 0x0000037624c2da0b in fnfn (s=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:144
        ep = <optimized out>
        prev = <optimized out>
#11 std::thread::local::LocalKey<T>::with<closure,core::result::Result<(), Box<Any>>> (self=<optimized out>, f=...) at src/libstd/thread/local.rs:217
No locals.
#12 std::sys_common::unwind::try::inner_try (f=0x3761abcc9d0, data=<optimized out>) at src/libstd/sys/common/unwind/mod.rs:141
No locals.
#13 std::sys_common::unwind::try<closure> (f=...) at src/libstd/sys/common/unwind/mod.rs:126
No locals.
#14 std::thread::catch_panic<fn(),()> (f=<optimized out>) at src/libstd/thread/mod.rs:381
        result = <optimized out>
        result = <optimized out>
#15 std::rt::lang_start (main=0x7f7cc32910 <driver::main> "UH\211\345]\351v\376\377\377f\017\037D", argc=<optimized out>, argv=<optimized out>) at src/libstd/rt.rs:55
No locals.
#16 0x00000376247d06a4 in __libc_start_main (main=0x7f7cc32920 <main>, argc=2, argv=0x3c8df332838, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x3c8df332828) at libc-start.c:289
Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
        result = <optimized out>
        unwind_buf = pthread_unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 384831440184636276, 547554011472, 4161273014320, 0, 0, 384831440344019828, 300268635666477940}, mask_was_saved = 0}}Python Exception <type 'exceptions.TypeError'> Invalid lookup for a field not contained in the value.: 
, priv = {pad = {0x0, 0x0, 0x37625687148, 0x3c8df332820}Python Exception <type 'exceptions.AttributeError'> 'NoneType' object has no attribute 'startswith': 
, data = {prev = 0x0, cleanup = 0x0, canceltype = 627601736}}}
        not_first_call = <optimized out>
#17 0x0000007f7cc32809 in _start () at ../sysdeps/x86_64/start.S:108
No locals.

@ghost
Copy link
Author

ghost commented Oct 30, 2015

I guess the question is: why take 10 times more time when RUST_BACKTRACE is set in the environment AND not display any backtrace? (or, just forget the AND part)

$ unset RUST_BACKTRACE

$ which rustc
/home/zazdxscf/build/1nonpkgs/rust/rust/x86_64-unknown-linux-gnu/stage2/bin/rustc

$ time  rustc -VV
error: Option 'version' given more than once.

real    0m0.055s
user    0m0.027s
sys 0m0.023s

$ time RUST_BACKTRACE=1 rustc -VV
error: Option 'version' given more than once.

real    0m0.619s
user    0m0.563s
sys 0m0.050s

$ time RUST_BACKTRACE=1 rustc -Vv
rustc 1.6.0-dev (cc8d398e2 2015-10-30)
binary: rustc
commit-hash: cc8d398e28b6b1918ef85479c2d040dfd0fe582d
commit-date: 2015-10-30
host: x86_64-unknown-linux-gnu
release: 1.6.0-dev

real    0m0.055s
user    0m0.027s
sys 0m0.027s

Can someone confirm, by doing the above steps, that their rustc takes like 10 times more (even though only half a second total) when RUST_BACKTRACE is set? Please. So that I know whether it's my other unusual ./configure options causing this or not.

Apparently only when compiling with --enable-debuginfo then, only stage2 rustc (whose files inside /lib directory(excluding rustlib/ directory) is 100meg bigger than stage1 - that is, like 180meg vs 80meg) does take like 45 seconds to complete on errors. Anything else (any stage0 or stage1; or a stage2 with --disable-debuginfo) takes 0.6sec on errors when RUST_BACKTRACE is set, and 0.05 sec when it's not set).

But, even with --enable-debuginfo, stage2 rustc will still do 0.6sec if the stage2 compilation is not finished(eg. rust is still being compiled, but stage2 rustc is already available at some point during this compilation). So when compilation is all done, stage2 rustc (with debuginfo) will always take around 44sec on errors when RUST_BACKTRACE is set.

In other words, the 10 times delay that occurs when RUST_BACKTRACE is set is exacerbated by having built rust with --enable-debuginfo (so it now takes like 44 seconds instead of 0.6 seconds)

This also means that stage1 is not built with --enable-debuginfo yet. Only stage2 is. (not only because stage2's lib files weigh 100 meg more than stage1's, but also because stage1 takes 0.6s and stage2 takes 44s - see above). Although it looks like it should be: 92276dc#diff-f40c1cf0c004abfc517b43348f11ab48R137

@ghost ghost changed the title rustc hangs for 20-45 seconds on errors(of any kind) when RUST_BACKTRACE is set rustc(with --enable-debuginfo) hangs for 20-45 seconds on errors(of any kind) when RUST_BACKTRACE is set Oct 31, 2015
@ghost
Copy link
Author

ghost commented Nov 1, 2015

Closing, as I've decided, as a lifestyle choice, to avoid focusing on bugs(so passionately) and rather just work around them, mainly because that is just wasting everybody's time needlessly.

For current issue, the workaround is: either not use --enable-debuginfo when compiling rust, or/and make sure RUST_BACKTRACE is unset in environment.

Peace out.

@ghost ghost closed this as completed Nov 1, 2015
@arielb1 arielb1 reopened this Nov 1, 2015
@arielb1
Copy link
Contributor

arielb1 commented Nov 1, 2015

I can confirm this and would like to see it fixed.

@wthrowe
Copy link
Contributor

wthrowe commented Nov 5, 2015

I did a bit of testing here. As suggested by the previous posts it seems to be spending all its time in src/libbacktrace/mmap.c:backtrace_alloc (traversing the linked list, apparently). I removed the statements suppressing the backtrace output and found that most of the time is being spent between these two lines:

   4:     0x7f864c947cea - sys_common::unwind::begin_unwind::begin_unwind::h127295331395719567
                        at src/libstd/sys/common/unwind/mod.rs:224

(with another reasonable chunk two frames further out).

The basic cause of this seems to be that backtrace_alloc is slow and is being called a lot of times (roughly 300,000). On my machine, RUST_BACKTRACE unset takes under 0.1 second and RUST_BACKTRACE=1 takes about a minute. If I swap out the backtrace allocators for malloc and similar, that drops to around 1.2 seconds (this isn't a solution because it's not safe to call malloc in some places you'd want a backtrace). At that point we seem to be dominated by the processing in src/libbacktrace/dwarf.c:read_function_entry.

@alexcrichton
Copy link
Member

@wthrowe that's... weird! Do you know why backtrace wants to allocate so much memory? Is something accidentally being called a huge number of times?

@wthrowe
Copy link
Contributor

wthrowe commented Nov 6, 2015

read_function_entry is being called a huge number of times, mostly by recursively calling itself apparently. I'm not quite sure what that is doing, but I think it may be examining every symbol in...something? I don't have a good feeling for whether what is going on there is reasonable or not.

In the experimenting direction, I tried moving the struct function defined near the top of read_function_entry onto the stack and storing the one in struct function_addrs by value. This reduced the number of calls to the allocator by a factor of three but apparently somehow made them even slower so the runtime actually increased (still apparently allocation-dominated). My current hypothesis is that the backtrace allocator is getting into some pathological state. I will investigate further.

@wthrowe
Copy link
Contributor

wthrowe commented Nov 6, 2015

Hypothesis confirmed. The linked list of freed chunks of memory available for reuse accumulates a huge number of chunks that are too small to fulfill most allocation requests, so some requests have to check tens of thousands of chunks for sufficient space before giving up and mmaping more memory.

As an experiment, I modified the allocator to only check the first (most recently added) 10 chunks on the list. This dropped the runtime to a similar value to my previous malloc tests (a bit over a second). After 100,000 allocations the modified version has a resident memory size of 67.5 MB, as opposed to the original version which uses 66.6 MB, so while this does decrease them memory efficiency of the allocator the effect is not large. I'm not sure if this is the correct path to fixing the problem; it may make more sense to add logic to the memory freeing code to drop small chunks (but the definition of "small" is not obvious).

I think I'll file an upstream bug with gcc (which appears to be the main repository for libbacktrace) to see if they have any ideas.

@wthrowe
Copy link
Contributor

wthrowe commented Nov 6, 2015

Upstream bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68239

@alexcrichton
Copy link
Member

Impressive investigation @wthrowe! Thanks for filing the upstream bug!

bors added a commit that referenced this issue Apr 2, 2016
…hton

allow RUST_BACKTRACE=0 to act as if unset

**UPDATE:** `RUST_BACKTRACE=0` to act as if the env. var is unset! (now `0` is what `disabled` was for, below)

When RUST_BACKTRACE is set to "disabled" then this acts as if the env. var is unset. So, either make sure `RUST_BACKTRACE` is not set OR set it to `disabled` to achieve the same effect.

Sample usage:

```bash
$ rustc -o /tmp/a.out -- <(echo 'fn main(){ panic!() }') && RUST_BACKTRACE=disabled /tmp/a.out
!! executing '/home/zazdxscf/build/1nonpkgs/rust/rust//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-o /tmp/a.out -- /dev/fd/63'
thread '<main>' panicked at 'explicit panic', /dev/fd/63:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

$ rustc -o /tmp/a.out -- <(echo 'fn main(){ panic!() }') && RUST_BACKTRACE=1 /tmp/a.out
!! executing '/home/zazdxscf/build/1nonpkgs/rust/rust//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-o /tmp/a.out -- /dev/fd/63'
thread '<main>' panicked at 'explicit panic', /dev/fd/63:1
stack backtrace:
   1:     0x55709e8148c0 - sys::backtrace::tracing::imp::write::h140f24a0cfc189b98Ru
   2:     0x55709e816a5b - panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::closure.45165
   3:     0x55709e8166e8 - panicking::default_hook::hed419823688cb82aXoA
   4:     0x55709e810fff - sys_common::unwind::begin_unwind_inner::hbb9642f6e212d56fmHt
   5:     0x55709e810513 - sys_common::unwind::begin_unwind::h16232867470678019594
   6:     0x55709e810489 - main::hb524f9576270962feaa
   7:     0x55709e816314 - sys_common::unwind::try::try_fn::h1274188004693518534
   8:     0x55709e813dfb - __rust_try
   9:     0x55709e815dab - rt::lang_start::h712b1cd650781872ahA
  10:     0x55709e810679 - main
  11:     0x7efd1026859f - __libc_start_main
  12:     0x55709e810348 - _start
  13:                0x0 - <unknown>
```

Some programs(eg. [vim's syntactic](https://github.com/scrooloose/syntastic) used by [rust.vim](https://github.com/rust-lang/rust.vim)) cannot unset the env. var RUST_BACKTRACE if it's already set(eg. in .bashrc) but [they can set it to some value](https://github.com/respeccing/gentooskyline/blob/cb5533e1598f871d3fdf7c3d8248ce767b5b9360/system/Z575/OSes/gentoo/on_baremetal/filesystem_now/gentoo/home/zazdxscf/build/1nonpkgs/rust.vim/upd#L17), and I needed to ensure the env. var is unset in order to avoid this issue: #29293

**EDIT:** Sample usage 2:

```bash
$ export RUST_BACKTRACE=1

$ rustc -o /tmp/a.out -- <(echo 'fn main(){ panic!() }') && /tmp/a.out
!! executing '/home/zazdxscf/build/1nonpkgs/rust/rust//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-o /tmp/a.out -- /dev/fd/63'
thread '<main>' panicked at 'explicit panic', /dev/fd/63:1
stack backtrace:
   1:     0x55c2696738c0 - sys::backtrace::tracing::imp::write::h140f24a0cfc189b98Ru
   2:     0x55c269675a5b - panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::closure.45165
   3:     0x55c2696756e8 - panicking::default_hook::hed419823688cb82aXoA
   4:     0x55c26966ffff - sys_common::unwind::begin_unwind_inner::hbb9642f6e212d56fmHt
   5:     0x55c26966f513 - sys_common::unwind::begin_unwind::h16023941661074805588
   6:     0x55c26966f489 - main::hb524f9576270962feaa
   7:     0x55c269675314 - sys_common::unwind::try::try_fn::h1274188004693518534
   8:     0x55c269672dfb - __rust_try
   9:     0x55c269674dab - rt::lang_start::h712b1cd650781872ahA
  10:     0x55c26966f679 - main
  11:     0x7f593d58459f - __libc_start_main
  12:     0x55c26966f348 - _start
  13:                0x0 - <unknown>

$ rustc -o /tmp/a.out -- <(echo 'fn main(){ panic!() }') && RUST_BACKTRACE=disabled /tmp/a.out
!! executing '/home/zazdxscf/build/1nonpkgs/rust/rust//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-o /tmp/a.out -- /dev/fd/63'
thread '<main>' panicked at 'explicit panic', /dev/fd/63:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

```
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jan 11, 2017
In rust-lang#37280 we enabled line number debugging information in release artifacts,
primarily to close out rust-lang#36452 where debugging information was critical for MSVC
builds of Rust to be useful in production. This commit, however, apparently had
some unfortunate side effects.

Namely it was noticed in rust-lang#37477 that if `RUST_BACKTRACE=1` was set then any
compiler error would take a very long time for the compiler to exit. The cause
of the problem here was somewhat deep:

* For all compiler errors, the compiler will `panic!` with a known value. This
  tears down the main compiler thread and allows cleaning up all the various
  resources. By default, however, this panic output is suppressed for "normal"
  compiler errors.
* When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a
  backtrace.
* The libbacktrace library hits a pathological case where it spends a very long
  time in its custom allocation function, `backtrace_alloc`, because the
  compiler has so much debugging information. More information about this can be
  found in rust-lang#29293 with a summary at the end of rust-lang#37477.

To solve this problem this commit simply removes debuginfo from the compiler but
not from the standard library. This should allow us to keep rust-lang#36452 closed while
also closing rust-lang#37477. I've measured the difference to be orders of magnitude
faster than it was before, so we should see a much quicker time-to-exit after a
compile error when `RUST_BACKTRACE=1` is set.

Closes rust-lang#37477
Closes rust-lang#37571
bors added a commit that referenced this issue Jan 11, 2017
rustbuild: Don't enable debuginfo in rustc

In #37280 we enabled line number debugging information in release artifacts,
primarily to close out #36452 where debugging information was critical for MSVC
builds of Rust to be useful in production. This commit, however, apparently had
some unfortunate side effects.

Namely it was noticed in #37477 that if `RUST_BACKTRACE=1` was set then any
compiler error would take a very long time for the compiler to exit. The cause
of the problem here was somewhat deep:

* For all compiler errors, the compiler will `panic!` with a known value. This
  tears down the main compiler thread and allows cleaning up all the various
  resources. By default, however, this panic output is suppressed for "normal"
  compiler errors.
* When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a
  backtrace.
* The libbacktrace library hits a pathological case where it spends a very long
  time in its custom allocation function, `backtrace_alloc`, because the
  compiler has so much debugging information. More information about this can be
  found in #29293 with a summary at the end of #37477.

To solve this problem this commit simply removes debuginfo from the compiler but
not from the standard library. This should allow us to keep #36452 closed while
also closing #37477. I've measured the difference to be orders of magnitude
faster than it was before, so we should see a much quicker time-to-exit after a
compile error when `RUST_BACKTRACE=1` is set.

Closes #37477
Closes #37571
brson pushed a commit to brson/rust that referenced this issue Jan 17, 2017
In rust-lang#37280 we enabled line number debugging information in release artifacts,
primarily to close out rust-lang#36452 where debugging information was critical for MSVC
builds of Rust to be useful in production. This commit, however, apparently had
some unfortunate side effects.

Namely it was noticed in rust-lang#37477 that if `RUST_BACKTRACE=1` was set then any
compiler error would take a very long time for the compiler to exit. The cause
of the problem here was somewhat deep:

* For all compiler errors, the compiler will `panic!` with a known value. This
  tears down the main compiler thread and allows cleaning up all the various
  resources. By default, however, this panic output is suppressed for "normal"
  compiler errors.
* When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a
  backtrace.
* The libbacktrace library hits a pathological case where it spends a very long
  time in its custom allocation function, `backtrace_alloc`, because the
  compiler has so much debugging information. More information about this can be
  found in rust-lang#29293 with a summary at the end of rust-lang#37477.

To solve this problem this commit simply removes debuginfo from the compiler but
not from the standard library. This should allow us to keep rust-lang#36452 closed while
also closing rust-lang#37477. I've measured the difference to be orders of magnitude
faster than it was before, so we should see a much quicker time-to-exit after a
compile error when `RUST_BACKTRACE=1` is set.

Closes rust-lang#37477
Closes rust-lang#37571
@steveklabnik steveklabnik added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Mar 24, 2017
@ghost
Copy link
Author

ghost commented Aug 20, 2017

With RUST_BACKTRACE=1 (for a rustc compiled with debuginfo = true), it currently takes, for me, 2min 20sec with 100% CPU usage (one full core(@2.4Ghz fixed)), out of 4 cores) to continue from a panic:

$ time rustc -vv
!! LD_LIBRARY_PATH=/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:
!! Executing '/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-vv'
error: Option 'verbose' given more than once.


real	2m20.208s
user	2m19.662s
sys	0m0.441s

That's 3115 times more time than it takes when RUST_BACKTRACE=0 :) (2m20.208s vs 0m0.045s)

rustc version used
$ rustc -vV
!! LD_LIBRARY_PATH=/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:
!! Executing '/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-vV'
rustc 1.21.0-dev (7f397bdb0 2017-08-19)
binary: rustc
commit-hash: 7f397bdb062fe13a4707219a2f32486c5294f642
commit-date: 2017-08-19
host: x86_64-unknown-linux-gnu
release: 1.21.0-dev
LLVM version: 4.0
`config.toml` used to compile rustc
[llvm]
optimize = true
release-debuginfo = false
assertions = false
ccache = "/usr/bin/ccache"
static-libstdcpp = false
ninja = true
targets = "X86"
link-jobs = 4
[build]
build = "x86_64-unknown-linux-gnu"
host = ["x86_64-unknown-linux-gnu"]
target = ["x86_64-unknown-linux-gnu"]
cargo = "/home/xftroxgpx/.cargo/bin/cargo"
rustc = "/home/xftroxgpx/.cargo/bin/rustc"
docs = true
compiler-docs = true
submodules = true
locked-deps = false
full-bootstrap = false
extended = false
verbose = 0
sanitizers = false
profiler = false
openssl-static = false
low-priority = true
[install]
[rust]
optimize = true
codegen-units = 1 #We currently have the capability to do multiple codegen units in parallel. Unfortunately, one drawback of using this functionality is that using multiple codegen units loses optimization opportunities, like inlining, between the units. src: https://internals.rust-lang.org/t/towards-a-second-edition-of-the-compiler/5582
debug-assertions = false
debuginfo = true
debuginfo-lines = true
debuginfo-only-std = false
debug-jemalloc = false
backtrace = true
channel = "dev"
rpath = true
optimize-tests = false
debuginfo-tests = false
codegen-tests = false
ignore-git = false
[target.x86_64-unknown-linux-gnu]
[dist]
src-tarball = false

Rust nightly (via rustup) takes only under 1 second:

$ time ~/.cargo/bin/rustc -vv
error: Option 'verbose' given more than once.


real	0m1.161s
user	0m0.773s
sys	0m0.082s

$ time ~/.cargo/bin/rustc -vv
error: Option 'verbose' given more than once.


real	0m0.814s
user	0m0.775s
sys	0m0.040s
$ time ~/.cargo/bin/rustc -vv
error: Option 'verbose' given more than once.


real	0m0.790s
user	0m0.754s
sys	0m0.038s
time with RUST_BACKTRACE=0
$ time RUST_BACKTRACE=0 rustc -vv
!! LD_LIBRARY_PATH=/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:
!! Executing '/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-vv'
error: Option 'verbose' given more than once.


real	0m0.045s
user	0m0.033s
sys	0m0.012s
$ time RUST_BACKTRACE=0 rustc -vv
!! LD_LIBRARY_PATH=/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/xftroxgpx/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:
!! Executing '/home/xftroxgpx/build/2nonpkgs/rust.stuff/rust/rust/build//x86_64-unknown-linux-gnu/stage2/bin//rustc' with args: '-vv'
error: Option 'verbose' given more than once.


real	0m0.046s
user	0m0.035s
sys	0m0.011s

$ time RUST_BACKTRACE=0 ~/.cargo/bin/rustc -vv
error: Option 'verbose' given more than once.


real	0m0.146s
user	0m0.131s
sys	0m0.017s
$ time RUST_BACKTRACE=0 ~/.cargo/bin/rustc -vv
error: Option 'verbose' given more than once.


real	0m0.138s
user	0m0.120s
sys	0m0.020s

(sidenote: looks like nightly is "significantly" slower.)

alexcrichton added a commit to alexcrichton/rust that referenced this issue Oct 25, 2017
This is sort of a long overdue change from the investigation in rust-lang#29293
and rust-lang#37477. The released binaries of rustc don't have debug information and so
don't actively suffer this problem but this can hit local development of rustc
and also larger programs compiled against libstd generating backtraces.

The main purpose of the mmap allocator in libacktrace is to be usable from a
signal handler, but we don't do that, so the normal allocator using malloc/free
should work well for us.
bors added a commit that referenced this issue Oct 26, 2017
std: Disable usage of mmap allocator in libbacktrace

This is sort of a long overdue change from the investigation in #29293
and #37477. The released binaries of rustc don't have debug information and so
don't actively suffer this problem but this can hit local development of rustc
and also larger programs compiled against libstd generating backtraces.

The main purpose of the mmap allocator in libacktrace is to be usable from a
signal handler, but we don't do that, so the normal allocator using malloc/free
should work well for us.
ianlancetaylor added a commit to ianlancetaylor/libbacktrace that referenced this issue Jan 25, 2018
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jul 17, 2020
This commit is a proof-of-concept for switching the standard library's
backtrace symbolication mechanism on most platforms from libbacktrace to
gimli. The standard library's support for `RUST_BACKTRACE=1` requires
in-process parsing of object files and DWARF debug information to
interpret it and print the filename/line number of stack frames as part
of a backtrace.

Historically this support in the standard library has come from a
library called "libbacktrace". The libbacktrace library seems to have
been extracted from gcc at some point and is written in C. We've had a
lot of issues with libbacktrace over time, unfortunately, though. The
library does not appear to be actively maintained since we've had
patches sit for months-to-years without comments. We have discovered a
good number of soundness issues with the library itself, both when
parsing valid DWARF as well as invalid DWARF. This is enough of an issue
that the libs team has previously decided that we cannot feed untrusted
inputs to libbacktrace. This also doesn't take into account the
portability of libbacktrace which has been difficult to manage and
maintain over time. While possible there are lots of exceptions and it's
the main C dependency of the standard library right now.

For years it's been the desire to switch over to a Rust-based solution
for symbolicating backtraces. It's been assumed that we'll be using the
Gimli family of crates for this purpose, which are targeted at safely
and efficiently parsing DWARF debug information. I've been working
recently to shore up the Gimli support in the `backtrace` crate. As of a
few weeks ago the `backtrace` crate, by default, uses Gimli when loaded
from crates.io. This transition has gone well enough that I figured it
was time to start talking seriously about this change to the standard
library.

This commit is a preview of what's probably the best way to integrate
the `backtrace` crate into the standard library with the Gimli feature
turned on. While today it's used as a crates.io dependency, this commit
switches the `backtrace` crate to a submodule of this repository which
will need to be updated manually. This is not done lightly, but is
thought to be the best solution. The primary reason for this is that the
`backtrace` crate needs to do some pretty nontrivial filesystem
interactions to locate debug information. Working without `std::fs` is
not an option, and while it might be possible to do some sort of
trait-based solution when prototyped it was found to be too unergonomic.
Using a submodule allows the `backtrace` crate to build as a submodule
of the `std` crate itself, enabling it to use `std::fs` and such.

Otherwise this adds new dependencies to the standard library. This step
requires extra attention because this means that these crates are now
going to be included with all Rust programs by default. It's important
to note, however, that we're already shipping libbacktrace with all Rust
programs by default and it has a bunch of C code implementing all of
this internally anyway, so we're basically already switching
already-shipping functionality to Rust from C.

* `object` - this crate is used to parse object file headers and
  contents. Very low-level support is used from this crate and almost
  all of it is disabled. Largely we're just using struct definitions as
  well as convenience methods internally to read bytes and such.

* `addr2line` - this is the main meat of the implementation for
  symbolication. This crate depends on `gimli` for DWARF parsing and
  then provides interfaces needed by the `backtrace` crate to turn an
  address into a filename / line number. This crate is actually pretty
  small (fits in a single file almost!) and mirrors most of what
  `dwarf.c` does for libbacktrace.

* `miniz_oxide` - the libbacktrace crate transparently handles
  compressed debug information which is compressed with zlib. This crate
  is used to decompress compressed debug sections.

* `gimli` - not actually used directly, but a dependency of `addr2line`.

* `adler32`- not used directly either, but a dependency of
  `miniz_oxide`.

The goal of this change is to improve the safety of backtrace
symbolication in the standard library, especially in the face of
possibly malformed DWARF debug information. Even to this day we're still
seeing segfaults in libbacktrace which could possibly become security
vulnerabilities. This change should almost entirely eliminate this
possibility whilc also paving the way forward to adding more features
like split debug information.

Some references for those interested are:

* Original addition of libbacktrace - rust-lang#12602
* OOM with libbacktrace - rust-lang#24231
* Backtrace failure due to use of uninitialized value - rust-lang#28447
* Possibility to feed untrusted data to libbacktrace - rust-lang#21889
* Soundness fix for libbacktrace - rust-lang#33729
* Crash in libbacktrace - rust-lang#39468
* Support for macOS, never merged - ianlancetaylor/libbacktrace#2
* Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477
* Update procedure is quite complicated due to how many patches we
  need to carry - rust-lang#50955
* Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060
* Segfault in libbacktrace on macOS - rust-lang#71397

Switching to Rust will not make us immune to all of these issues. The
crashes are expected to go away, but correctness and performance may
still have bugs arise. The gimli and `backtrace` crates, however, are
actively maintained unlike libbacktrace, so this should enable us to at
least efficiently apply fixes as situations come up.
bors added a commit to rust-lang-ci/rust that referenced this issue Jul 18, 2020
…Simulacrum

std: Switch from libbacktrace to gimli

This commit is a proof-of-concept for switching the standard library's
backtrace symbolication mechanism on most platforms from libbacktrace to
gimli. The standard library's support for `RUST_BACKTRACE=1` requires
in-process parsing of object files and DWARF debug information to
interpret it and print the filename/line number of stack frames as part
of a backtrace.

Historically this support in the standard library has come from a
library called "libbacktrace". The libbacktrace library seems to have
been extracted from gcc at some point and is written in C. We've had a
lot of issues with libbacktrace over time, unfortunately, though. The
library does not appear to be actively maintained since we've had
patches sit for months-to-years without comments. We have discovered a
good number of soundness issues with the library itself, both when
parsing valid DWARF as well as invalid DWARF. This is enough of an issue
that the libs team has previously decided that we cannot feed untrusted
inputs to libbacktrace. This also doesn't take into account the
portability of libbacktrace which has been difficult to manage and
maintain over time. While possible there are lots of exceptions and it's
the main C dependency of the standard library right now.

For years it's been the desire to switch over to a Rust-based solution
for symbolicating backtraces. It's been assumed that we'll be using the
Gimli family of crates for this purpose, which are targeted at safely
and efficiently parsing DWARF debug information. I've been working
recently to shore up the Gimli support in the `backtrace` crate. As of a
few weeks ago the `backtrace` crate, by default, uses Gimli when loaded
from crates.io. This transition has gone well enough that I figured it
was time to start talking seriously about this change to the standard
library.

This commit is a preview of what's probably the best way to integrate
the `backtrace` crate into the standard library with the Gimli feature
turned on. While today it's used as a crates.io dependency, this commit
switches the `backtrace` crate to a submodule of this repository which
will need to be updated manually. This is not done lightly, but is
thought to be the best solution. The primary reason for this is that the
`backtrace` crate needs to do some pretty nontrivial filesystem
interactions to locate debug information. Working without `std::fs` is
not an option, and while it might be possible to do some sort of
trait-based solution when prototyped it was found to be too unergonomic.
Using a submodule allows the `backtrace` crate to build as a submodule
of the `std` crate itself, enabling it to use `std::fs` and such.

Otherwise this adds new dependencies to the standard library. This step
requires extra attention because this means that these crates are now
going to be included with all Rust programs by default. It's important
to note, however, that we're already shipping libbacktrace with all Rust
programs by default and it has a bunch of C code implementing all of
this internally anyway, so we're basically already switching
already-shipping functionality to Rust from C.

* `object` - this crate is used to parse object file headers and
  contents. Very low-level support is used from this crate and almost
  all of it is disabled. Largely we're just using struct definitions as
  well as convenience methods internally to read bytes and such.

* `addr2line` - this is the main meat of the implementation for
  symbolication. This crate depends on `gimli` for DWARF parsing and
  then provides interfaces needed by the `backtrace` crate to turn an
  address into a filename / line number. This crate is actually pretty
  small (fits in a single file almost!) and mirrors most of what
  `dwarf.c` does for libbacktrace.

* `miniz_oxide` - the libbacktrace crate transparently handles
  compressed debug information which is compressed with zlib. This crate
  is used to decompress compressed debug sections.

* `gimli` - not actually used directly, but a dependency of `addr2line`.

* `adler32`- not used directly either, but a dependency of
  `miniz_oxide`.

The goal of this change is to improve the safety of backtrace
symbolication in the standard library, especially in the face of
possibly malformed DWARF debug information. Even to this day we're still
seeing segfaults in libbacktrace which could possibly become security
vulnerabilities. This change should almost entirely eliminate this
possibility whilc also paving the way forward to adding more features
like split debug information.

Some references for those interested are:

* Original addition of libbacktrace - rust-lang#12602
* OOM with libbacktrace - rust-lang#24231
* Backtrace failure due to use of uninitialized value - rust-lang#28447
* Possibility to feed untrusted data to libbacktrace - rust-lang#21889
* Soundness fix for libbacktrace - rust-lang#33729
* Crash in libbacktrace - rust-lang#39468
* Support for macOS, never merged - ianlancetaylor/libbacktrace#2
* Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477
* Update procedure is quite complicated due to how many patches we
  need to carry - rust-lang#50955
* Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060
* Segfault in libbacktrace on macOS - rust-lang#71397

Switching to Rust will not make us immune to all of these issues. The
crashes are expected to go away, but correctness and performance may
still have bugs arise. The gimli and `backtrace` crates, however, are
actively maintained unlike libbacktrace, so this should enable us to at
least efficiently apply fixes as situations come up.

---

I want to note that my purpose for creating a PR here is to start a conversation about this. I think that all the various pieces are in place that this is compelling enough that I think this transition should be talked about seriously. There are a number of items which still need to be addressed before actually merging this PR, however:

* [ ] `gimli` needs to be published to crates.io
* [ ] `addr2line` needs a publish
* [ ] `miniz_oxide` needs a publish
* [ ] Tests probably shouldn't recommend the `gimli` crate's traits for implementing
* [ ] The `backtrace` crate's branch changes need to be merged to the master branch (rust-lang/backtrace-rs#349)
* [ ] The support for `libbacktrace` on some platforms needs to be audited to see if we should support more strategies in the gimli implementation - rust-lang/backtrace-rs#325, rust-lang/backtrace-rs#326, rust-lang/backtrace-rs#350, rust-lang/backtrace-rs#351

Most of the merging/publishing I'm not actively pushing on right now. It's a bit wonky for crates to support libstd so I'm holding off on pulling the trigger everywhere until there's a bit more discussion about how to go through with this. Namely rust-lang/backtrace-rs#349 I'm going to hold off merging until we decide to go through with the submodule strategy.

In any case this is a pretty major change, so I suspect that the compiler team is likely going to be interested in this. I don't mean to force changes by dumping a bunch of code by any means. Integration of external crates into the standard library is so difficult I wanted to have a proof-of-concept to review while talking about whether to do this at all (hence the PR), but I'm more than happy to follow any processes needed to merge this. I must admit though that I'm not entirely sure myself at this time what the process would be to decide to merge this, so I'm hoping others can help me figure that out!
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jul 28, 2020
This commit is a proof-of-concept for switching the standard library's
backtrace symbolication mechanism on most platforms from libbacktrace to
gimli. The standard library's support for `RUST_BACKTRACE=1` requires
in-process parsing of object files and DWARF debug information to
interpret it and print the filename/line number of stack frames as part
of a backtrace.

Historically this support in the standard library has come from a
library called "libbacktrace". The libbacktrace library seems to have
been extracted from gcc at some point and is written in C. We've had a
lot of issues with libbacktrace over time, unfortunately, though. The
library does not appear to be actively maintained since we've had
patches sit for months-to-years without comments. We have discovered a
good number of soundness issues with the library itself, both when
parsing valid DWARF as well as invalid DWARF. This is enough of an issue
that the libs team has previously decided that we cannot feed untrusted
inputs to libbacktrace. This also doesn't take into account the
portability of libbacktrace which has been difficult to manage and
maintain over time. While possible there are lots of exceptions and it's
the main C dependency of the standard library right now.

For years it's been the desire to switch over to a Rust-based solution
for symbolicating backtraces. It's been assumed that we'll be using the
Gimli family of crates for this purpose, which are targeted at safely
and efficiently parsing DWARF debug information. I've been working
recently to shore up the Gimli support in the `backtrace` crate. As of a
few weeks ago the `backtrace` crate, by default, uses Gimli when loaded
from crates.io. This transition has gone well enough that I figured it
was time to start talking seriously about this change to the standard
library.

This commit is a preview of what's probably the best way to integrate
the `backtrace` crate into the standard library with the Gimli feature
turned on. While today it's used as a crates.io dependency, this commit
switches the `backtrace` crate to a submodule of this repository which
will need to be updated manually. This is not done lightly, but is
thought to be the best solution. The primary reason for this is that the
`backtrace` crate needs to do some pretty nontrivial filesystem
interactions to locate debug information. Working without `std::fs` is
not an option, and while it might be possible to do some sort of
trait-based solution when prototyped it was found to be too unergonomic.
Using a submodule allows the `backtrace` crate to build as a submodule
of the `std` crate itself, enabling it to use `std::fs` and such.

Otherwise this adds new dependencies to the standard library. This step
requires extra attention because this means that these crates are now
going to be included with all Rust programs by default. It's important
to note, however, that we're already shipping libbacktrace with all Rust
programs by default and it has a bunch of C code implementing all of
this internally anyway, so we're basically already switching
already-shipping functionality to Rust from C.

* `object` - this crate is used to parse object file headers and
  contents. Very low-level support is used from this crate and almost
  all of it is disabled. Largely we're just using struct definitions as
  well as convenience methods internally to read bytes and such.

* `addr2line` - this is the main meat of the implementation for
  symbolication. This crate depends on `gimli` for DWARF parsing and
  then provides interfaces needed by the `backtrace` crate to turn an
  address into a filename / line number. This crate is actually pretty
  small (fits in a single file almost!) and mirrors most of what
  `dwarf.c` does for libbacktrace.

* `miniz_oxide` - the libbacktrace crate transparently handles
  compressed debug information which is compressed with zlib. This crate
  is used to decompress compressed debug sections.

* `gimli` - not actually used directly, but a dependency of `addr2line`.

* `adler32`- not used directly either, but a dependency of
  `miniz_oxide`.

The goal of this change is to improve the safety of backtrace
symbolication in the standard library, especially in the face of
possibly malformed DWARF debug information. Even to this day we're still
seeing segfaults in libbacktrace which could possibly become security
vulnerabilities. This change should almost entirely eliminate this
possibility whilc also paving the way forward to adding more features
like split debug information.

Some references for those interested are:

* Original addition of libbacktrace - rust-lang#12602
* OOM with libbacktrace - rust-lang#24231
* Backtrace failure due to use of uninitialized value - rust-lang#28447
* Possibility to feed untrusted data to libbacktrace - rust-lang#21889
* Soundness fix for libbacktrace - rust-lang#33729
* Crash in libbacktrace - rust-lang#39468
* Support for macOS, never merged - ianlancetaylor/libbacktrace#2
* Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477
* Update procedure is quite complicated due to how many patches we
  need to carry - rust-lang#50955
* Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060
* Segfault in libbacktrace on macOS - rust-lang#71397

Switching to Rust will not make us immune to all of these issues. The
crashes are expected to go away, but correctness and performance may
still have bugs arise. The gimli and `backtrace` crates, however, are
actively maintained unlike libbacktrace, so this should enable us to at
least efficiently apply fixes as situations come up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

10 participants