Compiler crashes when trying to compile core with static relocation model

[Serentty]

I'm trying to compile Rust code for MS-DOS, and have written a custom target. In this target I have the relocation model set to static, since including that kind of information is impossible in a COM executable (which I'm using because it's trivial to generate them with linker scripts, unlike MZ executables). However, when compiling core, the compiler simply crashes with a segmentation fault. It seems to get most of the way through compiling core, as it actually crashes during or after LTO (I can't really tell). Here is the last part of the output of running the build with RUST_LOG=debug.

 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(<alloc::AllocErr as fmt::Debug>::fmt)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(<alloc::CannotReallocInPlace as fmt::Debug>::fmt)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(mem::size_of::<u8>)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(mem::size_of::<fmt::ArgumentV1>)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(mem::size_of::<fmt::rt::v1::Argument>)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(mem::size_of::<&str>)
 INFO 2019-02-18T19:19:11Z: rustc_codegen_ssa::base: codegen_instance(mem::size_of::<u32>)
Post-codegen
Ty interner             total           ty region  both
    Adt               :   7819  8.6%,  0.2%   0.0%  0.0%
    Array             :   1969  2.2%,  0.0%   0.0%  0.0%
    Slice             :    222  0.2%,  0.0%   0.0%  0.0%
    RawPtr            :    222  0.2%,  0.0%   0.0%  0.0%
    Ref               :  42210 46.3%,  0.1%   0.0%  0.0%
    FnDef             :  21176 23.2%,  0.0%   0.0%  0.0%
    FnPtr             :   8613  9.4%,  0.0%   0.0%  0.0%
    Placeholder       :      0  0.0%,  0.0%   0.0%  0.0%
    Generator         :      0  0.0%,  0.0%   0.0%  0.0%
    GeneratorWitness  :      0  0.0%,  0.0%   0.0%  0.0%
    Dynamic           :    253  0.3%,  0.0%   0.0%  0.0%
    Closure           :   1799  2.0%,  0.2%   0.0%  0.0%
    Tuple             :   2164  2.4%,  0.2%   0.0%  0.0%
    Bound             :     17  0.0%,  0.0%   0.0%  0.0%
    Param             :    240  0.3%,  0.0%   0.0%  0.0%
    Infer             :     16  0.0%,  0.0%   0.0%  0.0%
    UnnormalizedProjection:      0  0.0%,  0.0%   0.0%  0.0%
    Projection        :   4429  4.9%,  0.0%   0.0%  0.0%
    Opaque            :      0  0.0%,  0.0%   0.0%  0.0%
    Foreign           :      1  0.0%,  0.0%   0.0%  0.0%
                  total  91150         0.8%   0.0%  0.0%
Substs interner: #65877
Region interner: #30211
Stability interner: #322
Allocation interner: #1060
Layout interner: #411
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: 641 symbols to preserve in this crate
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: going for that thin, thin LTO
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 0 - core.1v8m1ai7-cgu.15
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 1 - core.1v8m1ai7-cgu.0
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 2 - core.1v8m1ai7-cgu.2
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 3 - core.1v8m1ai7-cgu.5
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 4 - core.1v8m1ai7-cgu.1
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 5 - core.1v8m1ai7-cgu.6
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 6 - core.1v8m1ai7-cgu.7
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 7 - core.1v8m1ai7-cgu.14
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 8 - core.1v8m1ai7-cgu.3
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 9 - core.1v8m1ai7-cgu.9
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 10 - core.1v8m1ai7-cgu.8
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 11 - core.1v8m1ai7-cgu.11
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 12 - core.1v8m1ai7-cgu.12
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 13 - core.1v8m1ai7-cgu.4
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 14 - core.1v8m1ai7-cgu.13
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: local module: 15 - core.1v8m1ai7-cgu.10
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: thin LTO data created
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: thin LTO import map loaded
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: checking which modules can be-reused and which have to be re-optimized.
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.15: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.0: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.2: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.5: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.1: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.6: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.7: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.14: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.3: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.9: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.8: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.11: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.12: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.4: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.13: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto:  - core.1v8m1ai7-cgu.10: re-compiled
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: running thin lto passes over core.1v8m1ai7-cgu.11
 INFO 2019-02-18T19:19:11Z: rustc_codegen_llvm::back::lto: running thin lto passes over core.1v8m1ai7-cgu.15
[2019-02-18T19:19:20Z INFO  cargo::core::compiler::job_queue] end: core v0.0.0 (/home/seren/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore) => Target(lib)/Profile(release) => Target
[2019-02-18T19:19:20Z INFO  cargo::util::rustc] updated rustc info cache
[2019-02-18T19:19:20Z DEBUG cargo] exit_with_error; err=CliError { error: Some(ProcessError { desc: "process didn\'t exit successfully: `rustc --crate-name core /home/seren/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/lib.rs --color always --crate-type lib --emit=dep-info,link -C opt-level=3 -C panic=abort -C metadata=83dd25ed9bb01753 -C extra-filename=-83dd25ed9bb01753 --out-dir /tmp/xargo.Ts6CcVcYOJgB/target/dos/release/deps --target /home/seren/Code/Rust/dos/dos.json -L dependency=/tmp/xargo.Ts6CcVcYOJgB/target/dos/release/deps -L dependency=/tmp/xargo.Ts6CcVcYOJgB/target/release/deps` (signal: 11, SIGSEGV: invalid memory reference)", exit: Some(ExitStatus(ExitStatus(139))), output: None }

Could not compile `core`.), unknown: false, exit_code: 101 }
error: Could not compile `core`.

Caused by:
  process didn't exit successfully: `rustc --crate-name core /home/seren/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/lib.rs --color always --crate-type lib --emit=dep-info,link -C opt-level=3 -C panic=abort -C metadata=83dd25ed9bb01753 -C extra-filename=-83dd25ed9bb01753 --out-dir /tmp/xargo.Ts6CcVcYOJgB/target/dos/release/deps --target /home/seren/Code/Rust/dos/dos.json -L dependency=/tmp/xargo.Ts6CcVcYOJgB/target/dos/release/deps -L dependency=/tmp/xargo.Ts6CcVcYOJgB/target/release/deps` (signal: 11, SIGSEGV: invalid memory reference)
error: `"cargo" "rustc" "-p" "alloc" "--release" "--manifest-path" "/tmp/xargo.Ts6CcVcYOJgB/Cargo.toml" "--target" "dos.json" "--" "-Z" "force-unstable-if-unmarked"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace

[superobertking]

I also met this issue when I was compiling code for Pintos. Using lldb, I found that the compiler crashed inside LLVM function (anonymous namespace)::MachineCSE::runOnMachineFunction. It dereferenced a null pointer. This might be a LLVM robustness bug.

(lldb) c
Process 4803 resuming
Process 4803 stopped
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #0: 0x00000001016ce319 librustc_driver-6503a362531c867a.dylib`(anonymous namespace)::MachineCSE::runOnMachineFunction(llvm::MachineFunction&) + 4201
librustc_driver-6503a362531c867a.dylib`(anonymous namespace)::MachineCSE::runOnMachineFunction:
->  0x1016ce319 <+4201>: movq   0x10(%rax), %rax
    0x1016ce31d <+4205>: cmpw   $0x10, (%rax)
    0x1016ce321 <+4209>: jne    0x1016ce2d0               ; <+4128>
    0x1016ce323 <+4211>: movq   0x20(%r12), %rax
Target 0: (rustc) stopped.
(lldb) p $rax
(unsigned long) $0 = 0
(lldb) disassemble
librustc_driver-6503a362531c867a.dylib`(anonymous namespace)::MachineCSE::runOnMachineFunction:
...
    0x1016ce2ef <+4159>:  movq   -0x30(%rbp), %r12
    0x1016ce2f3 <+4163>:  movq   0x88(%r12), %rdi
    0x1016ce2fb <+4171>:  movl   %r15d, %esi
    0x1016ce2fe <+4174>:  callq  0x10173d700               ; llvm::MachineRegisterInfo::hasOneNonDBGUse(unsigned int) const
    0x1016ce303 <+4179>:  movb   %al, -0x48(%rbp)
    0x1016ce306 <+4182>:  movq   0x88(%r12), %rdi
    0x1016ce30e <+4190>:  movl   %r15d, %esi
    0x1016ce311 <+4193>:  callq  0x10173d5e0               ; llvm::MachineRegisterInfo::getVRegDef(unsigned int) const
    0x1016ce316 <+4198>:  movq   %rax, %r12
->  0x1016ce319 <+4201>:  movq   0x10(%rax), %rax
    0x1016ce31d <+4205>:  cmpw   $0x10, (%rax)
    0x1016ce321 <+4209>:  jne    0x1016ce2d0               ; <+4128>
    0x1016ce323 <+4211>:  movq   0x20(%r12), %rax
    0x1016ce328 <+4216>:  movl   0x24(%rax), %ecx
    0x1016ce32b <+4219>:  testl  %ecx, %ecx
    0x1016ce32d <+4221>:  jns    0x1016ce2d0               ; <+4128>
    0x1016ce32f <+4223>:  testl  $0xfff00, (%rax)          ; imm = 0xFFF00
    0x1016ce335 <+4229>:  jne    0x1016ce2d0               ; <+4128>
    0x1016ce337 <+4231>:  testl  $0xfff00, 0x20(%rax)      ; imm = 0xFFF00
    0x1016ce33e <+4238>:  jne    0x1016ce2d0               ; <+4128>
    0x1016ce340 <+4240>:  movq   -0x30(%rbp), %rax
    0x1016ce344 <+4244>:  movq   0x88(%rax), %rdi
    0x1016ce34b <+4251>:  movl   %ecx, %esi
    0x1016ce34d <+4253>:  movl   %ecx, -0x38(%rbp)
    0x1016ce350 <+4256>:  movl   %r15d, %edx
    0x1016ce353 <+4259>:  xorl   %ecx, %ecx
    0x1016ce355 <+4261>:  callq  0x10173bf00               ; llvm::MachineRegisterInfo::constrainRegAttrs(unsigned int, unsigned int, unsigned int)
    0x1016ce35a <+4266>:  testb  %al, %al
...
(lldb) bt
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x00000001016ce319 librustc_driver-6503a362531c867a.dylib`(anonymous namespace)::MachineCSE::runOnMachineFunction(llvm::MachineFunction&) + 4201
    frame #1: 0x00000001016dec0d librustc_driver-6503a362531c867a.dylib`llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 285
    frame #2: 0x0000000101f70271 librustc_driver-6503a362531c867a.dylib`llvm::FPPassManager::runOnFunction(llvm::Function&) + 1057
    frame #3: 0x0000000101f70573 librustc_driver-6503a362531c867a.dylib`llvm::FPPassManager::runOnModule(llvm::Module&) + 131
    frame #4: 0x0000000101f70a49 librustc_driver-6503a362531c867a.dylib`llvm::legacy::PassManagerImpl::run(llvm::Module&) + 969
    frame #5: 0x0000000100692c93 librustc_driver-6503a362531c867a.dylib`LLVMRustWriteOutputFile + 563
    frame #6: 0x0000000100617f06 librustc_driver-6503a362531c867a.dylib`rustc_codegen_llvm::back::write::write_output_file::hc095d205b92ff551 (.llvm.11416920907895951992) + 86
    frame #7: 0x0000000100673ff8 librustc_driver-6503a362531c867a.dylib`rustc_codegen_llvm::back::write::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h550e26036a74232a (.llvm.2814728182923408674) + 1560
    frame #8: 0x0000000100672463 librustc_driver-6503a362531c867a.dylib`rustc::util::common::time_ext::hc32b343c971dafa4 + 163
    frame #9: 0x000000010061e403 librustc_driver-6503a362531c867a.dylib`_$LT$rustc_codegen_llvm..LlvmCodegenBackend$u20$as$u20$rustc_codegen_ssa..traits..write..WriteBackendMethods$GT$::codegen::h4171fac8b00b20c5 + 3635
    frame #10: 0x000000010052e773 librustc_driver-6503a362531c867a.dylib`rustc_codegen_ssa::back::write::execute_work_item::h8258a8c1c35d32be + 499
    frame #11: 0x0000000100665a7f librustc_driver-6503a362531c867a.dylib`std::sys_common::backtrace::__rust_begin_short_backtrace::h938c64ba5fbd7ad1 + 239
    frame #12: 0x000000010058bbcb librustc_driver-6503a362531c867a.dylib`std::panicking::try::do_call::h76276dd7b0e89039 (.llvm.2242646780768277059) + 43
    frame #13: 0x000000010616721f libstd-bd6d3f2e95da2fcf.dylib`__rust_maybe_catch_panic + 31
    frame #14: 0x0000000100518ef6 librustc_driver-6503a362531c867a.dylib`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6b9bb22ee9c4c669 + 134
    frame #15: 0x000000010613878e libstd-bd6d3f2e95da2fcf.dylib`_$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h1933a4bdd3bf4ecd + 62
    frame #16: 0x0000000106165f5e libstd-bd6d3f2e95da2fcf.dylib`std::sys::unix::thread::Thread::new::thread_start::h8ae6457b30b94596 + 142
    frame #17: 0x00007fff5e4d52eb libsystem_pthread.dylib`_pthread_body + 126
    frame #18: 0x00007fff5e4d8249 libsystem_pthread.dylib`_pthread_start + 66
    frame #19: 0x00007fff5e4d440d libsystem_pthread.dylib`thread_start + 13

rustc 1.42.0-nightly (0de96d3 2019-12-19)
cargo 1.41.0-nightly (626f0f40e 2019-12-03)

[superobertking]

But I also found that there is a conflict in my compile parameters. I had relro-level set to full and relocation-model set to static at the same time. Removing either of them would let the compiler work fine as usual.

Maybe you can provide your custom target json file and command lines (including environment variable exports) and see if there is any similar issue.

I also think there should be a bug report to the upstream LLVM. Rust compiler is really unprotected beyond FFI.