Remove the `alloc_jemalloc` crate

[alexcrichton]

This commit removes the alloc_jemalloc crate from the standard library and all related configuration. We will no longer be shipping this unstable crate. Rationale for this is provided on #36963 and the many linked issues, but I can inline rationale here if desired!

We currently rely on jemalloc for increased perf in the Rust compiler, however. This perf run shows that if we switch to glibc 2.23's allocator that it's slower than jemalloc across many benchmarks. This perf run, however, shows that if we use jemalloc-sys from crates.io then rustc actually gets faster across all benchmarks! (presumably because it has a more recent version of jemalloc than our submodule).

As a result, it's expected that this doesn't regress any code (as it's just removing an unstable crate) and it should actually improve rustc performance because it updates jemalloc.

Closes #36963

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

cc @sfackler

[rust-highfive]

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:03:31]    Compiling syn v0.14.9
[00:03:43]    Compiling serde_json v1.0.31
[00:03:44]    Compiling serde_derive v1.0.75
[00:04:04]    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
[00:04:04] error[E0425]: cannot find value `whitelisted` in this scope
[00:04:04]   --> tools/tidy/src/cargo.rs:87:27
[00:04:04]    |
[00:04:04] 87 |         let whitelisted = whitelisted || krate.starts_with("panic");
[00:04:04] 
[00:04:04] error: aborting due to previous error
[00:04:04] 
[00:04:04] For more information about this error, try `rustc --explain E0425`.
[00:04:04] For more information about this error, try `rustc --explain E0425`.
[00:04:04] error: Could not compile `tidy`.
[00:04:04] To learn more, run the command again with --verbose.
[00:04:04] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--manifest-path" "/checkout/src/tools/tidy/Cargo.toml" "--features" "" "--message-format" "json"
[00:04:04] expected success, got: exit code: 101
[00:04:04] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:04] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:04] Build completed unsuccessfully in 0:00:41
[00:04:04] make: *** [tidy] Error 1
[00:04:04] Makefile:79: recipe for target 'tidy' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0fb07fbe
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:2661640e:start=1540088918894153030,finish=1540088918898374219,duration=4221189
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0fd63c58
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0fb7a3b0
travis_time:start:0fb7a3b0
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:16a7b79d
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[sfackler]

Woo!

There is one pretty significant difference between jemalloc 4 and 5. The new version depends on a background thread to clear out allocation caches, and the background thread defaults to off. Without it, we saw a very large increase in heap memory on a production service until we enabled background threads. See jemalloc/jemalloc#1128 for some background.

While this doesn't matter for rustc at all, it will probably have an impact on long running services. We may want to fold the switch to the system allocator as the default into this change as well?

[lnicola]

Ah, that might explain https://perf.rust-lang.org/compare.html?start=22cc2ae8057d14e980b7c784e1eb2eee26b59e7d&end=10c95ccfa7a7adc12f4e608621ca29f9b98eed29&stat=max-rss.

[bors]

☔️ The latest upstream changes (presumably #54919) made this pull request unmergeable. Please resolve the merge conflicts.

[SimonSapin]

We may want to fold the switch to the system allocator as the default into this change as well?

As far as I understand, that switch is already part of this PR.

[alexcrichton]

@sfackler ah yeah as @SimonSapin mentioned this PR is doing the switch as well as the update.

Despite it looking like we enable the background thread by default in jemalloc-sys the build script only actually disables the background thread if it's turned off. @sfackler do you know the right option necessary to enable the background thread so we can test in rustc?

[sfackler]

Those options compile in support for the background thread but it's still disabled at runtime by default. I don't think it would have much of an effect on rustc memory usage - my understanding is that it's used to clean up caches on idle threads and that active threads clean themselves up periodically when touching the allocator.

There are a bunch of settings you can poke at to control how aggressively jemalloc caches if you want: http://jemalloc.net/jemalloc.3.html

[estebank]

@bors r+

[bors]

📌 Commit 94cb105 has been approved by estebank

[bors]

⌛️ Testing commit 94cb105 with merge cb4b372...

[rust-highfive]

The job dist-x86_64-musl of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:47:25] failures:
[00:47:25] 
[00:47:25] ---- [ui] ui/huge-enum.rs stdout ----
[00:47:25] 
[00:47:25] error: Error: expected failure status (Some(1)) but received status None.
[00:47:25] status: signal: 11
[00:47:25] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/huge-enum.rs" "--target=x86_64-unknown-linux-musl" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "-Clinker=/musl-x86_64/bin/musl-gcc" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/auxiliary" "-A" "unused"
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] ------------------------------------------
[00:47:25] stderr:
[00:47:25] stderr:
[00:47:25] ------------------------------------------
[00:47:25] {"message":"the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture","code":null,"level":"error","spans":[],"children":[],"rendered":"error: the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture\n\n"}
[00:47:25] {"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"}
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] thread '[ui] ui/huge-enum.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3284:9
[00:47:25] note: Run with `RUST_BACKTRACE=1` for a backtrace.
---
[00:47:25] 
[00:47:25] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:503:22
[00:47:25] 
[00:47:25] 
[00:47:25] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-musl" "--mode" "ui" "--target" "x86_64-unknown-linux-musl" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "/musl-x86_64/bin/musl-gcc" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--llvm-version" "8.0.0svn\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[00:47:25] 
[00:47:25] 
[00:47:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target x86_64-unknown-linux-musl
[00:47:25] Build completed unsuccessfully in 0:44:38
---
travis_time:end:2b62d04c:start=1540164656425485779,finish=1540164656435475220,duration=9989441
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0faaa9c0
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:start:crashlog
obj/cores/core.2489.!checkout!obj!build!x86_64-unknown-linux-gnu!stage2!bin!rustc
[New LWP 2506]
[New LWP 2503]
[New LWP 2507]
[New LWP 2504]
[New LWP 2489]
[New LWP 2508]
warning: Could not load shared library symbols for 7 libraries, e.g. /lib/x86_64-linux-gnu/libc.so.6.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc /checkout/src/tes'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#1  0x00007f53f9bbda0f in llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&, llvm::raw_pwrite_stream&, llvm::raw_pwrite_stream*, llvm::TargetMachine::CodeGenFileType, bool, llvm::MachineModuleInfo*) () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#2  0x00007f53f8ac4354 in LLVMRustWriteOutputFile () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#3  0x00007f53f8978bbc in rustc_codegen_llvm::back::write::write_output_file::h43d626d1bbd593ab () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#4  0x00007f53f896a63d in rustc_codegen_llvm::back::write::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h334d90084152b97a () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#5  0x00007f53f8961c39 in rustc::util::common::time_ext::h1093c0a1462f2cda () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#6  0x00007f53f897b389 in rustc_codegen_llvm::back::write::codegen::h7e2546fe2c85c981 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#7  0x00007f53f897fbca in rustc_codegen_llvm::back::write::execute_work_item::h34a3780663354f1b () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#8  0x00007f53f893aba2 in std::sys_common::backtrace::__rust_begin_short_backtrace::h95caea62758ed294 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#9  0x00007f53f8972e49 in std::panicking::try::do_call::hfbe7bba3f27cb8ef () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#10 0x00007f54034da3da in __rust_maybe_catch_panic () at libpanic_unwind/lib.rs:102
#11 0x00007f53f89779f8 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hf492e470c0c53396 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#12 0x00007f54034cad9e in call_once<(),()> () at liballoc/boxed.rs:682
#13 std::sys_common::thread::start_thread::he4cc3b686d083587 () at libstd/sys_common/thread.rs:24
#14 0x00007f54034a0756 in std::sys::unix::thread::Thread::new::thread_start::hff4201b83401a7d4 () at libstd/sys/unix/thread.rs:90
#15 0x00007f53fd50d6ba in ?? ()
#16 0x0000000000000000 in ?? ()
travis_time:end:0faaa9c0:start=1540164656445078288,finish=1540164657544458273,duration=1099379985
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1eda26f9
travis_time:start:1eda26f9
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:13d3145f
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[bors]

💔 Test failed - status-travis

[gnzlbg]

I get that on these platforms global_allocator is not needed, but I still do wonder why this does not use it.

[alexcrichton]

The #[global_allocator] attribute is actually incompatible with this due to the way linkage works and such. I can go into details if necessary, but the answer to your question at least is "it's required by the way we build rustc"

[gnzlbg]

Ah damn, thanks for the explanation.

[alexcrichton]

@bors: retry

After many attempts locally I can't reproduce, so I'm curious if we can get another failure on bors...

[jonhoo]

Just for clarity, does this change just affect rustc, or does it also affect all crates? Since it closes #36963, does that mean that all crates will now be compiled with the system memory allocator, or are we just moving from alloc_jemalloc to jemalloc-sys?

[gnzlbg]

@alexcrichton what's this?

[alexcrichton]

Ah this is a mistake

[SimonSapin]

Yes, System is now the default on all platforms for all crate types (such as executable). It’s only rustc that this PR makes link to jemalloc-sys. I’ve written up an announcement:

• https://internals.rust-lang.org/t/jemalloc-was-just-removed-from-the-standard-library/8759
• https://www.reddit.com/r/rust/comments/9twam5/jemalloc_was_just_removed_from_the_standard/

[alexcrichton]

@jonhoo this affects all crates, no crate will be compiled against jemalloc by default now. The compilers we distribute for OSX/Linux will, however, still enable jemalloc (but only the compiler itself)

[nnethercote]

Here are the final perf results from the landing. Almost everything improved by at least 1%, and some were up to 15%. Pretty great!

[SimonSapin]

This is most likely from upgrading jemalloc from 4.5 to 5.1, isn’t it?

[lnicola]

Are there plans to do something about the memory usage regression?

[gnzlbg]

@lnicola the jemalloc-sys master branch has re-added support for enabling background threads unconditionally, which might help here - there is also an open issue about gathering statistics of rustc size-class usage which could help here too (gnzlbg/jemallocator#79).