New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the `alloc_jemalloc` crate #55238

Merged
merged 3 commits into from Nov 3, 2018

Conversation

Projects
None yet
@alexcrichton
Member

alexcrichton commented Oct 21, 2018

This commit removes the alloc_jemalloc crate from the standard library and all related configuration. We will no longer be shipping this unstable crate. Rationale for this is provided on #36963 and the many linked issues, but I can inline rationale here if desired!

We currently rely on jemalloc for increased perf in the Rust compiler, however. This perf run shows that if we switch to glibc 2.23's allocator that it's slower than jemalloc across many benchmarks. This perf run, however, shows that if we use jemalloc-sys from crates.io then rustc actually gets faster across all benchmarks! (presumably because it has a more recent version of jemalloc than our submodule).

As a result, it's expected that this doesn't regress any code (as it's just removing an unstable crate) and it should actually improve rustc performance because it updates jemalloc.

Closes #36963

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 21, 2018

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Oct 21, 2018

@rust-highfive

This comment was marked as resolved.

Collaborator

rust-highfive commented Oct 21, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:03:31]    Compiling syn v0.14.9
[00:03:43]    Compiling serde_json v1.0.31
[00:03:44]    Compiling serde_derive v1.0.75
[00:04:04]    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
[00:04:04] error[E0425]: cannot find value `whitelisted` in this scope
[00:04:04]   --> tools/tidy/src/cargo.rs:87:27
[00:04:04]    |
[00:04:04] 87 |         let whitelisted = whitelisted || krate.starts_with("panic");
[00:04:04] 
[00:04:04] error: aborting due to previous error
[00:04:04] 
[00:04:04] For more information about this error, try `rustc --explain E0425`.
[00:04:04] For more information about this error, try `rustc --explain E0425`.
[00:04:04] error: Could not compile `tidy`.
[00:04:04] To learn more, run the command again with --verbose.
[00:04:04] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--manifest-path" "/checkout/src/tools/tidy/Cargo.toml" "--features" "" "--message-format" "json"
[00:04:04] expected success, got: exit code: 101
[00:04:04] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:04] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:04] Build completed unsuccessfully in 0:00:41
[00:04:04] make: *** [tidy] Error 1
[00:04:04] Makefile:79: recipe for target 'tidy' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0fb07fbe
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:2661640e:start=1540088918894153030,finish=1540088918898374219,duration=4221189
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0fd63c58
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0fb7a3b0
travis_time:start:0fb7a3b0
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:16a7b79d
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@sfackler

This comment has been minimized.

Member

sfackler commented Oct 21, 2018

Woo!

There is one pretty significant difference between jemalloc 4 and 5. The new version depends on a background thread to clear out allocation caches, and the background thread defaults to off. Without it, we saw a very large increase in heap memory on a production service until we enabled background threads. See jemalloc/jemalloc#1128 for some background.

While this doesn't matter for rustc at all, it will probably have an impact on long running services. We may want to fold the switch to the system allocator as the default into this change as well?

@bors

This comment was marked as resolved.

Contributor

bors commented Oct 21, 2018

☔️ The latest upstream changes (presumably #54919) made this pull request unmergeable. Please resolve the merge conflicts.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Oct 21, 2018

We may want to fold the switch to the system allocator as the default into this change as well?

As far as I understand, that switch is already part of this PR.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Oct 21, 2018

@sfackler ah yeah as @SimonSapin mentioned this PR is doing the switch as well as the update.

Despite it looking like we enable the background thread by default in jemalloc-sys the build script only actually disables the background thread if it's turned off. @sfackler do you know the right option necessary to enable the background thread so we can test in rustc?

@sfackler

This comment has been minimized.

Member

sfackler commented Oct 21, 2018

Those options compile in support for the background thread but it's still disabled at runtime by default. I don't think it would have much of an effect on rustc memory usage - my understanding is that it's used to clean up caches on idle threads and that active threads clean themselves up periodically when touching the allocator.

There are a bunch of settings you can poke at to control how aggressively jemalloc caches if you want: http://jemalloc.net/jemalloc.3.html

@estebank

This comment has been minimized.

Contributor

estebank commented Oct 21, 2018

@bors r+

@bors

This comment was marked as outdated.

Contributor

bors commented Oct 21, 2018

📌 Commit 94cb105 has been approved by estebank

@bors

This comment was marked as outdated.

Contributor

bors commented Oct 21, 2018

⌛️ Testing commit 94cb105 with merge cb4b372...

bors added a commit that referenced this pull request Oct 21, 2018

Auto merge of #55238 - alexcrichton:rm-jemalloc, r=estebank
Remove the `alloc_jemalloc` crate

This commit removes the `alloc_jemalloc` crate from the standard library and all related configuration. We will no longer be shipping this unstable crate. Rationale for this is provided on #36963 and the many linked issues, but I can inline rationale here if desired!

We currently rely on jemalloc for increased perf in the Rust compiler, however. [This perf run shows](https://perf.rust-lang.org/compare.html?start=74ff7dcb1388e60a613cd6050bcd372a3cc4998b&end=7e7928dc0340d79b404e93f0c79eb4b946c1d669&stat=wall-time) that if we switch to glibc 2.23's allocator that it's slower than jemalloc across many benchmarks. [This perf run, however](https://perf.rust-lang.org/compare.html?start=22cc2ae8057d14e980b7c784e1eb2eee26b59e7d&end=10c95ccfa7a7adc12f4e608621ca29f9b98eed29), shows that if we use `jemalloc-sys` from crates.io then rustc actually gets faster across all benchmarks! (presumably because it has a more recent version of jemalloc than our submodule).

As a result, it's expected that this doesn't regress any code (as it's just removing an unstable crate) and it should actually improve rustc performance because it updates jemalloc.

Closes #36963
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Oct 21, 2018

The job dist-x86_64-musl of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:47:25] failures:
[00:47:25] 
[00:47:25] ---- [ui] ui/huge-enum.rs stdout ----
[00:47:25] 
[00:47:25] error: Error: expected failure status (Some(1)) but received status None.
[00:47:25] status: signal: 11
[00:47:25] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/huge-enum.rs" "--target=x86_64-unknown-linux-musl" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "-Clinker=/musl-x86_64/bin/musl-gcc" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/auxiliary" "-A" "unused"
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] ------------------------------------------
[00:47:25] stderr:
[00:47:25] stderr:
[00:47:25] ------------------------------------------
[00:47:25] {"message":"the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture","code":null,"level":"error","spans":[],"children":[],"rendered":"error: the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture\n\n"}
[00:47:25] {"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"}
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] thread '[ui] ui/huge-enum.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3284:9
[00:47:25] note: Run with `RUST_BACKTRACE=1` for a backtrace.
---
[00:47:25] 
[00:47:25] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:503:22
[00:47:25] 
[00:47:25] 
[00:47:25] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-musl" "--mode" "ui" "--target" "x86_64-unknown-linux-musl" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "/musl-x86_64/bin/musl-gcc" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--llvm-version" "8.0.0svn\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[00:47:25] 
[00:47:25] 
[00:47:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target x86_64-unknown-linux-musl
[00:47:25] Build completed unsuccessfully in 0:44:38
---
travis_time:end:2b62d04c:start=1540164656425485779,finish=1540164656435475220,duration=9989441
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0faaa9c0
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:start:crashlog
obj/cores/core.2489.!checkout!obj!build!x86_64-unknown-linux-gnu!stage2!bin!rustc
[New LWP 2506]
[New LWP 2503]
[New LWP 2507]
[New LWP 2504]
[New LWP 2489]
[New LWP 2508]
warning: Could not load shared library symbols for 7 libraries, e.g. /lib/x86_64-linux-gnu/libc.so.6.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc /checkout/src/tes'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#1  0x00007f53f9bbda0f in llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&, llvm::raw_pwrite_stream&, llvm::raw_pwrite_stream*, llvm::TargetMachine::CodeGenFileType, bool, llvm::MachineModuleInfo*) () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#2  0x00007f53f8ac4354 in LLVMRustWriteOutputFile () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#3  0x00007f53f8978bbc in rustc_codegen_llvm::back::write::write_output_file::h43d626d1bbd593ab () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#4  0x00007f53f896a63d in rustc_codegen_llvm::back::write::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h334d90084152b97a () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#5  0x00007f53f8961c39 in rustc::util::common::time_ext::h1093c0a1462f2cda () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#6  0x00007f53f897b389 in rustc_codegen_llvm::back::write::codegen::h7e2546fe2c85c981 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#7  0x00007f53f897fbca in rustc_codegen_llvm::back::write::execute_work_item::h34a3780663354f1b () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#8  0x00007f53f893aba2 in std::sys_common::backtrace::__rust_begin_short_backtrace::h95caea62758ed294 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#9  0x00007f53f8972e49 in std::panicking::try::do_call::hfbe7bba3f27cb8ef () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#10 0x00007f54034da3da in __rust_maybe_catch_panic () at libpanic_unwind/lib.rs:102
#11 0x00007f53f89779f8 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hf492e470c0c53396 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#12 0x00007f54034cad9e in call_once<(),()> () at liballoc/boxed.rs:682
#13 std::sys_common::thread::start_thread::he4cc3b686d083587 () at libstd/sys_common/thread.rs:24
#14 0x00007f54034a0756 in std::sys::unix::thread::Thread::new::thread_start::hff4201b83401a7d4 () at libstd/sys/unix/thread.rs:90
#15 0x00007f53fd50d6ba in ?? ()
#16 0x0000000000000000 in ?? ()
travis_time:end:0faaa9c0:start=1540164656445078288,finish=1540164657544458273,duration=1099379985
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1eda26f9
travis_time:start:1eda26f9
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:13d3145f
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment was marked as outdated.

Contributor

bors commented Oct 21, 2018

💔 Test failed - status-travis

// enable this by default on other platforms, so other platforms aren't handled
// here yet.
#[cfg(feature = "jemalloc-sys")]
extern crate jemalloc_sys;

This comment has been minimized.

@gnzlbg

gnzlbg Oct 23, 2018

Contributor

I get that on these platforms global_allocator is not needed, but I still do wonder why this does not use it.

This comment has been minimized.

@alexcrichton

alexcrichton Oct 24, 2018

Member

The #[global_allocator] attribute is actually incompatible with this due to the way linkage works and such. I can go into details if necessary, but the answer to your question at least is "it's required by the way we build rustc"

This comment has been minimized.

@gnzlbg

gnzlbg Oct 24, 2018

Contributor

Ah damn, thanks for the explanation.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Oct 24, 2018

@bors: retry

After many attempts locally I can't reproduce, so I'm curious if we can get another failure on bors...

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Oct 24, 2018

There is one pretty significant difference between jemalloc 4 and 5. […]

I think that jemalloc-sys 0.1.2 packages exactly the same version of jemalloc that rustc currently uses. (That is, 4.5.0 plus a couple patches jemalloc/jemalloc@4.5.0...rust-lang:rust-2017-05-09)

Maybe this PR should use that, and later we can consider upgrading to jemalloc 5 separately?

% cd jemallocator/jemalloc-sys
% paste <(git log -p jemalloc|grep Date|grep -v 'Fri Apr 27') <(for c in $(git log -p jemalloc|grep '^+S'|cut -d' ' -f3); (cd jemalloc; echo -n "$c "; git show $c:ChangeLog|head|grep \*))
Date:   Wed May 16 19:16:50 2018 +0200	61efbda7098de6fe64c362d309824864308c36d4 * 5.1.0 (May 4th, 2018)
Date:   Thu May 3 16:37:24 2018 +0200	b8f4c730eff28edee4b583ff5b6ee1fac0f26c27 * 5.0.1 (July 1, 2017)
Date:   Thu May 3 00:23:40 2018 +0200	b8f4c730eff28edee4b583ff5b6ee1fac0f26c27 * 5.0.1 (July 1, 2017)
Date:   Thu Nov 23 19:08:53 2017 +0100	3c8e8db1b57e8101c33b2738213aad07cecbb3d3 * 5.0.1 (July 1, 2017)
Date:   Thu Jul 27 11:07:15 2017 -0300	b6bdb328cf73b67129f7e0a92a513d158763aadb * 4.5.0 (February 28, 2017)
Date:   Mon May 29 21:03:47 2017 +0800	3288e0659c08fb5006f6d6dd4b5675ed0c2c432a * 4.5.0 (February 28, 2017)
Date:   Thu Mar 30 22:15:07 2017 +0800	11bfb0dcf85f7aa92abd30524bb1e42e18d108c6 * 4.1.0 (February 28, 2016)
Date:   Mon Mar 7 13:15:29 2016 -0800	aab1c0a0e0b39825b16673128729ef46310a5da8 * 4.1.0 (February 28, 2016)
Date:   Wed Jan 20 17:03:24 2016 -0800	f84e30927284b0c500ed3eaf09e8e159da20ddaf * 4.0.4 (October 24, 2015)
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 3, 2018

The job dist-mips-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[01:16:51] travis_fold:start:stage2-cargo
travis_time:start:stage2-cargo
Building stage2 tool cargo (mips-unknown-linux-gnu)
[01:16:52]  Downloading crates ...
[01:17:12] warning: spurious network error (2 tries remaining): [6] Couldn't resolve host name (Could not resolve host: crates.io)
[01:17:32] warning: spurious network error (1 tries remaining): [6] Couldn't resolve host name (Could not resolve host: crates.io)
[01:17:52] error: failed to download from `https://crates.io/api/v1/crates/openssl-src/111.0.1+1.1.1/download`
[01:17:52] Caused by:
[01:17:52]   [6] Couldn't resolve host name (Could not resolve host: crates.io)
[01:17:52] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "mips-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--manifest-path" "/checkout/src/tools/cargo/Cargo.toml" "--features" "rustc-workspace-hack/all-static" "--message-format" "json"
[01:17:52] expected success, got: exit code: 101
---
travis_time:end:06636400:start=1541227133940370831,finish=1541227133946776653,duration=6405822
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:13c4cb92
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:05d06bfd
travis_time:start:05d06bfd
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:11563e00
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@kennytm

This comment has been minimized.

Member

kennytm commented Nov 3, 2018

@bors

This comment has been minimized.

Contributor

bors commented Nov 3, 2018

⌛️ Testing commit 14c6835 with merge b6e8f9d...

bors added a commit that referenced this pull request Nov 3, 2018

Auto merge of #55238 - alexcrichton:rm-jemalloc, r=estebank
Remove the `alloc_jemalloc` crate

This commit removes the `alloc_jemalloc` crate from the standard library and all related configuration. We will no longer be shipping this unstable crate. Rationale for this is provided on #36963 and the many linked issues, but I can inline rationale here if desired!

We currently rely on jemalloc for increased perf in the Rust compiler, however. [This perf run shows](https://perf.rust-lang.org/compare.html?start=74ff7dcb1388e60a613cd6050bcd372a3cc4998b&end=7e7928dc0340d79b404e93f0c79eb4b946c1d669&stat=wall-time) that if we switch to glibc 2.23's allocator that it's slower than jemalloc across many benchmarks. [This perf run, however](https://perf.rust-lang.org/compare.html?start=22cc2ae8057d14e980b7c784e1eb2eee26b59e7d&end=10c95ccfa7a7adc12f4e608621ca29f9b98eed29), shows that if we use `jemalloc-sys` from crates.io then rustc actually gets faster across all benchmarks! (presumably because it has a more recent version of jemalloc than our submodule).

As a result, it's expected that this doesn't regress any code (as it's just removing an unstable crate) and it should actually improve rustc performance because it updates jemalloc.

Closes #36963
@bors

This comment has been minimized.

Contributor

bors commented Nov 3, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: estebank
Pushing b6e8f9d to master...

@jonhoo

This comment has been minimized.

Contributor

jonhoo commented Nov 3, 2018

Just for clarity, does this change just affect rustc, or does it also affect all crates? Since it closes #36963, does that mean that all crates will now be compiled with the system memory allocator, or are we just moving from alloc_jemalloc to jemalloc-sys?

@@ -2464,6 +2504,11 @@ impl OngoingCodegen {
}
}
// impl Drop for OngoingCodegen {

This comment has been minimized.

@gnzlbg

gnzlbg Nov 3, 2018

Contributor

@alexcrichton what's this?

This comment has been minimized.

@alexcrichton

alexcrichton Nov 3, 2018

Member

Ah this is a mistake

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Nov 3, 2018

Yes, System is now the default on all platforms for all crate types (such as executable). It’s only rustc that this PR makes link to jemalloc-sys. I’ve written up an announcement:

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Nov 3, 2018

@jonhoo this affects all crates, no crate will be compiled against jemalloc by default now. The compilers we distribute for OSX/Linux will, however, still enable jemalloc (but only the compiler itself)

@nnethercote

This comment has been minimized.

Contributor

nnethercote commented Nov 12, 2018

Here are the final perf results from the landing. Almost everything improved by at least 1%, and some were up to 15%. Pretty great!

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Nov 12, 2018

This is most likely from upgrading jemalloc from 4.5 to 5.1, isn’t it?

@lnicola

This comment has been minimized.

Contributor

lnicola commented Nov 12, 2018

Are there plans to do something about the memory usage regression?

@gnzlbg

This comment has been minimized.

Contributor

gnzlbg commented Nov 12, 2018

@lnicola the jemalloc-sys master branch has re-added support for enabling background threads unconditionally, which might help here - there is also an open issue about gathering statistics of rustc size-class usage which could help here too (alexcrichton/jemallocator#79).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment