rustc: Allow changing the default allocator #27400

alexcrichton · Jul 30, 2015

This commit is an implementation of RFC 1183 which allows swapping out
the default allocator on nightly Rust. No new stable surface area should be
added as a part of this commit.

Two new attributes have been added to the compiler:

#![needs_allocator] - this is used by liballoc (and likely only liballoc) to
indicate that it requires an allocator crate to be in scope.
#![allocator] - this is a indicator that the crate is an allocator which can
satisfy the needs_allocator attribute above.

The ABI of the allocator crate is defined to be a set of symbols that implement
the standard Rust allocation/deallocation functions. The symbols are not
currently checked for exhaustiveness or typechecked. There are also a number of
restrictions on these crates:

An allocator crate cannot transitively depend on a crate that is flagged as
needing an allocator (e.g. allocator crates can't depend on liballoc).
There can only be one explicitly linked allocator in a final image.
If no allocator is explicitly requested one will be injected on behalf of the
compiler. Binaries and Rust dylibs will use jemalloc by default where
available and staticlibs/other dylibs will use the system allocator by
default.

Two allocators are provided by the distribution by default, alloc_system and
alloc_jemalloc which operate as advertised.

Closes #27389

rust-highfive · Jul 30, 2015

r? @pcwalton

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · Jul 30, 2015

r? @brson

cc @nnethercote

pnkfelix · Jul 30, 2015

pnkfelix reviewed Jul 30, 2015

View changes

pnkfelix · Jul 30, 2015

src/librustc/metadata/creader.rs

@@ -49,15 +49,16 @@ impl<'a, 'v> visit::Visitor<'v> for CrateReader<'a> {
 }

 fn dump_crates(cstore: &CStore) {
-    debug!("resolved crates:");


Was the switch from debug! to info! (and likewise from log::DEBUG to log::INFO) deliberate, or an accident?

I would have thought we could (and should) leave it as it was, under debug!

Yeah this was currently intentional but I wouldn't mind changing it back. Whenever someone's having weird behavior with loading crates I typically ask them to set RUST_LOG=rustc::metadata::creader,rustc::metadata::loader to get info like this, but debug! means it's all compiled out by default.

Along those lines I'd somewhat prefer to have it be info! but I don't have much of a preference either way

nnethercote · Jul 30, 2015

@alexcrichton: can I ask what the code would look like for a program (such as Servo) to implement allocation wrappers? I know we talked about this at Whistler but I don't remember the details and now that you have an implementation to reference it might be easier to describe. Thank you.

alexcrichton · Jul 30, 2015

Certainly! You'd basically just create a copy of src/liballoc_jemalloc/lib.rs, filling in different implementations for the various allocation functions. You'd then just say extern crate foo to link in your allocator and you'd be good to go!

retep998 · Jul 30, 2015

retep998 reviewed Jul 30, 2015

View changes

retep998 · Jul 30, 2015

src/liballoc_system/lib.rs

+mod imp {
+    use core::mem::size_of;
+    use libc::{BOOL, DWORD, HANDLE, LPVOID, SIZE_T, INVALID_HANDLE_VALUE};
+    use libc::{WriteFile};


WriteFile and INVALID_HANDLE_VALUE aren't needed.

retep998 · Jul 30, 2015

retep998 reviewed Jul 30, 2015

View changes

retep998 · Jul 30, 2015

src/liballoc_system/lib.rs

+    struct Header(*mut u8);
+
+    const HEAP_REALLOC_IN_PLACE_ONLY: DWORD = 0x00000010;
+    const STD_OUTPUT_HANDLE: DWORD = -11i32 as u32;


Also STD_OUTPUT_HANDLE isn't needed anymore.

retep998 · Jul 30, 2015

retep998 reviewed Jul 30, 2015

View changes

retep998 · Jul 30, 2015

src/liballoc_system/lib.rs

+
+    extern "system" {
+        fn GetProcessHeap() -> HANDLE;
+        fn GetStdHandle(nStdHandle: DWORD) -> HANDLE;


GetStdHandle isn't needed.

retep998 · Jul 31, 2015

retep998 reviewed Jul 31, 2015

View changes

retep998 · Jul 31, 2015

src/liballoc_system/lib.rs

+
+#[cfg(windows)]
+mod imp {
+    use core::mem::size_of;


size_of is also unused it seems.

brson · Jul 31, 2015

This feels very complex, makes me want to remove jemalloc.

retep998 · Jul 31, 2015

Considering how jemalloc is responsible for #26647 and is also really brittle when used under conemu #14600 (comment).

You know, this reminds me of the time we had green threaded IO, then split it using virtual dispatch between a green and a native portion, then later dropped the green part entirely.

steveklabnik · Jul 31, 2015

jemalloc tends to have _way_ better performance than libc's malloc or the system malloc, though. At least, in my understanding...

alexcrichton · Jul 31, 2015

@brson

This feels very complex, makes me want to remove jemalloc.

Could you elaborate a bit on this? I can certainly more aggressively document various aspects and I certainly wouldn't mind rewriting various code paths. I don't think that we want to just look at this and jettison jemalloc, though, as there are very real other use cases for swapping out for custom allocators (such as Servo's memory tracking, embedded systems with different allocators, etc).

If we followed reasoning like this I would also say that supporting both dylibs/rlibs at the same time is pretty complex, but I don't feel the need to remove that distinction. There's concrete use cases for both so we just need to support both in as robust a manner as possible.

@retep998

You know, this reminds me of the time we had green threaded IO, then split it using virtual dispatch between a green and a native portion, then later dropped the green part entirely.

I don't think that this is the same situation as here most notably because there's no virtual dispatch here. This is a static decision which is made that has no costs associated with it (unlike the virtual dispatch).

Like I said above, complexity is not a reason to jettison something, it's a reason to rethink why it's so complex and consider other options, but in this case I don't consider removing jemalloc an option.

If there are bugs on Windows caused by our usage of jemalloc (I was unaware of these), then we should track them down and fix them, but removing it from all platforms is a pretty heavy hammer.

alexcrichton · Jul 31, 2015

I'm currently running a crater build for this PR

alexcrichton · Aug 1, 2015

Crater reports one regression but I have confirmed locally that it was spurious.

nnethercote · Aug 2, 2015

A few comments...

The ability to provide custom allocators is very important and useful.
In the context of Rust, I've heard it said many times that jemalloc is faster, but I haven't seen any measurements showing this. (Hmm, now that I think of it, somebody may have mentioned that using jemalloc sped up rustc significantly? Not sure.)
Servo uses a mixture of jemalloc (for Rust code) and the system heap (for C/C++ code, e.g. SpiderMonkey). Servo's memory reporting code thus has machinery for dealing with both: it reports the total size of both heaps, and it uses je_malloc_usable_size to measure jemalloc heap blocks and malloc_usable_size (on Linux) or malloc_size (on Mac) to measure system heap blocks.

bors · Aug 3, 2015

☔️ The latest upstream changes (presumably #27210) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 5, 2015

☔️ The latest upstream changes (presumably #27393) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 5, 2015

☔️ The latest upstream changes (presumably #27458) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · Aug 13, 2015

@bors: r=brson abd839

bors · Aug 13, 2015

🙀 abd839 is not a valid commit SHA. Please try again with adbd839.

alexcrichton · Aug 13, 2015

@bors: r=brson adbd839

bors · Aug 14, 2015

🔒 Merge conflict

bors · Aug 14, 2015

☔️ The latest upstream changes (presumably #27684) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · Aug 14, 2015

@bors: r=brson 9d21851

bors · Aug 14, 2015

⌛️ Testing commit 9d21851 with merge 7959211...

bors · Aug 14, 2015

💔 Test failed - auto-linux-64-x-android-t

alexcrichton · Aug 14, 2015

@bors: retry

On Fri, Aug 14, 2015 at 12:19 PM, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-linux-64-x-android-t
http://buildbot.rust-lang.org/builders/auto-linux-64-x-android-t/builds/6061

—
Reply to this email directly or view it on GitHub
#27400 (comment).

bors · Aug 14, 2015

⌛️ Testing commit 9d21851 with merge d5f5dca...

bors · Aug 14, 2015

💔 Test failed - auto-linux-64-x-android-t

alexcrichton · Aug 14, 2015

@bors: r=brson 45bf1ed

bors · Aug 14, 2015

⌛️ Testing commit 45bf1ed with merge 290cf6b...

alexcrichton · Aug 14, 2015

@bors: retry force

bors · Aug 14, 2015

⌛️ Testing commit 45bf1ed with merge ab450ef...

bors · Aug 15, 2015

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

rust-highfive assigned pcwalton Jul 30, 2015

rust-highfive assigned brson and unassigned pcwalton Jul 30, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently to 8517834 Jul 30, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 8517834 to 6b95b66 Jul 30, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 6b95b66 to 5dbf9be Jul 31, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 5dbf9be to 31b083f Aug 4, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently from 9e1e50d to 49632c0 Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 49632c0 to 234b1bc Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from adbd839 to 9d21851 Aug 14, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 9d21851 to 45bf1ed Aug 14, 2015

bors merged commit 45bf1ed into rust-lang:master Aug 15, 2015
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

bors referenced this pull request Aug 15, 2015
Merged
SIMD groundwork part 1 #27169
3 of 5 tasks complete

alexcrichton deleted the alexcrichton:less-jemalloc branch Aug 17, 2015

geofft referenced this pull request Aug 31, 2015
Closed
Update patches for current Rust #14

apasel422 referenced this pull request Sep 4, 2015
Closed
Valgrind doesn't report allocations with jemalloc on nightly #28224

dherman referenced this pull request Sep 21, 2015
Closed
default to system rust when stable is ready #2

SimonSapin referenced this pull request Apr 18, 2018
Closed
Tracking issue for changing the global, default allocator (RFC 1974) #27389
2 of 7 tasks complete

rust-lang/rust

Join GitHub today

rustc: Allow changing the default allocator #27400

Conversation

alexcrichton commented Jul 30, 2015

rust-highfive assigned pcwalton Jul 30, 2015

This comment has been minimized.

rust-highfive commented Jul 30, 2015

This comment has been minimized.

alexcrichton commented Jul 30, 2015

rust-highfive assigned brson and unassigned pcwalton Jul 30, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently to 8517834 Jul 30, 2015

pnkfelix reviewed Jul 30, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nnethercote commented Jul 30, 2015

This comment has been minimized.

alexcrichton commented Jul 30, 2015

retep998 reviewed Jul 30, 2015 View changes

This comment has been minimized.

retep998 reviewed Jul 30, 2015 View changes

This comment has been minimized.

retep998 reviewed Jul 30, 2015 View changes

This comment has been minimized.

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 8517834 to 6b95b66 Jul 30, 2015

retep998 reviewed Jul 31, 2015 View changes

This comment has been minimized.

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 6b95b66 to 5dbf9be Jul 31, 2015

This comment has been minimized.

brson commented Jul 31, 2015

This comment has been minimized.

retep998 commented Jul 31, 2015

This comment has been minimized.

steveklabnik commented Jul 31, 2015

This comment has been minimized.

alexcrichton commented Jul 31, 2015

This comment has been minimized.

alexcrichton commented Jul 31, 2015

This comment has been minimized.

alexcrichton commented Aug 1, 2015

This comment has been minimized.

nnethercote commented Aug 2, 2015

This comment has been minimized.

bors commented Aug 3, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 5dbf9be to 31b083f Aug 4, 2015

This comment has been minimized.

bors commented Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently from 9e1e50d to 49632c0 Aug 5, 2015

This comment has been minimized.

bors commented Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 49632c0 to 234b1bc Aug 5, 2015

This comment has been minimized.

alexcrichton commented Aug 13, 2015

This comment has been minimized.

bors commented Aug 13, 2015

This comment has been minimized.

alexcrichton commented Aug 13, 2015

This comment has been minimized.

bors commented Aug 14, 2015

This comment has been minimized.

bors commented Aug 14, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from adbd839 to 9d21851 Aug 14, 2015

This comment has been minimized.

alexcrichton commented Aug 14, 2015

This comment has been minimized.

bors commented Aug 14, 2015

bors added a commit that referenced this pull request Aug 14, 2015

This comment has been minimized.

bors commented Aug 14, 2015

This comment has been minimized.

alexcrichton commented Aug 14, 2015

This comment has been minimized.

bors commented Aug 14, 2015

bors added a commit that referenced this pull request Aug 14, 2015

This comment has been minimized.

bors commented Aug 14, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from 9d21851 to 45bf1ed Aug 14, 2015

This comment has been minimized.

alexcrichton commented Aug 14, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently to `8517834` Jul 30, 2015

pnkfelix reviewed Jul 30, 2015

View changes

retep998 reviewed Jul 30, 2015

View changes

retep998 reviewed Jul 30, 2015

View changes

retep998 reviewed Jul 30, 2015

View changes

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `8517834` to `6b95b66` Jul 30, 2015

retep998 reviewed Jul 31, 2015

View changes

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `6b95b66` to `5dbf9be` Jul 31, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `5dbf9be` to `31b083f` Aug 4, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch 2 times, most recently from `9e1e50d` to `49632c0` Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `49632c0` to `234b1bc` Aug 5, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `adbd839` to `9d21851` Aug 14, 2015

alexcrichton force-pushed the alexcrichton:less-jemalloc branch from `9d21851` to `45bf1ed` Aug 14, 2015

bors merged commit `45bf1ed` into rust-lang:master Aug 15, 2015
2 checks passed