Add a GNU make jobserver implementation to Cargo #4110
Conversation
|
r? @brson (rust_highfive has picked a reviewer for you, use r? to override) |
|
r? @matklad |
|
When reviewing this, it may also be worth taking a brief look at the crate's source as well |
| // we're only sending "longer living" messages and we should also | ||
| // destroy all references to the channel before this function exits as | ||
| // the destructor for the `helper` object will ensure the associated | ||
| // thread i sno longer running. |
There was a problem hiding this comment.
Interesting, looks like this transmute is always safe, because Sender is contravariant.
There was a problem hiding this comment.
That's what I originally thought yeah but actually I don't think so (unfortunately). I believe Drop for Sender may run destructors for items in the internal queue (not received yet), which means that if you persist a Sender beyond the lifetime of the item I think it'll access data outside of its lifetime.
(not in this case though, the Sender should always go away with the stack frame.
src/cargo/util/config.rs
Outdated
| @@ -51,6 +53,9 @@ impl Config { | |||
| extra_verbose: Cell::new(false), | |||
| frozen: Cell::new(false), | |||
| locked: Cell::new(false), | |||
| // This should be called early on in the process, so in theory the | |||
| // unsafety is ok here. (taken ownership of random fds) | |||
| jobserver: unsafe { jobserver::Client::from_env() }, | |||
There was a problem hiding this comment.
There should be at most one job server, and it's not entirely obvious that we call Config::new exactly once. Perhaps, we can wrap jobserver creation in a jobserver_singleton function, which uses ONCE_INIT or lazy_static! internally?
There was a problem hiding this comment.
Oops excellent point
|
Ok I've pushed up w/ a |
|
☔ The latest upstream changes (presumably #4090) made this pull request unmergeable. Please resolve the merge conflicts. |
|
LGTM, but perhaps we want to add a test here? I understand that there's an impressive amount of tests in I am thinking of two useful test:
|
|
Oh, and what should we do if Cargo is launched with jobjserver, but also with an explicit |
|
All excellent suggestions! I've updated the PR (thanks so much for the thorough reviews!) |
|
r+, but looks like the CI failure on windows is legit: https://ci.appveyor.com/project/rust-lang-libs/cargo/build/1.0.1971#L271
Well, I learn a lot of stuff from them :) |
|
Looks like a corner case! |
This commit adds a GNU make jobserver implementation to Cargo, both as a client of existing jobservers and also a creator of new jobservers. The jobserver is actually just an IPC semaphore which manifests itself as a pipe with N bytes of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol is then if you want to run a job you read acquire the semaphore (read a byte on Unix or wait on the semaphore on Windows) and then you release it when you're done. All the hairy details of the jobserver implementation are housed in the `jobserver` crate on crates.io instead of Cargo. This should hopefully make it much easier for the compiler to also share a jobserver implementation eventually. The main tricky bit here is that on Unix and Windows acquiring a jobserver token will block the calling thread. We need to either way for a running job to exit or to acquire a new token when we want to spawn a new job. To handle this the current implementation spawns a helper thread that does the blocking and sends a message back to Cargo when it receives a token. It's a little trickier with shutting down this thread gracefully as well but more details can be found in the `jobserver` crate. Unfortunately crates are unlikely to see an immediate benefit of this once implemented. Most crates are run with a manual `make -jN` and this overrides the jobserver in the environment, creating a new jobserver in the sub-make. If the `-jN` argument is removed, however, then `make` will share Cargo's jobserver and properly limit parallelism. Closes rust-lang#1744
|
@bor: r=matklad |
|
@bors: r=matklad |
|
📌 Commit cbf25a9 has been approved by |
Add a GNU make jobserver implementation to Cargo This commit adds a GNU make jobserver implementation to Cargo, both as a client of existing jobservers and also a creator of new jobservers. The jobserver is actually just an IPC semaphore which manifests itself as a pipe with N bytes of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol is then if you want to run a job you read acquire the semaphore (read a byte on Unix or wait on the semaphore on Windows) and then you release it when you're done. All the hairy details of the jobserver implementation are housed in the `jobserver` crate on crates.io instead of Cargo. This should hopefully make it much easier for the compiler to also share a jobserver implementation eventually. The main tricky bit here is that on Unix and Windows acquiring a jobserver token will block the calling thread. We need to either way for a running job to exit or to acquire a new token when we want to spawn a new job. To handle this the current implementation spawns a helper thread that does the blocking and sends a message back to Cargo when it receives a token. It's a little trickier with shutting down this thread gracefully as well but more details can be found in the `jobserver` crate. Unfortunately crates are unlikely to see an immediate benefit of this once implemented. Most crates are run with a manual `make -jN` and this overrides the jobserver in the environment, creating a new jobserver in the sub-make. If the `-jN` argument is removed, however, then `make` will share Cargo's jobserver and properly limit parallelism. Closes #1744
|
☀️ Test successful - status-appveyor, status-travis |
This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.
Integrate jobserver support to parallel codegen This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.
Integrate jobserver support to parallel codegen This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.
This looks like it was a bug ever present from the original implementation of a GNU jobserver in rust-lang#4110, but we currently unconditionally request a token is allocated for any job we pull off our job queue. Rather we only need to request tokens for everything but the first job because we already have an implicit token for that job.
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
jobservercrate on crates.io instead of Cargo. This should hopefully make itmuch easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the
jobservercrate.Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual
make -jNand this overrides thejobserver in the environment, creating a new jobserver in the sub-make. If the
-jNargument is removed, however, thenmakewill share Cargo's jobserver andproperly limit parallelism.
Closes #1744