Integrate jobserver support to parallel codegen #42682

alexcrichton · 2017-06-15T18:40:35Z

This commit integrates the jobserver crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

Primarily the compiler can cooperate with Cargo on parallelism. When you run
cargo build -j4 then this'll make sure that the entire build process between
Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
instances which may all try to spawn lots of threads.
Secondarily rustc/Cargo can now integrate with a foreign GNU make jobserver.
This means that if you call cargo/rustc from make or another
jobserver-compatible implementation it'll use foreign parallelism settings
instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.

rust-highfive · 2017-06-15T18:40:38Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2017-06-15T18:40:42Z

r? @michaelwoerister

michaelwoerister

Very nice! I'm excited about this :)

Regarding the implementation, I'm not quite clear on how token handling works there. Wouldn't it be easier to just move one token into each spawn_work and let it go out of scope there?

michaelwoerister · 2017-06-16T09:26:25Z

src/librustc_trans/back/lto.rs

@@ -82,16 +84,11 @@ pub fn run(sess: &session::Session,
    // For each of our upstream dependencies, find the corresponding rlib and
    // load the bitcode from the archive. Then merge it into the current LLVM
    // module that we've got.
-    link::each_linked_rlib(sess, &mut |cnum, path| {
-        // `#![no_builtins]` crates don't participate in LTO.
-        if sess.cstore.is_no_builtins(cnum) {


Did you remove this on purpose?

I see, it's added in later again.

Yeah this query just ended up having a lot of dependencies on sess so I figured it'd be best to move it way up to the beginning instead of only running it back here.

michaelwoerister · 2017-06-16T09:42:43Z

src/librustc_trans/back/write.rs

-                        execute_work_item(&cgcx, work);
+    let mut tokens = Vec::new();
+    let mut running = 0;
+    while work_items.len() > 0 || running > 0 {


Could you add a comment here saying something to the effect of "This is our 'main loop', taking care of spawning worker threads and communicating with live ones via message passing -- so we have to keep it running as long as there's still work that hasn't been doled out to a worker (work_items > 0) or if there are still live workers to be communicated with (running > 0)."

michaelwoerister · 2017-06-16T10:02:18Z

src/librustc_trans/back/write.rs

+                       scope,
+                       tx.clone(),
+                       work_items.pop().unwrap(),
+                       work_items.len());


I'm not very fond this: mutating work_items via pop and then taking its len. I assume that we have a defined evaluation order of function arguments, but I don't like relying on it.

michaelwoerister · 2017-06-16T10:03:01Z

src/librustc_trans/back/write.rs

+        // possible. Remember that we have an ambient token available to us
+        // hence the `+1` here.
+        //
+        // Also note that we may actually acquire more tokens than we need, so


When does that happen? If we abort early because of an error?

michaelwoerister · 2017-06-16T10:03:57Z

src/librustc_trans/back/write.rs

+        //
+        // Also note that we may actually acquire more tokens than we need, so
+        // in that case just truncate the `tokens` list every time we pass
+        // through here.


Could you add that truncating implies dropping and thus releasing tokens?

michaelwoerister · 2017-06-16T10:20:51Z

src/librustc_trans/back/write.rs

+                       work_items.len());
+            running += 1;
+        }
+        tokens.truncate(running.saturating_sub(1));


I'm not quite sure how this works. Can't this cause tokens to be lost without a spawn_work having been called for them?

michaelwoerister · 2017-06-16T10:23:08Z

src/librustc_trans/back/write.rs

+
+        // Set up a destructor which will fire off a message that we're done as
+        // we exit.
+        struct Bomb {


We should have something like this in libstd.

michaelwoerister · 2017-06-16T10:27:08Z

src/librustc_trans/back/write.rs

+        if sess.cstore.is_no_builtins(cnum) {
+            return
+        }
+        each_linked_rlib.push((cnum, path.to_path_buf()));


If the each_linked_rlib field is LTO-specific, we should probably change to the name to reflect this.

michaelwoerister · 2017-06-16T10:29:23Z

src/librustc_trans/back/write.rs

+        // Execute the work itself, and if it finishes successfully then flag
+        // ourselves as a success as well.
+        if execute_work_item(&cgcx, work).is_err() {
+            drop(cgcx.tx.send(Message::AbortIfErrors));


One could argue that it would be cleaner to also mem::forget the bomb in this case.

Yeah I wasn't quite sure how this should be handled, I think that if you see a FatalError then a diagnostic has already been sent off, which in turn already sent AbortIfErrors. In that sense it may be fruitless to send another message here, so I'll just ignore the result.

michaelwoerister · 2017-06-16T10:30:33Z

src/librustc_trans/scope.rs

+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+//! Scoped threads, copied from `crossbeam`


Could we also use crossbeam directly?

Hm upon further inspection, I don't see why not!

alexcrichton · 2017-06-16T15:27:47Z

Ok, updated! @michaelwoerister I added a large comment above the "main loop" which I believe should answer your questions about the token management, but if you'd like me to clarify anything please just let me know!

bors · 2017-06-18T23:25:12Z

☔ The latest upstream changes (presumably #42676) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister · 2017-06-19T08:12:25Z

src/librustc_trans/back/write.rs

+    // manner we can ensure that the maximum number of parallel workers is
+    // capped at any one point in time.
+    //
+    // The jobserver protocol is a little unique, however. We, as a running


Because concurrent programming isn't complicated enough by itself already 😛

michaelwoerister · 2017-06-19T08:32:52Z

Thanks for the clarifying comment about the jobserver protocol!

r=me once the merge conflict is fixed.

alexcrichton · 2017-06-19T14:19:37Z

@bors: r=michaelwoerister

bors · 2017-06-19T14:19:39Z

📌 Commit 5d00e5e has been approved by michaelwoerister

bors · 2017-06-20T12:24:52Z

⌛ Testing commit 5d00e5ea2892d376aa18f8db2e4db435e097a81a with merge 6cb3b992db1aa5c9952c187dca9b30ec0c3d98d4...

bors · 2017-06-20T13:25:18Z

💔 Test failed - status-appveyor

alexcrichton · 2017-06-20T15:55:37Z

@bors: r=michaelwoerister

bors · 2017-06-20T15:55:39Z

📌 Commit a014634 has been approved by michaelwoerister

bors · 2017-06-20T19:34:42Z

⌛ Testing commit a014634c1a0ee939207dcd1e8d64dfcd0ebec586 with merge 016496955016d5d75d27180c10a158aad0083c8d...

bors · 2017-06-20T21:08:25Z

💔 Test failed - status-appveyor

alexcrichton · 2017-06-21T14:02:09Z

@bors: r=michaelwoerister

bors · 2017-06-21T14:02:10Z

📌 Commit 451d392 has been approved by michaelwoerister

This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.

alexcrichton · 2017-06-21T14:17:00Z

@bors: r=michaelwoerister

bors · 2017-06-21T14:17:02Z

📌 Commit 201f069 has been approved by michaelwoerister

bors · 2017-06-21T18:22:19Z

⌛ Testing commit 201f069 with merge 694adee...

Integrate jobserver support to parallel codegen This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.

bors · 2017-06-21T21:24:05Z

💔 Test failed - status-travis

alexcrichton · 2017-06-21T21:31:33Z

@bors: retry * osx timed out

…

On Wed, Jun 21, 2017 at 4:24 PM, bors ***@***.***> wrote: 💔 Test failed - status-travis <https://travis-ci.org/rust-lang/rust/builds/245482773?utm_source=github_status&utm_medium=notification> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#42682 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD95FEbgYHMXulmMHTszBVJR6hpkE9Lks5sGYoOgaJpZM4N7l7L> .

bors · 2017-06-22T00:32:48Z

⌛ Testing commit 201f069 with merge 80271e8...

Integrate jobserver support to parallel codegen This commit integrates the `jobserver` crate into the compiler. The crate was previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose here is to two-fold: * Primarily the compiler can cooperate with Cargo on parallelism. When you run `cargo build -j4` then this'll make sure that the entire build process between Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc instances which may all try to spawn lots of threads. * Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver. This means that if you call cargo/rustc from `make` or another jobserver-compatible implementation it'll use foreign parallelism settings instead of creating new ones locally. As the number of parallel codegen instances in the compiler continues to grow over time with the advent of incremental compilation it's expected that this'll become more of a problem, so this is intended to nip concurrent concerns in the bud by having all the tools to cooperate! Note that while rustc has support for itself creating a jobserver it's far more likely that rustc will always use the jobserver configured by Cargo. Cargo today will now set a jobserver unconditionally for rustc to use.

bors · 2017-06-22T02:56:10Z

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing 80271e8 to master...

jdm · 2017-06-28T07:51:13Z

So does this only support invoking cargo/rustc from make, but the behaviour of invoking make from a Cargo build script is unchanged?

alexcrichton · 2017-06-30T06:52:30Z

@jdm it's a little more nuanced than that. Cargo also creates a jobserver in addition to consuming one, meaning that rustc will basically always use that jobserver now. If Cargo inherits a jobserver though then rustc likely will too.

You need to tweak makefiles calling rustc/cargo though to actually let them inherit the jobserver, notably adding a + to the beginning of the rule definition.

For build scripts invoking make the make subprocess will inherit Cargo's jobserver if no -j argument is passed, but if -jN is passed then that'll override the inherited jobserver.

This should significantly speed up debug and test builds + cargo check. With rust-lang/rust#42682, cargo/rustc no longer spawns lots and lots of workers even when called recursively. Still not enabled by default in release mode: https://internals.rust-lang.org/t/help-test-out-thinlto/6017

…atsakis Remove some `ignore-stage1` annotations. These tests appear to no longer need the `ignore-stage1` marker. - `run-make-fulldeps/issue-37839` and `run-make-fulldeps/issue-37893`: I believe these were due to the use of proc-macros, and probably were just missed in rust-lang#49219 which fixed the proc-macro compatibility. - `compile-fail/asm-src-loc-codegen-units.rs`: This was due to an old issue with landing pads (as mentioned in the linked issue rust-lang#20184). `-Zno-landing-pads` was an option when building the first stage (it was much faster), but somewhere along the way (I think the switch from makefiles to rustbuild), the option was removed. - NOTE: This test doesn't actually test what it was originally written for, and is probably mostly pointless now. This test was asserting the message "build without -C codegen-units for more exact errors", but that was removed in rust-lang#42682. It is now in essence identical to `asm-src-loc.rs`.

rust-highfive assigned arielb1 Jun 15, 2017

rust-highfive assigned michaelwoerister and unassigned arielb1 Jun 15, 2017

alexcrichton force-pushed the jobserver branch 3 times, most recently from c765aad to 4e8e13a Compare June 15, 2017 20:01

michaelwoerister reviewed Jun 16, 2017

View reviewed changes

shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 16, 2017

alexcrichton force-pushed the jobserver branch from 4e8e13a to b364714 Compare June 16, 2017 15:27

alexcrichton force-pushed the jobserver branch from b364714 to 0f66436 Compare June 17, 2017 19:13

michaelwoerister reviewed Jun 19, 2017

View reviewed changes

alexcrichton force-pushed the jobserver branch from 0f66436 to 5d00e5e Compare June 19, 2017 14:19

alexcrichton force-pushed the jobserver branch from 5d00e5e to a014634 Compare June 20, 2017 15:55

arielb1 added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 20, 2017

alexcrichton force-pushed the jobserver branch from 451d392 to 46dc6da Compare June 21, 2017 14:16

alexcrichton force-pushed the jobserver branch from 46dc6da to 201f069 Compare June 21, 2017 14:16

bors merged commit 201f069 into rust-lang:master Jun 22, 2017

This was referenced Jun 22, 2017

rustc: Implement stack probes for x86 #42816

Merged

rustc: Implement the #[global_allocator] attribute #42727

Merged

alexcrichton deleted the jobserver branch June 22, 2017 04:48

rillian mentioned this pull request Jun 23, 2017

ICE compiling rustc-serialize under sccache 'failed to acquire jobserver token' #42867

Closed

jdm mentioned this pull request Jun 30, 2017

Use cargo's jobserver instead of specifying -j manually servo/mozjs#120

Closed

ehuss mentioned this pull request Jul 3, 2020

Remove some ignore-stage1 annotations. #73981

Merged

gouenji-shuuya mentioned this pull request Mar 26, 2023

Make process improvements ambuda-org/vidyut#59

Open

mistmist mentioned this pull request Apr 27, 2023

Develop/Document multi-level parallelism policy commercialhaskell/stack#644

Closed

Integrate jobserver support to parallel codegen #42682

Integrate jobserver support to parallel codegen #42682

Conversation

alexcrichton commented Jun 15, 2017

rust-highfive commented Jun 15, 2017

alexcrichton commented Jun 15, 2017

michaelwoerister left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Jun 16, 2017

bors commented Jun 18, 2017

Choose a reason for hiding this comment

michaelwoerister commented Jun 19, 2017

alexcrichton commented Jun 19, 2017

bors commented Jun 19, 2017

bors commented Jun 20, 2017

bors commented Jun 20, 2017

alexcrichton commented Jun 20, 2017

bors commented Jun 20, 2017

bors commented Jun 20, 2017

bors commented Jun 20, 2017

alexcrichton commented Jun 21, 2017

bors commented Jun 21, 2017

alexcrichton commented Jun 21, 2017

bors commented Jun 21, 2017

bors commented Jun 21, 2017

bors commented Jun 21, 2017

alexcrichton commented Jun 21, 2017 via email

bors commented Jun 22, 2017

bors commented Jun 22, 2017

jdm commented Jun 28, 2017

alexcrichton commented Jun 30, 2017