Skip to content

/cargo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a GNU make jobserver implementation to Cargo #4110

Merged
merged 1 commit into from Jun 2, 2017

Conversation

Reviewers

@matklad matklad

Assignees

@matklad matklad

Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@alexcrichton @rust-highfive @bors @matklad @brson
@alexcrichton
Copy link
Member

alexcrichton commented May 30, 2017

This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.

All the hairy details of the jobserver implementation are housed in the
jobserver crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.

The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the jobserver crate.

Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual make -jN and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
-jN argument is removed, however, then make will share Cargo's jobserver and
properly limit parallelism.

Closes #1744

@rust-highfive rust-highfive assigned brson May 30, 2017

@rust-highfive

This comment has been minimized.

Sign in to view
Copy link

rust-highfive commented May 30, 2017

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton alexcrichton referenced this pull request May 30, 2017

Closed

Add support for make jobservers #1744

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented May 30, 2017

r? @matklad

@rust-highfive rust-highfive assigned matklad and unassigned brson May 30, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented May 30, 2017

When reviewing this, it may also be worth taking a brief look at the crate's source as well
@matklad

matklad reviewed May 30, 2017
View changes

src/cargo/ops/cargo_rustc/job_queue.rs
// we're only sending "longer living" messages and we should also
// destroy all references to the channel before this function exits as
// the destructor for the `helper` object will ensure the associated
// thread i sno longer running.

This comment has been minimized.

Sign in to view
Copy link
@matklad

matklad May 30, 2017

Member

s/i sno/is no

This comment has been minimized.

Sign in to view
Copy link
@matklad

matklad May 30, 2017

Member

Interesting, looks like this transmute is always safe, because Sender is contravariant.

This comment has been minimized.

Sign in to view
Copy link
@alexcrichton

alexcrichton May 31, 2017

Author Member

That's what I originally thought yeah but actually I don't think so (unfortunately). I believe Drop for Sender may run destructors for items in the internal queue (not received yet), which means that if you persist a Sender beyond the lifetime of the item I think it'll access data outside of its lifetime.

(not in this case though, the Sender should always go away with the stack frame.
src/cargo/util/config.rs Outdated
@@ -51,6 +53,9 @@ impl Config {
extra_verbose: Cell::new(false),
frozen: Cell::new(false),
locked: Cell::new(false),
// This should be called early on in the process, so in theory the
// unsafety is ok here. (taken ownership of random fds)
jobserver: unsafe { jobserver::Client::from_env() },

This comment has been minimized.

Sign in to view
Copy link
@matklad

matklad May 30, 2017

Member

There should be at most one job server, and it's not entirely obvious that we call Config::new exactly once. Perhaps, we can wrap jobserver creation in a jobserver_singleton function, which uses ONCE_INIT or lazy_static! internally?

This comment has been minimized.

Sign in to view
Copy link
@alexcrichton

alexcrichton May 31, 2017

Author Member

Oops excellent point

@alexcrichton alexcrichton force-pushed the alexcrichton:jobserver branch from 892e7fb to 0e89a79 May 31, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented May 31, 2017

Ok I've pushed up w/ a ONCE_INIT + an update to the jobserver crate
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented May 31, 2017

☔️ The latest upstream changes (presumably #4090) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton force-pushed the alexcrichton:jobserver branch from 0e89a79 to 3220daf Jun 1, 2017

@matklad

This comment has been minimized.

Sign in to view
Copy link
Member

matklad commented Jun 1, 2017

LGTM, but perhaps we want to add a test here? I understand that there's an impressive amount of tests in jobserver-rs, but the amount of logic in Cargo is non-trivial, and we even have a transmute in here!

I am thinking of two useful test:

  1. Launch cargo itself with -j argument and print env in build.rs, to check that Cargo itself can setup a jobserver.

  2. Like 1, but launch cargo via make to check that it can be a client of jobserver (and we might want to put some logging when receiving tokens from jobserver, to check that we actually receive them).
@matklad

This comment has been minimized.

Sign in to view
Copy link
Member

matklad commented Jun 1, 2017

Oh, and what should we do if Cargo is launched with jobjserver, but also with an explicit -j argument? Looks like currently we just ignore -j altogether. Should we perhaps print a warning?

@alexcrichton alexcrichton force-pushed the alexcrichton:jobserver branch from 3220daf to 434177c Jun 1, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Jun 1, 2017

All excellent suggestions! I've updated the PR

(thanks so much for the thorough reviews!)
@matklad

This comment has been minimized.

Sign in to view
Copy link
Member

matklad commented Jun 2, 2017

r+, but looks like the CI failure on windows is legit: https://ci.appveyor.com/project/rust-lang-libs/cargo/build/1.0.1971#L271

thanks so much for the thorough reviews!

Well, I learn a lot of stuff from them :)
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Jun 2, 2017

Looks like a corner case!
@alexcrichton
Add a GNU make jobserver implementation to Cargo 
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.

All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.

The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.

Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.

Closes #1744
Loading status checks…
cbf25a9

@alexcrichton alexcrichton force-pushed the alexcrichton:jobserver branch from 434177c to cbf25a9 Jun 2, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Jun 2, 2017

@bor: r=matklad
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Jun 2, 2017

@bors: r=matklad
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Jun 2, 2017

📌 Commit cbf25a9 has been approved by matklad
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Jun 2, 2017

⌛️ Testing commit cbf25a9 with merge 8fb6e7c...

bors added a commit that referenced this pull request Jun 2, 2017

@bors
Auto merge of #4110 - alexcrichton:jobserver, r=matklad 
Add a GNU make jobserver implementation to Cargo

This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.

All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.

The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.

Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.

Closes #1744
Loading status checks…
8fb6e7c
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Jun 2, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: matklad
Pushing 8fb6e7c to master...

@bors bors merged commit cbf25a9 into rust-lang:master Jun 2, 2017
3 checks passed

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

@bors bors referenced this pull request Jun 2, 2017

Merged

Optimize a slew of Cargo internals #4118

@alexcrichton alexcrichton deleted the alexcrichton:jobserver branch Jun 6, 2017

@alexcrichton alexcrichton referenced this pull request Jun 13, 2017

Closed

MAKEFLAGS / MFLAGS env vars are set for nmake.exe which breaks it #4156

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
20aad92

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
d36e8c8

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
f1a9ba7

@alexcrichton alexcrichton referenced this pull request Jun 15, 2017

Merged

Integrate jobserver support to parallel codegen #42682

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
a618364

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
c765aad

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 15, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
4e8e13a

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 16, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
b364714

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 17, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
0f66436

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 19, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
5d00e5e

bors added a commit to rust-lang/rust that referenced this pull request Jun 20, 2017

@bors
Auto merge of #42682 - alexcrichton:jobserver, r=michaelwoerister 
Integrate jobserver support to parallel codegen

This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
6cb3b99

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 20, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
a014634

bors added a commit to rust-lang/rust that referenced this pull request Jun 20, 2017

@bors
Auto merge of #42682 - alexcrichton:jobserver, r=michaelwoerister 
Integrate jobserver support to parallel codegen

This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
0164969

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 20, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
2376e0d

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 21, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
451d392

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 21, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
46dc6da

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jun 21, 2017

@alexcrichton
Integrate jobserver support to parallel codegen 
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
201f069

bors added a commit to rust-lang/rust that referenced this pull request Jun 21, 2017

@bors
Auto merge of #42682 - alexcrichton:jobserver, r=michaelwoerister 
Integrate jobserver support to parallel codegen

This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
694adee

bors added a commit to rust-lang/rust that referenced this pull request Jun 22, 2017

@bors
Auto merge of #42682 - alexcrichton:jobserver, r=michaelwoerister 
Integrate jobserver support to parallel codegen

This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:

* Primarily the compiler can cooperate with Cargo on parallelism. When you run
  `cargo build -j4` then this'll make sure that the entire build process between
  Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
  instances which may all try to spawn lots of threads.

* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
  This means that if you call cargo/rustc from `make` or another
  jobserver-compatible implementation it'll use foreign parallelism settings
  instead of creating new ones locally.

As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!

Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
Loading status checks…
80271e8

@SimonSapin SimonSapin referenced this pull request Oct 11, 2017

Open

Better heuristics for build parallelism #13770

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.