Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable pipelined compilation by default #7143

Merged
merged 1 commit into from Jul 31, 2019

Conversation

@alexcrichton
Copy link
Member

commented Jul 17, 2019

This commit enables pipelined compilation by default in Cargo now that
the requisite support has been stablized in rust-lang/rust#62766. This
involved minor updates in a number of locations here and there, but
nothing of meat has changed from the original implementation (just
tweaks to how rustc is called).

@rust-highfive

This comment has been minimized.

Copy link

commented Jul 17, 2019

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jul 17, 2019

rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in rust-lang#60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes rust-lang#60419
Closes rust-lang#60987
Closes rust-lang#60988

@alexcrichton alexcrichton force-pushed the alexcrichton:stabilize-pipelined-compilation branch from dbd6ef2 to 2fd1f06 Jul 17, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jul 17, 2019

r? @ehuss

(note that CI will fail and this can't land until rust-lang/rust#62766 propagates to nightly)

@rust-highfive rust-highfive assigned ehuss and unassigned nrc Jul 17, 2019

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jul 18, 2019

rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in rust-lang#60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes rust-lang#60419
Closes rust-lang#60987
Closes rust-lang#60988

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jul 24, 2019

rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in rust-lang#60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes rust-lang#60419
Closes rust-lang#60987
Closes rust-lang#60988

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jul 24, 2019

rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in rust-lang#60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes rust-lang#60419
Closes rust-lang#60987
Closes rust-lang#60988

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jul 26, 2019

rustc: Stabilize options for pipelined compilation
This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in rust-lang#60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes rust-lang#60419
Closes rust-lang#60987
Closes rust-lang#60988

bors added a commit to rust-lang/rust that referenced this pull request Jul 30, 2019

Auto merge of #62766 - alexcrichton:stabilize-pipelined-compilation, …
…r=oli-obk

rustc: Stabilize options for pipelined compilation

This commit stabilizes options in the compiler necessary for Cargo to
enable "pipelined compilation" by default. The concept of pipelined
compilation, how it's implemented, and what it means for rustc are
documented in #60988. This PR is coupled with a PR against Cargo
(rust-lang/cargo#7143) which updates Cargo's support for pipelined
compliation to rustc, and also enables support by default in Cargo.
(note that the Cargo PR cannot land until this one against rustc lands).

The technical changes performed here were to stabilize the functionality
proposed in #60419 and #60987, the underlying pieces to enable pipelined
compilation support in Cargo. The issues have had some discussion during
stabilization, but the newly stabilized surface area here is:

* A new `--json` flag was added to the compiler.
* The `--json` flag can be passed multiple times.
* The value of the `--json` flag is a comma-separated list of
  directives.
* The `--json` flag cannot be combined with `--color`
* The `--json` flag must be combined with `--error-format=json`
* The acceptable list of directives to `--json` are:
  * `diagnostic-short` - the `rendered` field of diagnostics will have a
    "short" rendering matching `--error-format=short`
  * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics
    will be colorized with ansi color codes embedded in the string field
  * `artifacts` - JSON blobs will be emitted for artifacts being emitted
    by the compiler

The unstable `-Z emit-artifact-notifications` and `--json-rendered`
flags have also been removed during this commit as well.

Closes #60419
Closes #60987
Closes #60988

@alexcrichton alexcrichton force-pushed the alexcrichton:stabilize-pipelined-compilation branch from 2fd1f06 to 09de563 Jul 30, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jul 30, 2019

Ok rust-lang/rust#62766 has landed, I've rebased this and updated all failing tests. This will fail CI since the compiler isn't on the nightly channel yet, but @ehuss if you're up for it this should be good for review!

@ehuss

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

I restarted CI, and it looks like stable/beta does not pass.

@alexcrichton alexcrichton force-pushed the alexcrichton:stabilize-pipelined-compilation branch from 09de563 to 0adbf0a Jul 31, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jul 31, 2019

Ah good point, I've updated the PR to just ignore those tests on non-nightly builders for now. Both of their assertions are tweaked when pipelining is enabled (both tests still morally pass on all compilers, it's just a difference of what should be asserted) and looking into the tests it seems not worth it to keep them running on stable/beta. Once pipelining rides to stable and we can remove the feature detection for that rustc flag we can update these tests to run on all channels again.

@@ -511,6 +511,6 @@ fn canonical_path() {
assert_deps_contains(
&p,
"target/debug/.fingerprint/foo-*/dep-lib-foo-*",
&[(1, "src/lib.rs"), (2, "debug/deps/libregdep-*.rlib")],
&[(1, "src/lib.rs"), (2, "debug/deps/libregdep-*.r*")],

This comment has been minimized.

Copy link
@ehuss

ehuss Jul 31, 2019

Contributor

Can we just change this to .rmeta, or at least leave a comment why the * is there?

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Jul 31, 2019

Author Member

Sure yeah, the r* is to handle rlib on stable/beta but rmeta on nightly. I'll just switch this to a nightly-only test though and make it rmeta

@ehuss

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

r=me with minor nit addressed.

Enable pipelined compilation by default
This commit enables pipelined compilation by default in Cargo now that
the requisite support has been stablized in rust-lang/rust#62766. This
involved minor updates in a number of locations here and there, but
nothing of meat has changed from the original implementation (just
tweaks to how rustc is called).

@alexcrichton alexcrichton force-pushed the alexcrichton:stabilize-pipelined-compilation branch from 0adbf0a to daa1bce Jul 31, 2019

@alexcrichton

This comment has been minimized.

Copy link
Member Author

commented Jul 31, 2019

@bors: r=ehuss

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

📌 Commit daa1bce has been approved by ehuss

@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

⌛️ Testing commit daa1bce with merge b21602c...

bors added a commit that referenced this pull request Jul 31, 2019

Auto merge of #7143 - alexcrichton:stabilize-pipelined-compilation, r…
…=ehuss

Enable pipelined compilation by default

This commit enables pipelined compilation by default in Cargo now that
the requisite support has been stablized in rust-lang/rust#62766. This
involved minor updates in a number of locations here and there, but
nothing of meat has changed from the original implementation (just
tweaks to how rustc is called).
@bors

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing b21602c to master...

@bors bors merged commit daa1bce into rust-lang:master Jul 31, 2019

10 checks passed

homu Test successful
Details
rust-lang.cargo Build #20190731.7 succeeded
Details
rust-lang.cargo (Linux beta) Linux beta succeeded
Details
rust-lang.cargo (Linux nightly) Linux nightly succeeded
Details
rust-lang.cargo (Linux stable) Linux stable succeeded
Details
rust-lang.cargo (Windows x86_64-msvc) Windows x86_64-msvc succeeded
Details
rust-lang.cargo (docs) docs succeeded
Details
rust-lang.cargo (macOS) macOS succeeded
Details
rust-lang.cargo (resolver) resolver succeeded
Details
rust-lang.cargo (rustfmt) rustfmt succeeded
Details

@alexcrichton alexcrichton deleted the alexcrichton:stabilize-pipelined-compilation branch Jul 31, 2019

bors added a commit to rust-lang/rust that referenced this pull request Aug 1, 2019

Auto merge of #63181 - ehuss:update-cargo-rls, r=alexcrichton
Update cargo, rls

## cargo

12 commits in d0f828419d6ce6be21a90866964f58eb2c07cd56..26092da337b948719549cd5ed3d1051fd847afd7
2019-07-23 21:58:59 +0000 to 2019-07-31 23:24:32 +0000
- tests: Enable features to fix unstabilized `#[bench]` (rust-lang/cargo#7198)
- Fix excluding target dirs from backups on OSX (rust-lang/cargo#7192)
- Handle symlinks to directories (rust-lang/cargo#6817)
- Enable pipelined compilation by default (rust-lang/cargo#7143)
- Refactor resolve `Method` (rust-lang/cargo#7186)
- Update `cargo_compile` module doc. (rust-lang/cargo#7187)
- Clean up TargetInfo (rust-lang/cargo#7185)
- Fix some issues with absolute paths in dep-info files. (rust-lang/cargo#7137)
- Update the `url` crate to 2.0 (rust-lang/cargo#7175)
- Tighten requirements for git2 crates (rust-lang/cargo#7176)
- Fix a deadlocking test with master libgit2 (rust-lang/cargo#7179)
- Fix detection of cyclic dependencies through `[patch]` (rust-lang/cargo#7174)

## rls

1 commits in 70347b5d4dfe78eeb9e6f6db85f773c8d43cd22b..93d9538c6000fcf6c8da763ef4ce7a8d407b7d24
2019-07-30 12:56:38 +0200 to 2019-07-31 21:42:49 +0200
- Update cargo (rust-lang/rls#1529)

alexcrichton added a commit to alexcrichton/cargo that referenced this pull request Aug 5, 2019

Add support for customizing JSON diagnostics from Cargo
Cargo has of rust-lang#7143 enabled pipelined compilation by default which
affects how the compiler is invoked, especially with respect to JSON
messages. This, in some testing, has proven to cause quite a few issues
with rustbuild's current integration with Cargo. This commit is aimed at
adding features to Cargo to solve this issue.

This commit adds the ability to customize the stream of JSON messages
coming from Cargo. The new feature for Cargo is that it now also mirrors
rustc in how you can configure the JSON stream. Multiple
`--message-format` arguments are now supported and the value specified
is a comma-separated list of directives. In addition to the existing
`human`, `short`, and `json` directives these new directives have been
added:

* `json-render-diagnostics` - instructs Cargo to render rustc
  diagnostics and only print out JSON messages for artifacts and Cargo
  things.

* `json-diagnostic-short` - indicates that the `rendered` field of rustc
  diagnostics should use the "short" rendering.

* `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc
  diagnostics should embed ansi color codes.

The first option here, `json-render-diagnostics`, will be used by
rustbuild unconditionally. Additionally `json-diagnostic-short` will be
conditionally used based on the input to rustbuild itself.

This should be enough for external tools to customize how Cargo is
invoked and how all kinds of JSON diagnostics get printed, and it's
thought that we can relatively easily tweak this as necessary to extend
it and such.

alexcrichton added a commit to alexcrichton/cargo that referenced this pull request Aug 5, 2019

Add support for customizing JSON diagnostics from Cargo
Cargo has of rust-lang#7143 enabled pipelined compilation by default which
affects how the compiler is invoked, especially with respect to JSON
messages. This, in some testing, has proven to cause quite a few issues
with rustbuild's current integration with Cargo. This commit is aimed at
adding features to Cargo to solve this issue.

This commit adds the ability to customize the stream of JSON messages
coming from Cargo. The new feature for Cargo is that it now also mirrors
rustc in how you can configure the JSON stream. Multiple
`--message-format` arguments are now supported and the value specified
is a comma-separated list of directives. In addition to the existing
`human`, `short`, and `json` directives these new directives have been
added:

* `json-render-diagnostics` - instructs Cargo to render rustc
  diagnostics and only print out JSON messages for artifacts and Cargo
  things.

* `json-diagnostic-short` - indicates that the `rendered` field of rustc
  diagnostics should use the "short" rendering.

* `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc
  diagnostics should embed ansi color codes.

The first option here, `json-render-diagnostics`, will be used by
rustbuild unconditionally. Additionally `json-diagnostic-short` will be
conditionally used based on the input to rustbuild itself.

This should be enough for external tools to customize how Cargo is
invoked and how all kinds of JSON diagnostics get printed, and it's
thought that we can relatively easily tweak this as necessary to extend
it and such.

alexcrichton added a commit to alexcrichton/cargo that referenced this pull request Aug 5, 2019

Add support for customizing JSON diagnostics from Cargo
Cargo has of rust-lang#7143 enabled pipelined compilation by default which
affects how the compiler is invoked, especially with respect to JSON
messages. This, in some testing, has proven to cause quite a few issues
with rustbuild's current integration with Cargo. This commit is aimed at
adding features to Cargo to solve this issue.

This commit adds the ability to customize the stream of JSON messages
coming from Cargo. The new feature for Cargo is that it now also mirrors
rustc in how you can configure the JSON stream. Multiple
`--message-format` arguments are now supported and the value specified
is a comma-separated list of directives. In addition to the existing
`human`, `short`, and `json` directives these new directives have been
added:

* `json-render-diagnostics` - instructs Cargo to render rustc
  diagnostics and only print out JSON messages for artifacts and Cargo
  things.

* `json-diagnostic-short` - indicates that the `rendered` field of rustc
  diagnostics should use the "short" rendering.

* `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc
  diagnostics should embed ansi color codes.

The first option here, `json-render-diagnostics`, will be used by
rustbuild unconditionally. Additionally `json-diagnostic-short` will be
conditionally used based on the input to rustbuild itself.

This should be enough for external tools to customize how Cargo is
invoked and how all kinds of JSON diagnostics get printed, and it's
thought that we can relatively easily tweak this as necessary to extend
it and such.

bors added a commit that referenced this pull request Aug 7, 2019

Auto merge of #7214 - alexcrichton:json-diagnostics, r=ehuss
Add support for customizing JSON diagnostics from Cargo

Cargo has of #7143 enabled pipelined compilation by default which
affects how the compiler is invoked, especially with respect to JSON
messages. This, in some testing, has proven to cause quite a few issues
with rustbuild's current integration with Cargo. This commit is aimed at
adding features to Cargo to solve this issue.

This commit adds the ability to customize the stream of JSON messages
coming from Cargo. The new feature for Cargo is that it now also mirrors
rustc in how you can configure the JSON stream. Multiple
`--message-format` arguments are now supported and the value specified
is a comma-separated list of directives. In addition to the existing
`human`, `short`, and `json` directives these new directives have been
added:

* `json-render-diagnostics` - instructs Cargo to render rustc
  diagnostics and only print out JSON messages for artifacts and Cargo
  things.

* `json-diagnostic-short` - indicates that the `rendered` field of rustc
  diagnostics should use the "short" rendering.

* `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc
  diagnostics should embed ansi color codes.

The first option here, `json-render-diagnostics`, will be used by
rustbuild unconditionally. Additionally `json-diagnostic-short` will be
conditionally used based on the input to rustbuild itself.

This should be enough for external tools to customize how Cargo is
invoked and how all kinds of JSON diagnostics get printed, and it's
thought that we can relatively easily tweak this as necessary to extend
it and such.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.