Skip to content

/rust

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustbuild: Add support for compiling Cargo #39917

Merged
merged 2 commits into from Mar 4, 2017

Conversation

Reviewers

@brson brson

@sfackler sfackler

@ishitatsuyuki ishitatsuyuki

Assignees

@brson brson

Labels
None yet
Projects
None yet
Milestone
No milestone
@alexcrichton
Copy link
Member

alexcrichton commented Feb 17, 2017

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

  • The rust-lang/rust repository has a Cargo submodule which is used to build a
    Cargo to pair with the rust-lang/rust release
  • Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
    master branch
  • When branching beta we'll create a new branch of Cargo (as we do today), and
    the first commit to the beta branch will be to update the Cargo submodule to
    this exact revision.
  • When branching stable, we'll ensure that the Cargo submodule is updated and
    then make a stable release.

Backports to Cargo will look like:

  • Send a PR to cargo's master branch
  • Send a PR to cargo's release branch (e.g. rust-1.16.0)
  • Send a PR to rust-lang/rust's beta branch updating the submodule
  • Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

  • Add $foo as a submodule in src/tools
  • Add a tool-$foo step which compiles $foo with the specified compiler,
    likely mirroring what Cargo does.
  • Add a dist-$foo step which uses src/tools/$foo and the tool-$foo output
    to create a rust-installer package for $foo likely mirroring what Cargo
    does.
  • Update the dist-extended step with a new dependency on dist-$foo
  • Update src/tools/build-manifest for the new component.

@rust-highfive rust-highfive assigned brson Feb 17, 2017

@rust-highfive

This comment has been minimized.

Sign in to view
Copy link
Collaborator

rust-highfive commented Feb 17, 2017

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from 684305c to becb091 Feb 17, 2017

@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 20, 2017

☔️ The latest upstream changes (presumably #39966) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from becb091 to 3120181 Feb 21, 2017

@brson

brson reviewed Feb 22, 2017
View changes

src/tools/cargotest/Cargo.toml
@@ -1,5 +1,5 @@
[package]
name = "cargotest"
name = "cargotest2"

This comment has been minimized.

Sign in to view
Copy link
@brson

brson Feb 22, 2017

Contributor

Why is this renamed?

This comment has been minimized.

Sign in to view
Copy link
@alexcrichton

alexcrichton Feb 22, 2017

Author Member

Oh this conflicted with the cargotest package in Cargo itself, so I figured it'd be easiest to rename this here
@brson

This comment has been minimized.

Sign in to view
Copy link
Contributor

brson commented Feb 22, 2017

@bors r+

All the openssl logic in bootstrap is 😢
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit 3120181 has been approved by brson

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from 3120181 to ed2f0d6 Feb 22, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 22, 2017

@bors: r=brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit ed2f0d6 has been approved by brson
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 22, 2017

FWIW rust-openssl may get support soon for building OpenSSL given a source tarball, so we could at least do away with all the grungy build logic.
@sfackler

sfackler reviewed Feb 22, 2017
View changes

src/bootstrap/native.rs Outdated
};

configure.arg(format!("--prefix={}", dst.display()));
configure.arg("no-dso");

This comment has been minimized.

Sign in to view
Copy link
@sfackler

sfackler Feb 22, 2017

Member

Might be worth adding no-ssl2 no-ssl3 no-comp additionally just to be safe.

This comment has been minimized.

Sign in to view
Copy link
@alexcrichton

alexcrichton Feb 22, 2017

Author Member

Thanks!

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from ed2f0d6 to b8fc75f Feb 22, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 22, 2017

@bors: r=brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit b8fc75f has been approved by brson

@steveklabnik steveklabnik referenced this pull request Feb 22, 2017

Closed

Tracking issue for RFC 1828: Rust Bookshelf #39588

9 of 10 tasks complete

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from b8fc75f to 28ea763 Feb 23, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 23, 2017

@bors: r=brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 23, 2017

📌 Commit 28ea763 has been approved by brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 24, 2017

☔️ The latest upstream changes (presumably #39851) made this pull request unmergeable. Please resolve the merge conflicts.
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 25, 2017

🔒 Merge conflict

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from 28ea763 to be5cb13 Feb 25, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 25, 2017

@bors: r=brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 25, 2017

📌 Commit be5cb13 has been approved by brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 25, 2017

⌛️ Testing commit be5cb13 with merge 9d6ed79...

bors added a commit that referenced this pull request Feb 25, 2017

@bors
Auto merge of #39917 - alexcrichton:build-cargo, r=brson 
rustbuild: Add support for compiling Cargo

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Loading status checks…
9d6ed79
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 25, 2017

💔 Test failed - status-travis

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from be5cb13 to 985be4f Feb 25, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 25, 2017

@bors: r=brson
@alexcrichton
rustbuild: Add support for compiling Cargo 
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Loading status checks…
44a01b8

@alexcrichton alexcrichton force-pushed the alexcrichton:build-cargo branch from 77fb172 to 44a01b8 Mar 3, 2017

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Mar 3, 2017

@bors: r=brson

Why thank you @bors, I did indeed not intend to check in 100MB build trees into the repo
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

📌 Commit 44a01b8 has been approved by brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

⌛️ Testing commit 44a01b8 with merge 7d2bdb0...

bors added a commit that referenced this pull request Mar 3, 2017

@bors
Auto merge of #39917 - alexcrichton:build-cargo, r=brson 
rustbuild: Add support for compiling Cargo

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Loading status checks…
7d2bdb0
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

💔 Test failed - status-travis
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Mar 3, 2017

@bors: retry
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

⌛️ Testing commit 44a01b8 with merge 4e0cfbd...

bors added a commit that referenced this pull request Mar 3, 2017

@bors
Auto merge of #39917 - alexcrichton:build-cargo, r=brson 
rustbuild: Add support for compiling Cargo

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Loading status checks…
4e0cfbd
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

💔 Test failed - status-travis
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Mar 3, 2017

@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 3, 2017

⌛️ Testing commit 44a01b8 with merge 5208090...

bors added a commit that referenced this pull request Mar 3, 2017

@bors
Auto merge of #39917 - alexcrichton:build-cargo, r=brson 
rustbuild: Add support for compiling Cargo

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Loading status checks…
5208090
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Mar 4, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: brson
Pushing 5208090 to master...

@bors bors merged commit 44a01b8 into rust-lang:master Mar 4, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

This was referenced Mar 4, 2017

Merged
Merged
Merged
Closed
Merged
@shepmaster

This comment has been minimized.

Sign in to view
Copy link
Member

shepmaster commented Mar 4, 2017

@alexcrichton after pulling this change, I can no longer build:

$ rm -rf build-release/
$ mkdir build-release
$ cd build-release/
$ ../configure
$ make
downloading https://static.rust-lang.org/dist/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
downloading https://static.rust-lang.org/dist/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
downloading https://s3.amazonaws.com/rust-lang-ci/cargo-builds/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
error: failed to read `/Users/shep/Projects/rust/src/tools/cargo/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
make: *** [all] Error 101

Is there another step I'm unaware of?
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Mar 4, 2017

Ah yup that'd do it.

Cargo's a submodule. We'll have to move submodule updates sooner in the process or warn about it. You can fix this with:

git submodule update --init

@alexcrichton alexcrichton deleted the alexcrichton:build-cargo branch Mar 4, 2017

@Mark-Simulacrum

This comment has been minimized.

Sign in to view
Copy link
Member

Mark-Simulacrum commented Mar 4, 2017

Should that perhaps be posted in a more visible location (e.g., internals)? I suspect pretty much all developers working on rustc will run into it.
@shepmaster

This comment has been minimized.

Sign in to view
Copy link
Member

shepmaster commented Mar 4, 2017

all developers working on rustc

I think everyone is playing that new Zelda game today 😉

More seriously, I just assumed that it would be fixed soon enough that not many people would run into it. I'm also a bit surprised that the build machines didn't have any issues. I guess there's differences between what a human does and the autobuilds?

This was referenced Mar 5, 2017

Closed
Closed
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Mar 6, 2017

I've sent a PR to fix the issue @shepmaster ran into.

Sorry for the breakage!

Also yeah the auto builds explicitly update submodules for... reasons. So that's why they didn't run into this.

@alexcrichton alexcrichton referenced this pull request Mar 24, 2017

Closed

Overall cycle time has regressed ~25% from 2hrs to 2.5hrs #40804

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.