Enable building complete release artifacts on Travis / Appveyor #38531

brson · 2016-12-22T03:02:19Z

Updated description

Active work items:

Turn on Travis/AppVeyor nightlies - update secrets.toml
Turn off Buildbot nightlies - comment out trigger in rust-buildbot
Separate --enable-extended from x.py test builds on all platforms (only Windows remaining)
Pull in Cargo as a submodule instead of from a pre-built location
Run cargo's test suite on select bots.
Turn off Cargo uploads

Original description

Converting our travis/appveyor CI to do full release builds. By doing release builds on every platform on every commit we should stop breaking nightlies so much.

switch homu to travis/appveyor, leaving buildbot in place (Fix fs tests on Windows systems with non-english locales. #38536 (comment))
modify rustbuild to build stage1 by default, with option for stage2 (rustbuild: Compile rustc twice, not thrice #38631)
add stage2 bootstrap bot to verify the full bootstrap works (rustbuild: Compile rustc twice, not thrice #38631)
add sccache to bots (rustbuild: Add sccache support #38359)
add remaining cross-host rustc/platform builders (travis: Expand dist builder coverage #39042)
add distcheck bot (Gate on distcheck on Travis #38708)
have rustbuild produce .sha256 files on all dist output (travis: Start uploading artifacts on commits #38748)
modify all builds to 'make dist' and upload (travis: Start uploading artifacts on commits #38748)
confirm all matrix entries are correct

More steps

Figure out rls submodule maintenance model
- in-tree vs. out-of-tree and how
Move rls into rust-lang-nursery (tools team)
Add rls submodule/source to src/
Add a --enable-extended / --enable-platform flag (rustbuild: Start building --enable-extended #39245)
- teach bootstrap to only build, test and dist the rls when this flag is set
- think about the name of this flag because it's important
Add --enable-extended to the Trivas/AppVeyor release builders (rustbuild: Start building --enable-extended #39245)
- this will start uploading rls packages to rust-lang-ci
Modify rust-buildbot to pass --enable-extended to host dist builds (rustbuild: Start building --enable-extended #39245)
- Starts building and uploading the RLS

At this point we've got rls sufficiently packaged that we can flip the
switch. Next steps are to flip that switch.

Make and deploy patch to rust-buildbot to add rls to manifests as optional
Make a new nightly to test RLS deployment
Add rls to the combined installer via rust-packaging
- tarballs, .pkgs, .msis
- so that non-rustup users can access

We'd want to do all the above before the March 16 branch for 1.17 beta.

At this point we're delivering the RLS via the buildbot release
builders. Next steps are to follow through on the release builder
conversion.

Add --enable-dist-msi and --enable-dist-pkg flags to configure (rustbuild: Start building --enable-extended #39245)
- move those packaging steps from rust-packaging into the tree as
  part of make dist
- remember that the rls is not included when --enable-extended is
  not enabled... hm
Again in-tree, create a new build target, x.py dist src/publish-ci-build (or something) to do the final steps of
arranging the bins, signing, building the manifest, and uploading to
static.rust-lang.org (rustbuild: Add manifest generation in-tree #39284)
- This is in-tree because it gives us a path for others being able
  to produce their own rustup dist servers, and also because it's just
  better to have all the code for packaging in the same git commit.
- Being in-tree has the complication that we have to run ./configure
  even though we don't need to build any Rust...
- All the steps need to (eventually) be discrete so that people
  who want to build manifests without our CI infrastructure can do so,
  but for our purposes we can have one x.py dist publish-ci-build
- 'publish-download' step:
  - for any given channel, download the appropriate bins from rust-lang-ci
    s3 bucket, put them in the right file system layout
  - this might want to verify hashes
- 'publish-hash' step:
- 'publish-signing' step:
- 'publish-manifest-build' step:
- 'publish-upload' step:
Build a docker container that runs a the above as a service
- publish to some staging area for testing before we turn it on in prod
- batch job every night
- cron job in a docker container?
Set up aws machine to run that service
- let it run in parallel to existing build service
- test with RUSTUP_DIST_SERVER
port cargo build to the rust extended build system
- simultaneously turn off cargo's existing publication
modify docker images to use correct old glibcs (travis: Move glibc backwards in time #39198)
- keep using crosstool ng
disable llvm assertions for all DEPLOY travisveyor configs (travis: Move glibc backwards in time #39198)
switch over to the new release builds, turn off buildbot

Other misc

Uploads can overwrite one another, probably messing up checksums (travis: Stop uploading sha256 files #39167)
Need to change release channel of artifacts to nightly (travis: Pass --release-channel=nightly on deploy #39050)
Need to compile against libstdc++ statically (travis: Pass --enable-llvm-static-stdcpp #39132)
Need to update all gcc toolchains to very old glibc (travis: Move glibc backwards in time #39198)
Shouldn't pass --enable-llvm-assertions all the time, especially for stable/beta releases (travis: Move glibc backwards in time #39198)
need a story for stable/beta releases

The text was updated successfully, but these errors were encountered:

brson · 2016-12-22T03:09:15Z

re https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489

Mark-Simulacrum · 2016-12-22T03:19:09Z

modify rustbuild to build stage1 by default, with option for stage2

What's the expected approach to dealing with situations where cfg(stage0) or cfg(not(stage0)) is currently used? Currently, since stage0 is bootstrap (old compiler) and stage1 is a new compiler, there is a transition path, but with this I'm not sure what the expected arguments to x.py are.

brson · 2016-12-22T03:22:31Z

@alexcrichton will need to address those details @Mark-Simulacrum.

alexcrichton · 2016-12-27T07:32:14Z

@Mark-Simulacrum you may be interested in #38631, but the gist is that nothing changes, it's all the same. It may not be clear what terminology means what, but #38631 as a concrete implementation should hopefully clarify what's intended.

Mark-Simulacrum · 2016-12-27T15:12:47Z

So instead of stage0 we should now cfg(stage1)?

Perhaps a better way to ask my question is: when changes are made to the compiler that make a given lang item no longer necessary, what cfg should it be annotated with?

alexcrichton · 2016-12-27T16:50:20Z

Oh no everything should proceed as usual. When the compiler changes a lang item you use #[cfg(stage0)] and #[cfg(not(stage0))], as we do today.

Is there something specific, though, that you're worried about breaking?

Mark-Simulacrum · 2016-12-27T21:15:38Z

In #38574, I removed the trans_exchange_free lang item, and as such a compile with --increnental (which I understand is equivalent to the changes suggested above) failed since the beta compiler expected it to be defined. Perhaps I missed something, though.

This commit optimizes the compile time for creating tarballs of cross-host compilers and as a proof of concept adds two to the standard Travis matrix. Much of this commit is further refactoring and refining of the `step.rs` definitions along with the interpretation of `--target` and `--host` flags. This has gotten confusing enough that I've also added a small test suite to `src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress. After this commit when you execute: ./x.py dist --host $MY_HOST --target $MY_HOST the build system will compile two compilers. The first is for the build platform and the second is for the host platform. This second compiler is then packaged up and placed into `build/dist` and is ready to go. With a fully cached LLVM and docker image I was able to create a cross-host compiler in around 20 minutes locally. Eventually we plan to add a whole litany of cross-host entries to the Travis matrix, but for now we're just adding a few before we eat up all the extra capacity. cc rust-lang#38531

rustbuild: Quickly `dist` cross-host compilers This commit optimizes the compile time for creating tarballs of cross-host compilers and as a proof of concept adds two to the standard Travis matrix. Much of this commit is further refactoring and refining of the `step.rs` definitions along with the interpretation of `--target` and `--host` flags. This has gotten confusing enough that I've also added a small test suite to `src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress. After this commit when you execute: ./x.py dist --host $MY_HOST --target $MY_HOST the build system will compile two compilers. The first is for the build platform and the second is for the host platform. This second compiler is then packaged up and placed into `build/dist` and is ready to go. With a fully cached LLVM and docker image I was able to create a cross-host compiler in around 20 minutes locally. Eventually we plan to add a whole litany of cross-host entries to the Travis matrix, but for now we're just adding a few before we eat up all the extra capacity. cc #38531

This commit optimizes the compile time for creating tarballs of cross-host compilers and as a proof of concept adds two to the standard Travis matrix. Much of this commit is further refactoring and refining of the `step.rs` definitions along with the interpretation of `--target` and `--host` flags. This has gotten confusing enough that I've also added a small test suite to `src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress. After this commit when you execute: ./x.py dist --host $MY_HOST --target $MY_HOST the build system will compile two compilers. The first is for the build platform and the second is for the host platform. This second compiler is then packaged up and placed into `build/dist` and is ready to go. With a fully cached LLVM and docker image I was able to create a cross-host compiler in around 20 minutes locally. Eventually we plan to add a whole litany of cross-host entries to the Travis matrix, but for now we're just adding a few before we eat up all the extra capacity. cc rust-lang#38531

rustbuild: Quickly `dist` cross-host compilers This commit optimizes the compile time for creating tarballs of cross-host compilers and as a proof of concept adds two to the standard Travis matrix. Much of this commit is further refactoring and refining of the `step.rs` definitions along with the interpretation of `--target` and `--host` flags. This has gotten confusing enough that I've also added a small test suite to `src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress. After this commit when you execute: ./x.py dist --host $MY_HOST --target $MY_HOST the build system will compile two compilers. The first is for the build platform and the second is for the host platform. This second compiler is then packaged up and placed into `build/dist` and is ready to go. With a fully cached LLVM and docker image I was able to create a cross-host compiler in around 20 minutes locally. Eventually we plan to add a whole litany of cross-host entries to the Travis matrix, but for now we're just adding a few before we eat up all the extra capacity. cc #38531

This commit expands the existing x86_64-musl entry in the Travis matrix to also build/test i586-unknown-linux-gnu and i686-unknown-linux-musl. cc rust-lang#38531 Closes rust-lang#39053

brson · 2017-01-19T00:03:32Z

Updated the op with more steps.

travis: Add i586 linux and i686 musl This commit expands the existing x86_64-musl entry in the Travis matrix to also build/test i586-unknown-linux-gnu and i686-unknown-linux-musl. cc rust-lang#38531 Closes rust-lang#35599 Closes rust-lang#39053

…brson travis: Expand the `cross` linux image This expands the `cross` travis matrix entry with a few more targets that our nightlies are building: * x86_64-rumprun-netbsd * arm-unknown-linux-musleabi * arm-unknown-linux-musleabihf * armv7-unknown-linux-musleabihf * mips-unknown-linux-musl * mipsel-unknown-linux-musl This commit doesn't compile custom toolchains like our current cross-image does, but instead compiles musl manually and then compiles libunwind manually (like x86_64) for use for the ARM targets and just uses openwrt toolchains for the mips targets. cc rust-lang#38531

appveyor: Test/Dist i586 MSVC This is a target that we're shipping today, so this commit adds this matrix entry to AppVeyor. This reuses the existing i686 MSVC matrix entry as it's currently finishing about a half hour under two hours, which should hopefully give it enough extra time to run this test suite. cc rust-lang#38531

travis: Stop uploading sha256 files We'll generate these later in the build process and otherwise they could just cause spurious failures with files overwriting one another. cc rust-lang#38531

rustbuild: Start building --enable-extended This commit adds a new flag to the configure script, `--enable-extended`, which is intended for specifying a desire to compile the full suite of Rust tools such as Cargo, the RLS, etc. This is also an indication that the build system should create combined installers such as the pkg/exe/msi artifacts. Currently the `--enable-extended` flag just indicates that combined installers should be built, and Cargo is itself not compiled just yet but rather only downloaded from its location. The intention here is to quickly get to feature parity with the current release process and then we can start improving it afterwards. All new files in this PR inside `src/etc/installer` are copied from the rust-packaging repository. cc #38531

rustbuild: Add manifest generation in-tree This commit adds a new tool, `build-manifest`, which is used to generate a distribution manifest of all produced artifacts. This tool is intended to replace the `build-rust-manifest.py` script that's currently located on the buildmaster. The intention is that we'll have a builder which periodically: * Downloads all artifacts for a commit * Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files as well as TOML manifests. * Upload all generated hashes and manifests to the directory the artifacts came from. * Upload *all* artifacts (tarballs and hashes and manifests) to an archived location. * If necessary, upload all artifacts to the main location. This script is intended to just be the second step here where orchestrating uploads and such will all happen externally from the build system itself. cc #38531

rustbuild: Add manifest generation in-tree This commit adds a new tool, `build-manifest`, which is used to generate a distribution manifest of all produced artifacts. This tool is intended to replace the `build-rust-manifest.py` script that's currently located on the buildmaster. The intention is that we'll have a builder which periodically: * Downloads all artifacts for a commit * Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files as well as TOML manifests. * Upload all generated hashes and manifests to the directory the artifacts came from. * Upload *all* artifacts (tarballs and hashes and manifests) to an archived location. * If necessary, upload all artifacts to the main location. This script is intended to just be the second step here where orchestrating uploads and such will all happen externally from the build system itself. cc rust-lang#38531

alexcrichton · 2017-01-29T22:41:07Z

I've started an external docker image for the bot which orchestrates/signs releases.

That's currently running at buildbot2.rust-lang.org and is running homu/cancelbot. It's also got an initial draft of a script to actually do the releases here, but it's still waiting on successful builds and such to actually work.

alexcrichton · 2017-04-18T12:40:35Z

We've now basically completed all of this, so closing.

brson added the T-tools label Dec 22, 2016

This was referenced Dec 26, 2016

rustbuild: copy libs for stage1 from stage0 when using incremental and enable-local-rust #38575

Closed

rustbuild: Compile rustc twice, not thrice #38631

Merged

alexcrichton mentioned this issue Dec 28, 2016

rustbuild: Compile all support tools in stage0 #38667

Merged

alexcrichton mentioned this issue Dec 31, 2016

rustbuild: Quickly dist cross-host compilers #38731

Merged

alexcrichton mentioned this issue Jan 14, 2017

travis: Add i586 linux and i686 musl #39068

Merged

This was referenced Jan 17, 2017

travis: Expand the cross linux image #39111

Merged

appveyor: Test/Dist i586 MSVC #39112

Merged

alexcrichton mentioned this issue Jan 19, 2017

travis: Stop uploading sha256 files #39167

Merged

zmanian mentioned this issue Jan 19, 2017

Security model / TUF rust-lang/crates.io#75

Open

alexcrichton mentioned this issue Jan 19, 2017

Add custom toolchains for powerpc* and s390x rust-lang-deprecated/rust-buildbot#149

Merged

alexcrichton mentioned this issue Jan 22, 2017

rustbuild: Start building --enable-extended #39245

Merged

alexcrichton mentioned this issue Jan 25, 2017

rustbuild: Add manifest generation in-tree #39284

Merged

stereobooster mentioned this issue Mar 2, 2017

Add support for cross-compiled extensions malept/thermite#32

Open

alexcrichton mentioned this issue Mar 12, 2017

Panic in cargo::version rust-lang/cargo#3819

Closed

alexcrichton closed this as completed Apr 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable building complete release artifacts on Travis / Appveyor #38531

Enable building complete release artifacts on Travis / Appveyor #38531

brson commented Dec 22, 2016 •

edited by alexcrichton

Loading

brson commented Dec 22, 2016

Mark-Simulacrum commented Dec 22, 2016

brson commented Dec 22, 2016

alexcrichton commented Dec 27, 2016

Mark-Simulacrum commented Dec 27, 2016

alexcrichton commented Dec 27, 2016

Mark-Simulacrum commented Dec 27, 2016

brson commented Jan 19, 2017

alexcrichton commented Jan 29, 2017

alexcrichton commented Apr 18, 2017

Enable building complete release artifacts on Travis / Appveyor #38531

Enable building complete release artifacts on Travis / Appveyor #38531

Comments

brson commented Dec 22, 2016 • edited by alexcrichton Loading

Updated description

Original description

More steps

Other misc

brson commented Dec 22, 2016

Mark-Simulacrum commented Dec 22, 2016

brson commented Dec 22, 2016

alexcrichton commented Dec 27, 2016

Mark-Simulacrum commented Dec 27, 2016

alexcrichton commented Dec 27, 2016

Mark-Simulacrum commented Dec 27, 2016

brson commented Jan 19, 2017

alexcrichton commented Jan 29, 2017

alexcrichton commented Apr 18, 2017

brson commented Dec 22, 2016 •

edited by alexcrichton

Loading