Generate XZ-compressed tarballs #41600

ranma42 · 2017-04-28T07:02:04Z

Integrate the new rust-installer and extend manifests with keys for xz-compressed tarballs.

One of the steps required for #21724

instead of combining the component and target into a URL.

The same unavailable target value is used in two different places. Abstracting it makes it easier to update it and recognise its purpose.

rust-highfive · 2017-04-28T07:02:15Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

ranma42 · 2017-04-28T07:03:28Z

I tried to add the required packages to the CI infrastructure, but I am not very confident that everything is in place. I trust the CI itself to warn about such issues.
Should I ALLOW_PR=1 to do that?

ishitatsuyuki · 2017-04-28T11:20:30Z

You can use ALLOW_PR to get rapid feedback; doing it on one of the builders (preferably x86_64-linux-gnu) doesn't jam the queue so much. Make sure you add some sort of WIP tag.

nikomatsakis · 2017-04-28T18:34:01Z

Who should review this? Not, I think, me.

nikomatsakis · 2017-04-28T18:34:24Z

r? @alexcrichton -- feel free to pick someone else, I figured you'd know the right person tho

alexcrichton

Looks excellent to me, thakns @ranma42!

Right now if this gets past bors then we unfortunately don't have many guarantees. It should exercise all xz-producing code but doesn't exercise the manifest generator, that'll happen when the next nightly is built.

You can also run everything locally (at least container-wise) with ./src/ci/docker/run.sh arm-android (etc)

alexcrichton · 2017-04-28T19:40:09Z

src/tools/build-manifest/src/main.rs

            };
+            let xz_filename = filename.replace(".tar.gz", ".tar.xz");


This doesn't currently validate that the xz file actually exists, could that be done as an extra sanity check?

self.digests should only contain digests for existing files. Do you expect the file to be deleted between the digest computation and the construction of the manifest?

Oh good point yeah, but that means that the url could be specified and the digest could be none, right? Could we statically prevent a situation such as that?

xz_url: xz_digest.as_ref().map(|_| self.url(&xz_filename)) makes sure that the url is only specified if the digest is Something (and that its contents are based on the filename, not on the digest value).

gah sorry, yes of course!

alexcrichton · 2017-04-28T19:41:08Z

src/ci/docker/x86_64-gnu-nopt/Dockerfile

@@ -11,6 +11,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
  cmake \
  sudo \
  gdb \
+  p7zip-full \


A bunch of containers don't actually produce tarballs, only the dist-* ones do I believe. Could this package be left out of all the containers (like this one) not producing tarballs?

Uhm... Actually I might have removed one too many: does the cross image also produce tarballs? Even if it is not dist-*, it is flagged DEPLOY=1.

Oops yes sorry DEPLOY=1 is the marker to look for.

alexcrichton · 2017-04-28T19:42:45Z

cc @brson

mattico · 2017-04-28T20:31:43Z

Out of curiosity: why are you using p7zip rather than xz-utils (which is already being installed)?

ranma42 · 2017-04-28T20:45:28Z

@mattico 7z is available on AppVeyor (where it is somewhat harder to install software). See rust-lang/rust-installer#57 (comment)

alexcrichton · 2017-04-29T17:17:16Z

Ok everything looks great to me @ranma42! Want to squash and remove '[WIP]' and r+?

ranma42 · 2017-04-29T19:44:57Z

Squashed and renamed :)

alexcrichton · 2017-04-30T04:29:00Z

@bors: r+

bors · 2017-04-30T04:29:01Z

📌 Commit 36b9ab7 has been approved by alexcrichton

bors · 2017-04-30T07:46:55Z

⌛ Testing commit 36b9ab7 with merge 5616298...

bors · 2017-04-30T07:58:07Z

💔 Test failed - status-travis

ranma42 · 2017-04-30T08:24:15Z

Uh, of course distcheck also requires 7z. I amended the last commit adding it (and mentioning in the commit message the affected docker images).

alexcrichton · 2017-04-30T09:12:20Z

@bors: r+

bors · 2017-04-30T09:12:20Z

📌 Commit bf4082a has been approved by alexcrichton

bors · 2017-05-03T05:52:34Z

⌛ Testing commit 6bf71f0 with merge d5948bd...

bors · 2017-05-03T06:04:09Z

💔 Test failed - status-travis

TimNN · 2017-05-03T06:05:45Z

Looks legitimate:

[00:03:47] make-tarballs: error: need 7z

(https://travis-ci.org/rust-lang/rust/jobs/228215768)

@bors r-

MacOSX does not ship `7z` nor `xz`. Let's use `xz`, just like on the other *nix systems.

ranma42 · 2017-05-03T06:11:27Z

I must have messed up when squashing the update of the rust-installer submodule, sorry. It should be fixed by my latest push.

alexcrichton · 2017-05-03T14:02:57Z

@bors: r+

bors · 2017-05-03T14:02:58Z

📌 Commit 5e522d7 has been approved by alexcrichton

Generate XZ-compressed tarballs Integrate the new `rust-installer` and extend manifests with keys for xz-compressed tarballs. One of the steps required for rust-lang#21724

bors · 2017-05-03T19:18:33Z

⌛ Testing commit 5e522d7 with merge 234472d...

aidanhs · 2017-05-03T19:39:09Z

Think this caused a rollup failure in #41725 on the osx xcode 7 builders

The command "brew update && brew install xz" failed and exited with 1 during

Can't decipher why from the logs, but both of the osx xcode 7 builders failed, so I suspect it'll fail here as well.

alexcrichton · 2017-05-03T19:55:50Z

@aidanhs oddly enough all builders have gotten past the xz installation stage here ...

alexcrichton · 2017-05-03T19:58:16Z

@bors: r-

er I believe @frewsxcv is correct

@ranma42 can you ensure that there are retry loops around the installations on OSX?

alexcrichton · 2017-05-03T19:58:31Z

@bors: retry

Wrap the installation on macOS with `travis_retry`.

ranma42 · 2017-05-03T20:42:50Z

I added travis_retry to the update and install commands, but it is not clear to me why the log was mangled in that way (as if two commands were running concurrently).

alexcrichton · 2017-05-03T22:32:21Z

@bors: r+

Yeah I also found that log... interestingly ordered

bors · 2017-05-03T22:32:22Z

📌 Commit 98dd82c has been approved by alexcrichton

Generate XZ-compressed tarballs Integrate the new `rust-installer` and extend manifests with keys for xz-compressed tarballs. One of the steps required for rust-lang#21724

Rollup of 6 pull requests - Successful merges: #41543, #41600, #41715, #41720, #41721, #41730 - Failed merges:

bors · 2017-05-04T01:40:12Z

⌛ Testing commit 98dd82c with merge b16c7a2...

As of rust-lang/rust#41600 we now need `xz` to build Rust from HEAD.

ranma42 added 2 commits April 25, 2017 16:08

Accept arbitrary filenames into URLs in build-manifest

ac8892b

instead of combining the component and target into a URL.

Define the unavailable target just once and re-use it

5439c9b

The same unavailable target value is used in two different places. Abstracting it makes it easier to update it and recognise its purpose.

rust-highfive assigned nikomatsakis Apr 28, 2017

ranma42 changed the title ~~Generate XZ-compressed tarballs~~ [WIP] Generate XZ-compressed tarballs Apr 28, 2017

shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Apr 28, 2017

rust-highfive assigned alexcrichton and unassigned nikomatsakis Apr 28, 2017

alexcrichton reviewed Apr 28, 2017

View reviewed changes

alexcrichton added the T-tools label Apr 28, 2017

ranma42 force-pushed the xz-dist branch from 0fcb111 to 36b9ab7 Compare April 29, 2017 19:44

ranma42 changed the title ~~[WIP] Generate XZ-compressed tarballs~~ Generate XZ-compressed tarballs Apr 29, 2017

ranma42 force-pushed the xz-dist branch from 36b9ab7 to bf4082a Compare April 30, 2017 08:23

ranma42 added 2 commits May 3, 2017 08:09

Emit information about XZ packages in manifest

5dfa08e

Fix MacOSX build

5e522d7

MacOSX does not ship `7z` nor `xz`. Let's use `xz`, just like on the other *nix systems.

ranma42 force-pushed the xz-dist branch from 6bf71f0 to 5e522d7 Compare May 3, 2017 06:11

frewsxcv mentioned this pull request May 3, 2017

Rollup of 4 pull requests #41725

Closed

Retry brew commands upon failure

98dd82c

Wrap the installation on macOS with `travis_retry`.

frewsxcv mentioned this pull request May 3, 2017

Rollup of 6 pull requests #41735

Merged

bors added a commit that referenced this pull request May 3, 2017

Auto merge of #41735 - frewsxcv:rollup, r=frewsxcv

b16c7a2

Rollup of 6 pull requests - Successful merges: #41543, #41600, #41715, #41720, #41721, #41730 - Failed merges:

bors merged commit 98dd82c into rust-lang:master May 4, 2017

alexcrichton mentioned this pull request May 5, 2017

rustup nightly install failing rust-lang/rustup#1092

Closed

brandt added a commit to brandt/homebrew-core that referenced this pull request May 6, 2017

rust: fix missing build requirement xz

ceac172

As of rust-lang/rust#41600 we now need `xz` to build Rust from HEAD.

Generate XZ-compressed tarballs #41600

Generate XZ-compressed tarballs #41600

Conversation

ranma42 commented Apr 28, 2017

rust-highfive commented Apr 28, 2017

ranma42 commented Apr 28, 2017 • edited Loading

ishitatsuyuki commented Apr 28, 2017

nikomatsakis commented Apr 28, 2017

nikomatsakis commented Apr 28, 2017

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ranma42 Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Apr 28, 2017

mattico commented Apr 28, 2017

ranma42 commented Apr 28, 2017

alexcrichton commented Apr 29, 2017

ranma42 commented Apr 29, 2017

alexcrichton commented Apr 30, 2017

bors commented Apr 30, 2017

bors commented Apr 30, 2017

bors commented Apr 30, 2017

ranma42 commented Apr 30, 2017

alexcrichton commented Apr 30, 2017

bors commented Apr 30, 2017

bors commented May 3, 2017

bors commented May 3, 2017

TimNN commented May 3, 2017

ranma42 commented May 3, 2017 • edited Loading

alexcrichton commented May 3, 2017

bors commented May 3, 2017

bors commented May 3, 2017

aidanhs commented May 3, 2017

alexcrichton commented May 3, 2017

alexcrichton commented May 3, 2017

alexcrichton commented May 3, 2017

ranma42 commented May 3, 2017

alexcrichton commented May 3, 2017

bors commented May 3, 2017

bors commented May 4, 2017

ranma42 commented Apr 28, 2017 •

edited

Loading

ranma42 Apr 28, 2017 •

edited

Loading

ranma42 commented May 3, 2017 •

edited

Loading