Update to the Rusty rust-installer #41843

cuviper · 2017-05-08T23:50:43Z

This updates the rust-installer submodule to the new version written in Rust (rust-lang/rust-installer#62), now moved to src/tools/rust-installer and invoked in dist.rs as a cargo-based tool command. All of the former shell-script invocations now invoke the tool, otherwise keeping the same arguments as before.

As a small bonus, rustc-src now also uses the same tarball generator, so it gains a smaller .tar.xz too.

Fixes #41569. r? @alexcrichton

cuviper · 2017-05-08T23:56:27Z

All of the generated dist tarballs look sane to me, and seem to install manually just fine.

However, this does fail one bootstrap test:

step: Step { name: "build-crate-std", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-test", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-rustc-main", stage: 0, host: "A", target: "A" }
step: Step { name: "build-crate-std", stage: 0, host: "A", target: "B" }
thread 'step::tests::dist_host_with_target_flag' panicked at 'assertion failed: step.target != "B"', step.rs:1582

I think that may just be a reflection of the fact that "dist" now depends on a rust-based tool, but I'd like a second opinion before I go clobbering that test into submission.

cc @brson This re-introduces the file-sorting optimization in the tarballs, so if there's a way to test rustup against this, we should. I'm now writing all directory names in the tarball before the sorted file list, so hopefully that will be enough to avoid a recurrence of rust-lang/rustup#1092.

alexcrichton · 2017-05-09T04:01:41Z

src/bootstrap/step.rs

@@ -755,6 +766,7 @@ pub fn build_rules<'a>(build: &'a Build) -> Rules {
         .host(true)
         .only_host_build(true)
         .dep(|s| s.name("tool-cargo"))
+         .dep(|s| s.name("tool-rust-installer").stage(0))


Oh here (and in other steps) I think you'll want to lock the Step here to build.config.build for both the host and the target triple, that's the architecture we want to build the tool for (as we're running it here, not packaging it)

alexcrichton · 2017-05-09T04:02:29Z

If you've got a checkout locally you may also want to run cargo vendor followed by ./x.py test src/tools/tidy, that will do a license check of crates in src/vendor to see whether it'll pass the distcheck bot (which'll do this automatically)

bors · 2017-05-09T06:02:36Z

☔ The latest upstream changes (presumably #41846) made this pull request unmergeable. Please resolve the merge conflicts.

cuviper · 2017-05-09T18:46:51Z

Rebased for the Cargo.lock conflict
Fixed step::tests::dist_host_with_target_flag by making the rust-installer dep build-only as suggested.
Verified that tidy's vendor license check passes.

alexcrichton · 2017-05-09T19:00:29Z

@bors: r+

How exciting!

bors · 2017-05-09T19:00:30Z

📌 Commit 8eaff4d has been approved by alexcrichton

frewsxcv · 2017-05-10T03:49:41Z

It looks like this fails on WIndows

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3199/job/21w6ovbl6nycrmcv

cuviper · 2017-05-10T04:30:43Z

Hmm, OK, I'll have to take a closer look at what sanitize_sh is doing. It might actually be harmful now.

cuviper · 2017-05-10T21:10:26Z

OK, I removed most of the sanitize_sh calls, where they don't actually reach shell scripts anymore. I don't have a working Windows environment to be sure this suffices though...

alexcrichton · 2017-05-10T21:50:46Z

@bors: r+

bors · 2017-05-10T21:50:47Z

📌 Commit 2729b71 has been approved by alexcrichton

@alexcrichton

…richton Update to the Rusty rust-installer This updates the rust-installer submodule to the new version written in Rust (rust-lang/rust-installer#62), now moved to `src/tools/rust-installer` and invoked in `dist.rs` as a cargo-based tool command. All of the former shell-script invocations now invoke the tool, otherwise keeping the same arguments as before. As a small bonus, `rustc-src` now also uses the same tarball generator, so it gains a smaller `.tar.xz` too. Fixes rust-lang#41569. r? @alexcrichton

frewsxcv · 2017-05-11T04:19:48Z

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3211/job/xorw4xry0k4k88rk

frewsxcv · 2017-05-11T04:20:06Z

@bors r-

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3211/job/xorw4xry0k4k88rk

cuviper · 2017-05-11T17:44:13Z

Hmm, it's a link error? I'm confused by this, as it built fine in the previous appveyor run, and I didn't change rust-installer itself since then. Has something else changed in the Windows build system?

  = note: LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
          liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol calloc imported in function lzma_alloc_zero
          liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol free imported in function lzma_end
          liblzma_sys-f02a9e47b09e132e.rlib(common.obj) : warning LNK4217: locally defined symbol malloc imported in function lzma_alloc
          liblzma_sys-f02a9e47b09e132e.rlib(lz_encoder.obj) : warning LNK4217: locally defined symbol memmove imported in function fill_window
          liblzma_sys-f02a9e47b09e132e.rlib(lzma_decoder.obj) : warning LNK4049: locally defined symbol memmove imported
          liblzma_sys-f02a9e47b09e132e.rlib(simple_coder.obj) : warning LNK4049: locally defined symbol memmove imported
          liblzma_sys-f02a9e47b09e132e.rlib(stream_encoder_mt.obj) : error LNK2019: unresolved external symbol __imp__beginthreadex referenced in function initialize_new_thread
          C:\projects\rust\build\x86_64-pc-windows-msvc\stage0-tools\x86_64-pc-windows-msvc\release\deps\rust_installer-32c88478981b650b.exe : fatal error LNK1120: 1 unresolved externals
          
error: aborting due to previous error
error: Could not compile `installer`.
To learn more, run the command again with --verbose.

alexcrichton · 2017-05-11T17:58:58Z

Oh I think that's a legitimate bug with the lzma-sys crate, it's right now apparently not compatible with -C target-feature=+crt-static, I'll look into fixing that.

alexcrichton · 2017-05-11T18:24:07Z

Looks like a fix was indeed needed, once that passes CI I'll publish to crates.io and you can update the dep here

cuviper · 2017-05-11T18:27:48Z

@alexcrichton I noticed that alexcrichton/xz2-rs#2 changed the VS version, which hasn't reached crates.io yet. Will that be a problem here?

Rollup of 15 pull requests - Successful merges: #41820, #41843, #41847, #41860, #41896, #41912, #41916, #41918, #41919, #41921, #41923, #41934, #41940, #41942, #41943 - Failed merges: #41914

Mark-Simulacrum · 2017-05-13T00:58:47Z

I suspect this failure in the rollup was caused by this PR.

@bors r-

[01:24:22] Dist extended stage2 (x86_64-pc-windows-msvc)
[01:27:05] thread 'main' panicked at 'fs::read_dir(src) failed with The system cannot find the path specified. (os error 3)', src\bootstrap\util.rs:61
[01:27:05] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:27:05] Build completed unsuccessfully in 1:15:33

cuviper · 2017-05-13T01:01:28Z

Do you have the full appveyor log? or a link to that build?
(but I probably won't be able to work on this over the weekend...)

Mark-Simulacrum · 2017-05-13T01:55:15Z

https://ci.appveyor.com/project/rust-lang/rust/build/1.0.3229

cuviper · 2017-05-13T05:37:44Z

Oh, I see. This is due to an off-hand optimization I made in rewriting combine-installer.sh. After extracting the input tarballs, instead of copying the components to the combined location, I just moved them. But here, both darwin and windows are expecting to still find those extracted components in the work-dir so they can also copy them into their OS-specific installers.

I don't love that sort of implicit expectation, but I'll go revert that part in rust-installer with a comment about this behavior. The alternative is to just extract them again explicitly -- which is also tempting.

cuviper · 2017-05-13T05:50:39Z

Hmm, or they could get components from the combined location. I think I like that better, if it works...

cuviper · 2017-05-13T21:00:38Z

OK, updated rust-installer to stop moving component dirs.

cc @Keruspe the submodule update now includes your changes too.

bors · 2017-05-14T05:59:26Z

☔ The latest upstream changes (presumably #41987) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2017-05-14T15:47:10Z

@cuviper Looks like you'll need to rebase to fix the merge conflict on Cargo.lock.

ranma42 · 2017-05-14T16:40:12Z

Also let me warn you in advance about alexcrichton/xz2-rs#4
On FreeBSD, lzma-sys 0.1.2 does not build, you need 0.1.3+

This gives us an extra rustc-src.tar.xz, which is 33% smaller than the .tar.gz!

There's no shell interpreting the file paths under the new Rusty rust-installer, so we don't need to use `sanitize_sh` for it. Plus, the drive-letter transformation is actually harmful for the now-native Windows rust-installer to understand those paths.

cuviper · 2017-05-15T04:15:18Z

OK, rebased, and I squashed the trivial dependency fixups. I have included lzma-sys 0.1.3 as suggested.

alexcrichton · 2017-05-15T14:01:20Z

@bors: r+

bors · 2017-05-15T14:01:21Z

📌 Commit 1e709fc has been approved by alexcrichton

bors · 2017-05-15T21:55:20Z

⌛ Testing commit 1e709fc with merge abe1183...

@alexcrichton

Update to the Rusty rust-installer This updates the rust-installer submodule to the new version written in Rust (rust-lang/rust-installer#62), now moved to `src/tools/rust-installer` and invoked in `dist.rs` as a cargo-based tool command. All of the former shell-script invocations now invoke the tool, otherwise keeping the same arguments as before. As a small bonus, `rustc-src` now also uses the same tarball generator, so it gains a smaller `.tar.xz` too. Fixes #41569. r? @alexcrichton

bors · 2017-05-16T01:01:27Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing abe1183 to master...

rust-highfive assigned alexcrichton May 8, 2017

alexcrichton reviewed May 9, 2017

View reviewed changes

arielb1 added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label May 9, 2017

cuviper force-pushed the oxidized-installer branch from 77875b3 to 8eaff4d Compare May 9, 2017 18:44

frewsxcv mentioned this pull request May 10, 2017

Rollup of 11 pull requests #41869

Closed

Keruspe mentioned this pull request May 10, 2017

rustbuild: add support for --bindir and --sysconfdir #41853

Merged

frewsxcv mentioned this pull request May 11, 2017

Rollup of 10 pull requests #41901

Closed

frewsxcv closed this May 11, 2017

frewsxcv reopened this May 11, 2017

Mark-Simulacrum mentioned this pull request May 12, 2017

Rollup of 15 pull requests #41954

Closed

cuviper mentioned this pull request May 13, 2017

Revert "Move the combined components instead of copying" rust-lang/rust-installer#63

Merged

cuviper added 4 commits May 14, 2017 21:06

Update to the oxidized rust-installer

020c5ec

Use "rust-installer tarball" to create rustc-src too

6edb4e2

This gives us an extra rustc-src.tar.xz, which is 33% smaller than the .tar.gz!

Force tool-rust-installer deps to build-only

3d6ee0a

cuviper force-pushed the oxidized-installer branch from 77b68ad to 1e709fc Compare May 15, 2017 04:12

bors merged commit 1e709fc into rust-lang:master May 16, 2017

This was referenced May 16, 2017

Add warnings when rustdoc html rendering differs #41991

Merged

Unify tools building #41639

Merged

[WIP] Reduce HTML output in rustdoc #41384

Closed

kennytm mentioned this pull request Aug 12, 2017

[gitignore] Add compiler-rt and rust-installer #43817

Closed

cuviper deleted the oxidized-installer branch September 26, 2017 06:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to the Rusty rust-installer #41843

Update to the Rusty rust-installer #41843

cuviper commented May 8, 2017

cuviper commented May 8, 2017

alexcrichton May 9, 2017

alexcrichton commented May 9, 2017

bors commented May 9, 2017

cuviper commented May 9, 2017

alexcrichton commented May 9, 2017

bors commented May 9, 2017

frewsxcv commented May 10, 2017

cuviper commented May 10, 2017

cuviper commented May 10, 2017

alexcrichton commented May 10, 2017

bors commented May 10, 2017

frewsxcv commented May 11, 2017

frewsxcv commented May 11, 2017

cuviper commented May 11, 2017

alexcrichton commented May 11, 2017

alexcrichton commented May 11, 2017

cuviper commented May 11, 2017

Mark-Simulacrum commented May 13, 2017

cuviper commented May 13, 2017

Mark-Simulacrum commented May 13, 2017

cuviper commented May 13, 2017

cuviper commented May 13, 2017

cuviper commented May 13, 2017

bors commented May 14, 2017

Mark-Simulacrum commented May 14, 2017

ranma42 commented May 14, 2017

cuviper commented May 15, 2017

alexcrichton commented May 15, 2017

bors commented May 15, 2017

bors commented May 15, 2017

bors commented May 16, 2017

Update to the Rusty rust-installer #41843

Update to the Rusty rust-installer #41843

Conversation

cuviper commented May 8, 2017

cuviper commented May 8, 2017

alexcrichton May 9, 2017

Choose a reason for hiding this comment

alexcrichton commented May 9, 2017

bors commented May 9, 2017

cuviper commented May 9, 2017

alexcrichton commented May 9, 2017

bors commented May 9, 2017

frewsxcv commented May 10, 2017

cuviper commented May 10, 2017

cuviper commented May 10, 2017

alexcrichton commented May 10, 2017

bors commented May 10, 2017

frewsxcv commented May 11, 2017

frewsxcv commented May 11, 2017

cuviper commented May 11, 2017

alexcrichton commented May 11, 2017

alexcrichton commented May 11, 2017

cuviper commented May 11, 2017

Mark-Simulacrum commented May 13, 2017

cuviper commented May 13, 2017

Mark-Simulacrum commented May 13, 2017

cuviper commented May 13, 2017

cuviper commented May 13, 2017

cuviper commented May 13, 2017

bors commented May 14, 2017

Mark-Simulacrum commented May 14, 2017

ranma42 commented May 14, 2017

cuviper commented May 15, 2017

alexcrichton commented May 15, 2017

bors commented May 15, 2017

bors commented May 15, 2017

bors commented May 16, 2017