Rework Rustbuild to an eagerly compiling approach #43059

Mark-Simulacrum · Jul 4, 2017

This introduces a new dependency on serde; I don't believe that's a problem since bootstrap is compiled with nightly/beta always so proc macros are available. Compile times are slightly longer -- about 2-3x (30 seconds vs. 10 seconds). I don't think this is too big a problem, especially since recompiling bootstrap is somewhat rare. I think we can remove the dependency on Serde if necessary, though, so let me know.

r? @alexcrichton

petrochenkov · Jul 4, 2017

I see large amounts of code being moved around in the first commit. Is this

Steps in the new system utilize types instead of strings to describe
themselves. This is more Rusty, and (in theory) catches mistakes
earlier.

the only effect (and motivation) for the first commit? Are the changes purely mechanical?
I see some drive-by changes there (renamings etc), but it's almost impossible to notice and review them.

Mark-Simulacrum · Jul 4, 2017

I hope that's the only effect; unfortunately, I don't see of any easy way to split the work up into multiple commits at this point. Or, at least, multiple commits that work.

I can spend some time to add a commit that just moves all code without changing it at all to use Step, I think, though it itself won't build.

Edit: To be clear, I will spend some time soon splitting up the code into a move-everything commit first.

Mark-Simulacrum · Jul 5, 2017

Okay, I've pushed up a new set of commits. This implementation strives to be the minimal possible to make things work, i.e., things aren't cleaned up pretty much at all. 18ed842 is the primary commit.

The new commit chain currently doesn't attempt to keep the code buildable up until the last commit -- this is so that the diff is better.

My current plan is to wait for us to review this, merge it, and then in future PRs clean the code up. I think that's the most efficient way to do this.

There are a few FIXMEs in the code that I'm not entirely sure how to fix, and they'll need to be handled before we merge -- I'll be investigating options later on.

bors · Jul 6, 2017

☔️ The latest upstream changes (presumably #40939) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · Jul 6, 2017

Rebased.

alexcrichton · Jul 7, 2017

alexcrichton reviewed Jul 7, 2017

View changes

Looks like a great direction to go in to me! I haven't exhaustively reviewed yet but opted to look at some of the new abstractions followed by a skim of check.rs. The overall framework looks pretty good to me, albeit not quite as ergonomic as I'd hoped :(. I think the handling of "default rules" is a little unfortunate but certainly not a dealbreaker.

I wonder if it'd be worth getting some more 'flavorful' suites running before doing a more in-depth review? WDYT?

alexcrichton · Jul 7, 2017

src/bootstrap/Cargo.toml

@@ -38,3 +38,6 @@ getopts = "0.2"
 rustc-serialize = "0.3"


If we take a dependency on serde I think we should be able to remove rustc-serialize and update toml, right?

alexcrichton · Jul 7, 2017

src/bootstrap/lib.rs

@@ -65,9 +65,15 @@
 //! also check out the `src/bootstrap/README.md` file for more information.

 #![deny(warnings)]
+#![feature(associated_consts)]
+#![feature(core_intrinsics)]


I'm not personally a huge fan of using unstable code in the bootstrap process, is there a way we could avoid this? Are these features required?

So associated constants should be stabilized soon I believe -- there is a PR I think already?

The core intrinsics here is just used for sanity checking in the cache -- once we get non 'static TypeId it should also be able to go away if we can serialize that, which should be possible.

I think it's fine to keep these for now, since the first is soon to be stabilized and the latter needs to be implemented.

I'm sort of ok with the associated consts, but I'd personally really prefer to remove core_intrinsics. I don't think a non-'static TypeId will exist any time soon due to soundness issues.

non-'static type id is tracked here: #41875.

Note that it's just a change to the intrinsic, not TypeId itself. The TypeId itself will still require 'static

Well, I'm probably in favor of not landing that change. It seems that even though the intrinsic is unsafe, from what I gather (not really well-versed in this subject) it's not possible to use the relaxed bound in any safe way anyway so it doesn't seem all that useful, I guess.

The intrinsic only gives you an integer. It can be safe but it's also useless, unless combined with unsafe code to build an Any-like abstraction.

Ah I see, yes I believe this was @Mark-Simulacrum's original suggested approach. On reflection, this is probably doable with no more boilerplate than wrapping the struct definition in a macro.

(also, for some reason I previously thought you couldn't have per-function static variables which didn't help my thinking)

You can put any module-level item (including whole modules!) inside any block of statements, but they are not parametrized by the surrounding function's generics or anything like that, it's just for name scoping. This is sometimes used inside the initializer of a static to declare more statics it refers to, or entire types & their trait impls, especially from macro-generated code (e.g. thread_local!).

Ah, that was it! I wanted statics per instance of a generic function which doesn't work (as you note), but generating each function individually by a macro is fine. That explains my confusion, thanks.

alexcrichton · Jul 7, 2017

src/bootstrap/builder.rs

+
+    fn run(self, builder: &'a Builder) -> Self::Output;
+
+    fn should_run(_builder: &'a Builder, _path: &Path) -> bool { false }


Could you add some doctests for this trait and the methods here? I'm just starting but I don't quite know what should_run and make_run to yet

alexcrichton · Jul 7, 2017

src/bootstrap/builder.rs

+        }
+    }
+
+    pub fn ensure<S: Step<'a> + Serialize>(&'a self, step: S) -> S::Output


Should Serialize be a supertrait of Step?

alexcrichton · Jul 7, 2017

src/bootstrap/check.rs

+    const ONLY_HOSTS: bool = true;
+
+    fn should_run(_builder: &Builder, path: &Path) -> bool {
+        path.ends_with("cargo") // FIXME: Why is this not src/tools/cargo?


Oh just historical, this should change to src/tools/cargo now

alexcrichton · Jul 7, 2017

src/bootstrap/check.rs

+
+static COMPILETESTS: &[(bool, &str, &str, &str)] = &[
+    // default, path, mode, suite
+    (true, "src/test/codegen", "codegen", "codegen"),


Could these switch to structs with field names?

alexcrichton · Jul 7, 2017

src/bootstrap/check.rs

+        builder.ensure(native::TestHelpers { target });
+
+        if mode == "debuginfo-gdb" {
+            builder.ensure(RemoteCopyLibs { compiler, target });


Shouldn't this happen for all test suites?

(e.g. run-pass needs libs in an emulator as well)

Mark-Simulacrum · Jul 7, 2017

I plan to test this out and fix related bugs on various platforms this weekend, though I don't have access to windows to test that platform. I've pushed up some more documentation and fixed the comments I believe.

The overall framework looks pretty good to me, albeit not quite as ergonomic as I'd hoped :(. I think the handling of "default rules" is a little unfortunate but certainly not a dealbreaker.

Could you clear up the ergonomics question here a little? Maybe we can change some things to make it better?

Mark-Simulacrum · Jul 7, 2017

Mark-Simulacrum reviewed Jul 7, 2017

View changes

Mark-Simulacrum · Jul 7, 2017

src/bootstrap/lib.rs

+//! provide those libraries for it; they are mostly equivalent to constructing
+//! the stage1/bin compiler so we don't go through them individually.
+//!
+//! ## Uplifiting stage1 {std,test,rustc}


typo here: Uplifiting

bors · Jul 9, 2017

☔️ The latest upstream changes (presumably #43117) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · Jul 9, 2017

Hm, looks like this will probably have to wait a bit -- I hope to get back on it next week though.

alexcrichton · Jul 10, 2017

Oh I don't think there's any blockers or anything on the ergonomic front, it's not like it's great today defining things in two places. I didn't have any actionable concerns in mind.

aidanhs · Jul 13, 2017

I've made a "everything becomes owned" PR at Mark-Simulacrum#1. It's fairly large unfortunately and adds a lot of to_owned/clone. However, the cache becomes simpler and there are no more intrinsics.

Mark-Simulacrum · Jul 15, 2017

@alexcrichton I'm changing this to waiting-on-review, since I think it's ready for that. There was at least one successful Travis build (8fd7703). As before, please be brutal with review.

bors · Jul 15, 2017

☔️ The latest upstream changes (presumably #43246) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · Jul 16, 2017

Rebased.

aidanhs · Jul 16, 2017

aidanhs reviewed Jul 16, 2017

View changes

aidanhs · Jul 16, 2017

src/bootstrap/cache.rs

+
+use builder::Step;
+
+pub struct Interned<T>(usize, PhantomData<*const T>);


Why not a T rather than a *const T?

We don't own the T here, and I think this is how to indicate that -- the T is owned by the TyInterner. Correct me if I'm wrong, though.

It's a phantomdata so I don't know that matters? Unless it's a convention thing.

aidanhs · Jul 16, 2017

Much more in favour of this method of caching (for other readers: things are 'interned' by just leaking them, eliminating all lifetime complexity).

I'll try and take a look through the rest of it later this week. Just so I'm clear, things that have been lost include:

available paths list
advance plan of what's going to be executed
?

bors · Jul 22, 2017

💔 Test failed - status-travis

Mark-Simulacrum · Jul 22, 2017

@bors r=alexcrichton

bors · Jul 22, 2017

📌 Commit 8f2e576 has been approved by alexcrichton

bors · Jul 22, 2017

⌛️ Testing commit 8f2e576 with merge 3e8ee24...

alexcrichton · Jul 22, 2017

@infinity0 ah yeah to clarify no funcitonal change is intended here (other than making things "better"), it's just a big change and probably a good idea to keep our eyes our for regressions!

bors · Jul 22, 2017

💔 Test failed - status-travis

Mark-Simulacrum · Jul 22, 2017

@bors r=alexcrichton

bors · Jul 22, 2017

📌 Commit 1c11823 has been approved by alexcrichton

bors · Jul 22, 2017

⌛️ Testing commit 1c11823 with merge 8d22af8...

infinity0 · Jul 22, 2017

OK, thanks for the explanations!

bors · Jul 22, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 8d22af8 to master...

infinity0 · Sep 18, 2017

This PR seems to have removed the --no-fail-fast behaviour for tests, is that correct?

infinity0 · Sep 18, 2017

At least, step.rs used to report on the number of failures then exit 1, now that file is removed and I can't grep for the same functionality elsewhere.

cuviper · Sep 18, 2017

cuviper reviewed Sep 18, 2017

View changes

cuviper · Sep 18, 2017

src/bootstrap/step.rs

-        let failures = self.build.delayed_failures.get();
-        if failures > 0 {
-            println!("\n{} command(s) did not execute successfully.\n", failures);
-            process::exit(1);


This is the code that @infinity0 is talking about. The delayed_failures counter still exists, and is incremented, but AFAICS nothing checks it anymore.

Looks like this just needs to go at the end of Build::build. I'll try that and send a PR.

Yeah, it's quite likely that we're just not checking it anymore. r? me on the PR please!

@infinity0 has a change to also summarize the failed commands, so I think they'll PR it.

rust-highfive assigned alexcrichton Jul 4, 2017

aidanhs added the S-waiting-on-review label Jul 5, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from ba0ade8 to 6a9ef25 Jul 5, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 6a9ef25 to 7bac3ef Jul 6, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 7bac3ef to 5148363 Jul 6, 2017

Mark-Simulacrum referenced this pull request Jul 10, 2017
Closed
x.py test with non-existent test directory should error #43121

arielb1 added S-waiting-on-author and removed S-waiting-on-review labels Jul 11, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from a674f24 to 8ec46cb Jul 11, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 4684859 to 981235c Jul 14, 2017

Mark-Simulacrum added S-waiting-on-review and removed S-waiting-on-author labels Jul 15, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 448be7d to 8972d5f Jul 16, 2017

Add make_run to distcheck.

Loading status checks…

8f2e576

Make distcheck work again.

Loading status checks…

1c11823

bors merged commit 1c11823 into rust-lang:master Jul 22, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

This was referenced Jul 22, 2017

Merged

Support dynamically-linked and/or native musl targets #40113

Merged

Bump master to 1.21.0 #43320

Merged

Support compiling rustc without LLVM (try 2) #42932

Closed

rustbuild: Add a way to build all tools as part of 'x.py build' #42979

Mark-Simulacrum added a commit to Mark-Simulacrum/rust that referenced this pull request Jul 23, 2017

Revert "Auto merge of rust-lang#43059 - Mark-Simulacrum:rustbuild-2.0…
…, r=alexcrichton" This reverts commit 8d22af8, reversing changes made to f8d485f.

a230f15

xen0n referenced this pull request Jul 24, 2017
Closed
rustc armv7 gnueabihf tarballs since 07-23 package files for x86-64 #43450

QuietMisdreavus referenced this pull request Feb 16, 2018
Merged
Remove hoedown from rustdoc #48274

rust-lang/rust

Join GitHub today

Rework Rustbuild to an eagerly compiling approach #43059

Conversation

Mark-Simulacrum commented Jul 4, 2017 • edited

rust-highfive assigned alexcrichton Jul 4, 2017

This comment has been minimized.

petrochenkov commented Jul 4, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 4, 2017 • edited

aidanhs added the S-waiting-on-review label Jul 5, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from ba0ade8 to 6a9ef25 Jul 5, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 5, 2017

This comment has been minimized.

bors commented Jul 6, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 6a9ef25 to 7bac3ef Jul 6, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 6, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 7bac3ef to 5148363 Jul 6, 2017

alexcrichton reviewed Jul 7, 2017 View changes

alexcrichton left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Jul 7, 2017

Mark-Simulacrum reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

bors commented Jul 9, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 9, 2017

Mark-Simulacrum referenced this pull request Jul 10, 2017

This comment has been minimized.

alexcrichton commented Jul 10, 2017

arielb1 added S-waiting-on-author and removed S-waiting-on-review labels Jul 11, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from a674f24 to 8ec46cb Jul 11, 2017

This comment has been minimized.

aidanhs commented Jul 13, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 4684859 to 981235c Jul 14, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 15, 2017

Mark-Simulacrum added S-waiting-on-review and removed S-waiting-on-author labels Jul 15, 2017

This comment has been minimized.

bors commented Jul 15, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from 448be7d to 8972d5f Jul 16, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 16, 2017

aidanhs reviewed Jul 16, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aidanhs commented Jul 16, 2017

This comment has been minimized.

bors commented Jul 22, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 22, 2017

This comment has been minimized.

bors commented Jul 22, 2017

This comment has been minimized.

bors commented Jul 22, 2017

bors added a commit that referenced this pull request Jul 22, 2017

This comment has been minimized.

alexcrichton commented Jul 22, 2017

This comment has been minimized.

Mark-Simulacrum commented Jul 4, 2017 •

edited

Mark-Simulacrum commented Jul 4, 2017 •

edited

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `ba0ade8` to `6a9ef25` Jul 5, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `6a9ef25` to `7bac3ef` Jul 6, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `7bac3ef` to `5148363` Jul 6, 2017

alexcrichton reviewed Jul 7, 2017

View changes

Mark-Simulacrum reviewed Jul 7, 2017

View changes

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `a674f24` to `8ec46cb` Jul 11, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `4684859` to `981235c` Jul 14, 2017

Mark-Simulacrum force-pushed the Mark-Simulacrum:rustbuild-2.0 branch from `448be7d` to `8972d5f` Jul 16, 2017

aidanhs reviewed Jul 16, 2017

View changes

bors merged commit `1c11823` into rust-lang:master Jul 22, 2017
2 checks passed

cuviper reviewed Sep 18, 2017

View changes