Enforce in bootstrap that build must have stage at least 1 #142581

Kobzol · 2025-06-16T15:38:35Z

This PR is a step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917. It's very hard or me to make self-contained changes to bootstrap at this moment, so this PR kind of does several things:

(first two commits) Try to reduce the usage of Std::new in bootstrap, and replace it with a Builder::std method (similar to Builder::compiler). This is mostly to remove executions of the Std step for stage 0, which doesn't make a lot of sense; I'd like to ideally have the invariant that when a step is invoked, it actually builds or does something. Eventually, I'd like for everything to go through Builder::std. (Note: I'm not totally married to this idea, if you don't like it, we can remove it from this PR. I mostly did it right now to remove stage 0 std steps from snapshot tests, which shouldn't be there, but we can also filter them out in a different way)
Make sure that when you pass x build compiler, only the Assemble root level step will be invoked, and not the Rustc step. Before, both were invoked, which actually ran Rustc twice, once with all crates filled, and once with no crates (but both actually represent the same situation). Since the Rustc::make_run step actually requests a compile that is one stage below it, this actually made build compiler --stage 0 work, which we don't want to have anymore.
Enforce a bootstrap-global invariant that all build commands are always on stage >=1. If you try to build anything on stage 0, it will print a warning and exit bootstrap. This follows the intuition from the new staging rules after the stage redesign; artifacts that are "stage 0" come outside of bootstrap, and we can't really build something for which we don't have source (although we can still test it, but that's for another PR).

Now the logic for build should be quite simple. For pretty much everything except for Std, you first use the stage0 compiler to build stage 1. Then you can build a stage 2 using the previously built stage 1 (and then you can continue to stage 3 etc.). And that's it. The default build stage for everything is 1 (modulo download-ci-rustc, but that's a separate can of worms).

The snapshot test infra isn't super nice at the moment, as one of next steps I want to create some simple Builder pattern that will allow us to configure the bootstrap invocations in a more "forward-compatible" way (e.g. now it's not possible to modify the config passed to configure_with_args).

There are some things not yet fully resolved for build stage 0:

Cargo is still a ModeRustc tool, even though it doesn't really have to be, it is buildable with the stage0 compiler
bootstrap tools (opt-dist, build-manifest etc.) are still called stage0 tools, and in the bootstrap output it says something like "stage 0 rustc builds stage 0 opt-dist". Which is a bit weird, but functionally there's no difference, it's just a slightly inconsistent output. We still haven't decided if we should make these tools ignore staging altogether (which is IMO the right choice) or if we want to allow building stage 1/2/3/... bootstrap tools.

r? @jieyouxu

try-job: x86_64-rust-for-linux

rustbot · 2025-06-16T15:38:39Z

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

Kobzol · 2025-06-16T15:59:12Z

This actually interacts with cross-compilation, as I mentioned here:

./x build --host 'i686-unknown-linux-gnu'
thread 'main' panicked at src/bootstrap/src/core/builder/mod.rs:1344:17:
It is not possible to build the standard library for `i686-unknown-linux-gnu` using the stage0 compiler.
You have to build a stage1 compiler for `x86_64-unknown-linux-gnu` first, and then use it to build a standard library for `i686-unknown-linux-gnu`.

You need to pass --stage 2 now to make this work. Yet another trade-off between "bootstrap keeping explicit stages for commands" vs "bootstrap should just make stuff work somehow"...

rustbot · 2025-06-17T07:12:20Z

This PR modifies src/bootstrap/defaults.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

jieyouxu · 2025-06-18T10:14:27Z

src/bootstrap/src/core/config/config.rs

+        // Now check that the selected stage makes sense, and if not, print a warning and end
+        match (config.stage, &config.cmd) {
+            (0, Subcommand::Build) => {
+                eprintln!("WARNING: cannot build anything on stage 0. Use at least stage 1.");
+                exit!(1)
+            }
+            _ => {}
+        }


Question: if we warn on the no-op that is ./x build --stage=0, what about ./x check --stage=0? I guess for the check case, we can't do that yet without untying bootstrap tool checking from staging?

Also, does this reject e.g. ./x build src/tools/opt-dist --stage=0?

EDIT: indeed,

#[test] #[should_panic] fn build_bootstrap_tool_stage_0() { insta::assert_snapshot!(get_steps("opt-dist", "build", Some(0)), @""); }

Check is a whole another beast, I'm working on a PR for it, but the individual bootstrap kinds (build/check/etc.) are mostly separate in the stage handling, so I will do it piece by piece.

Btw, I'm planning for check --stage 0 to also be an error.

jieyouxu · 2025-06-18T10:22:14Z

src/bootstrap/src/core/build_steps/compile.rs

+        // If only `compiler` was passed, do not run this step.
+        // Instead the `Assemble` step will take care of compiling Rustc.
+        if run.builder.paths == vec![PathBuf::from("compiler")] {
+            return;
+        }


Remark: this kinda feels iffy, in that I feel like the path filtering/handling logic that lead to

Rustc step was invoked twice, once with all crates and once with no crates (even though both of these mean the same thing).

makes this kinda weird.

I would love Rustc and Assemble to be just one step (to avoid things like this), but I guess that it's sort of an optimizatoin, Rustc allows building only selected crates, and doesn't involve building a ton of other things that you might not want for a quick build.

Yeah, that does make sense.

jieyouxu · 2025-06-18T10:24:32Z

I'm generally in favor of imposing more discipline (or more order, for that matter) on build staging as is done in this PR. Not super sure about rejecting ./x build <bootstrap_tool> --stage=0. Would like a second opinion.

Maybe r? @Mark-Simulacrum

Kobzol · 2025-06-18T11:23:25Z

Not super sure about rejecting ./x build <bootstrap_tool> --stage=0. Would like a second opinion.

That depends on how we decide #t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign. The bootstrap tools don't really participate in staging (unless you're cross-compiling, but we don't even support cross-compiling these tools, as they are host tools, essentially). So in theory we could allow them to be built with whatever --stage you pass. However, to achieve better consistency in bootstrap staging, they should in fact be treated stage 1 tools, as they are built with a stage 0 compiler. If we force their stage to be 1, I don't think that the error on --stage 0 should anger anyone, the most natural thing (x build <host-tool>) should just work. No need to pass --stage 0.

bors · 2025-06-18T22:19:28Z

☔ The latest upstream changes (presumably #142685) made this pull request unmergeable. Please resolve the merge conflicts.

…ompiler` path It was confusing that the `Rustc` step was invoked twice, once with all crates and once with no crates (even though both of these mean the same thing).

Kobzol · 2025-06-23T08:12:37Z

The pre-requisite PRs have finally landed, so I ported this PR on top of the new config test builder and step metadata. Now it should be hopefully quite clear which (important) steps are being executed when we do x build compiler/library/<rustc-tool>/<bootstrap-tool> [--stage 0/1/2].

@jieyouxu Could you please take another looks? Thanks!

Kobzol · 2025-06-23T14:57:49Z

I had to port a bunch of tests to the new snapshot test suite. Some of them were duplicated or seemed quite dubious, so I removed a few of them.

I expect that several step traces are not ideal here (in particular around cross-compilation), and we might want to re-evaluate them. But I would only do it after we finish the renumbering for check/clippy/test kinds and also for tools, which are all kinds of special. To avoid any intermediate hacks before we see how the staging traces work after everything has been ported.

jieyouxu · 2025-06-25T02:39:20Z

Not super sure about rejecting ./x build <bootstrap_tool> --stage=0. Would like a second opinion.

That depends on how we decide #t-infra/bootstrap > Proposal to cleanup stages and steps after the redesign. The bootstrap tools don't really participate in staging (unless you're cross-compiling, but we don't even support cross-compiling these tools, as they are host tools, essentially). So in theory we could allow them to be built with whatever --stage you pass. However, to achieve better consistency in bootstrap staging, they should in fact be treated stage 1 tools, as they are built with a stage 0 compiler. If we force their stage to be 1, I don't think that the error on --stage 0 should anger anyone, the most natural thing (x build <host-tool>) should just work. No need to pass --stage 0.

I think this goes back to what Ralf described as the "staging boundary" right? As in, where do you do a stage cut-off.

However, to achieve better consistency in bootstrap staging, they should in fact be treated stage 1 tools, as they are built with a stage 0 compiler.

Sounds a bit weird to me at first, because the stage 1 library is built with the stage 1 compiler, but now bootstrap tools (stage 1 too in this PR) are built with the stage 0 compiler. Previously, they were stuffed to stage 0, because the reasoning is moreso that "it's built by the stage 0 compiler".

After reading https://hackmd.io/QXj6LqntQTKPgm3uBuE-aA, I think I can see that with

This document assumes the "stage corresponds to what gets built, not what is used to build it" model. This is a consequence of the terminology introduced above. Stage 1 rustc is simply the first rustc built by bootstrap, stage 1 std is the first std built by bootstrap etc.
With this model, it does not seem to make sense to do anything with stage 0 sources.

In the sense that bootstrap tools were built from in-tree sources. I still feel a bit... confused, mentally, by this model. I think I can understand "well, you have to pick a mental model", and I can appreciate that trying to fit various kinds of "tools" into this mental model and making this coherent/self-consistent and easy to comprehend is not exactly trivial.

@rust-lang/bootstrap thoughts?

jieyouxu

The overall implementation looks good. Just a few nits/questions, and also see previously the discussion re. mental model.

jieyouxu · 2025-06-25T02:42:03Z

src/bootstrap/src/core/build_steps/tool.rs

+            StepMetadata::build("rustdoc", self.compiler.host)
+                // rustdoc is ToolRustc, so stage N rustdoc is built by stage N-1 rustc
+                // FIXME: make this stage deduction automatic somehow
+                // FIXME: log the compiler that actually built ToolRustc steps
+                .stage(self.compiler.stage.saturating_sub(1)),


Discussion [MENTAL-MODEL 1/N]: for example, like trying to fit rustc-tools into a self-consistent/coherent mental model

Yes, I already have a follow-up branch prepared that works on tools. They are so painful.. and also duplicated across check/build. I still have to untangle this mess, and move some tools to different modes.