Skip to content

Assorted bootstrap LLVM refactors (part 1/N) - #160645

Open
Kobzol wants to merge 10 commits into
rust-lang:mainfrom
Kobzol:bootstrap-llvm
Open

Assorted bootstrap LLVM refactors (part 1/N)#160645
Kobzol wants to merge 10 commits into
rust-lang:mainfrom
Kobzol:bootstrap-llvm

Conversation

@Kobzol

@Kobzol Kobzol commented Aug 6, 2026

Copy link
Copy Markdown
Member

This PR continues my LLVM and download-ci-llvm bootstrap refactors (it took me almost a year to get back to them, lol), with the goals of:

  • Remove dependency on implicit paths with artifacts. Ideally, everything should only depend on output paths from executed steps.
  • Reduce eager and implicit downloads and other shenanigans happening in config parsing, particularly around download-ci-llvm.
  • Allow downloading LLVM from CI for non-host targets. There is not really any reason why bootstrap shouldn't be able to download LLVM from CI if it's there for any given target, but right now it can only do so for the host target. This would be useful for debugging bootstrap itself, and perhaps for some people also for development.
  • Centralize the handling of LLVM into the Llvm step, and reduce overall knowledge between locally built and downloaded LLVM. Ideally, most of bootstrap shouldn't know about whether it uses a LLVM that was built locally or downloaded. But that is of course not the case today, because there are many places in bootstrap that ad-hoc work with some LLVM paths, and do things that depend on some part of bootstrap randomly modifying a config here or there, or doing some side effect, which then makes the other activity "work". But of course, this is very fragile, as we know.
  • If I manage to reach the previous goal, this should massively help unblock further unborking of download-ci-rustc. Those two features are quite interrelated, and sadly both are scattered across the codebase. I actually first wanted to start with refactoring download-ci-rustc, but I couldn't find a way to do that without first improving download-ci-llvm.

This PR contains a bunch of commits that slowly move us towards these goals. The refactors were quite explanatory, they are mostly a preparation for larger refactors that I had in mind, so nothing major. But as always, step by step.

As usually, best reviewed commit-by-commit.

Some notes for review:

  • The last commit might case a behavior change (well, or an error) on MSVC without Ninja, when filecheck is used without download-ci-llvm. I need to figure out how does the directory structure look like in that case, as I wasn't able to build LLVM with MSVC locally.
  • 07b88f2 removes a sanity check for FileCheck being present. It was a bit hacky, because it was only checking it if it wasn't locally built (or downloaded from CI...). To avoid introducing more hacks, I just removed this sanity check. If the code will need filecheck and it won't be available, it will error out at the usage site (or I could add an assert that the FileCheck binary is present in the FileCheck step if you want).

r? jieyouxu

Kobzol added 10 commits August 6, 2026 16:46
At this point in the code, the LLVM config could not have been set by `download-ci-llvm` yet, so we don't have to check it.
To make it consistent with `GccOutput`
So that it can be used explicitly in bootstrap, rather than depending on `builder.llvm_out`.
To remove dependency on implicit paths.
And replace it with an explicit `FileCheck` step
To avoid someone depending on implicit paths. Instead, the step should be executed and the path should be taken from its output.
…-llvm` is enabled

Note: this commit removed reading `FileCheck` from `<artifact-dir>/build/<profile>/bin`, and instead reads it from `<artifact-dir>/bin` directly when using MSVC without Ninja. I'm not 100% sure if this will work with that setup.
@rustbot

rustbot commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 6, 2026
@rustbot

rustbot commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

jieyouxu is currently at their maximum review capacity.
They may take a while to respond.

@rust-log-analyzer

Copy link
Copy Markdown
Collaborator

The job aarch64-gnu-llvm-21-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

error: `bootstrap` (lib) generated 1 warning (run `cargo fix --lib -p bootstrap` to apply 1 suggestion)
error: warnings are denied by `build.warnings` configuration
failed to run: /checkout/obj/build/aarch64-unknown-linux-gnu/stage0/bin/cargo build --jobs=default --manifest-path /checkout/src/bootstrap/Cargo.toml -Zroot-dir=/checkout --locked --features build-metrics
Build completed unsuccessfully in 0:00:26
make: *** [Makefile:102: prepare] Error 1
Command failed. Attempt 2/5:
##[group]Building bootstrap
warning: unused import: `crate::core::build_steps::llvm`
##[warning]  --> src/bootstrap/src/lib.rs:59:5
   |
59 | use crate::core::build_steps::llvm;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

error: `bootstrap` (lib) generated 1 warning (run `cargo fix --lib -p bootstrap` to apply 1 suggestion)
error: warnings are denied by `build.warnings` configuration
failed to run: /checkout/obj/build/aarch64-unknown-linux-gnu/stage0/bin/cargo build --jobs=default --manifest-path /checkout/src/bootstrap/Cargo.toml -Zroot-dir=/checkout --locked --features build-metrics
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:102: prepare] Error 1
Command failed. Attempt 3/5:
##[group]Building bootstrap
warning: unused import: `crate::core::build_steps::llvm`
##[warning]  --> src/bootstrap/src/lib.rs:59:5
   |
59 | use crate::core::build_steps::llvm;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

error: `bootstrap` (lib) generated 1 warning (run `cargo fix --lib -p bootstrap` to apply 1 suggestion)
error: warnings are denied by `build.warnings` configuration
failed to run: /checkout/obj/build/aarch64-unknown-linux-gnu/stage0/bin/cargo build --jobs=default --manifest-path /checkout/src/bootstrap/Cargo.toml -Zroot-dir=/checkout --locked --features build-metrics
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:102: prepare] Error 1
Command failed. Attempt 4/5:
##[group]Building bootstrap
warning: unused import: `crate::core::build_steps::llvm`
##[warning]  --> src/bootstrap/src/lib.rs:59:5
   |
59 | use crate::core::build_steps::llvm;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

error: `bootstrap` (lib) generated 1 warning (run `cargo fix --lib -p bootstrap` to apply 1 suggestion)
error: warnings are denied by `build.warnings` configuration
failed to run: /checkout/obj/build/aarch64-unknown-linux-gnu/stage0/bin/cargo build --jobs=default --manifest-path /checkout/src/bootstrap/Cargo.toml -Zroot-dir=/checkout --locked --features build-metrics
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:102: prepare] Error 1
Command failed. Attempt 5/5:
##[group]Building bootstrap
warning: unused import: `crate::core::build_steps::llvm`
##[warning]  --> src/bootstrap/src/lib.rs:59:5
   |
59 | use crate::core::build_steps::llvm;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default

error: `bootstrap` (lib) generated 1 warning (run `cargo fix --lib -p bootstrap` to apply 1 suggestion)
error: warnings are denied by `build.warnings` configuration
failed to run: /checkout/obj/build/aarch64-unknown-linux-gnu/stage0/bin/cargo build --jobs=default --manifest-path /checkout/src/bootstrap/Cargo.toml -Zroot-dir=/checkout --locked --features build-metrics
Build completed unsuccessfully in 0:00:00
make: *** [Makefile:102: prepare] Error 1
The command has failed after 5 attempts.
  local time: Thu Aug  6 15:06:31 UTC 2026
  network time: Thu, 06 Aug 2026 15:06:31 GMT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants