Build stage 0 rust-std from rust-src with local rust (again)

[weihanglo]

Summary

(I know building stage 0 std is not officially supported)

This is kinda a reopen of #145859, because in 1.91.0 we shipped 5ce678a that errors out if codegen-backend is set to an empty list [].

We set rust.codegen-backends = [] is to avoid in in bootstrap sanity check that it determines CMake requirement by looking whether codegen-backend contains LLVM.

rust/src/bootstrap/src/core/sanity.rs

Lines 140 to 164 in f8297e3

	// We need cmake, but only if we're actually building LLVM or sanitizers.
	let building_llvm = !build.config.llvm_from_ci
	&& build.hosts.iter().any(|host| {
	build.config.llvm_enabled(*host)
	&& build
	.config
	.target_config
	.get(host)
	.map(|config| config.llvm_config.is_none())
	.unwrap_or(true)
	});
	let need_cmake = building_llvm || build.config.any_sanitizers_to_build();
	if need_cmake && cmd_finder.maybe_have("cmake").is_none() {
	eprintln!(
	"
	Couldn't find required command: cmake
	You should install cmake, or set `download-ci-llvm = true` in the
	`[llvm]` section of `bootstrap.toml` to download LLVM rather
	than building it.
	"
	);
	crate::exit!(1);
	}

rust/src/bootstrap/src/core/config/config.rs

Lines 1700 to 1702 in f8297e3

	pub fn llvm_enabled(&self, target: TargetSelection) -> bool {
	self.enabled_codegen_backends(target).contains(&CodegenBackendKind::Llvm)
	}

Since we build rust-std with the local rust toolchain, we should not need any llvm rebuild.

---

Below is pretty much the same as #145859 except commit sha.

We are actually building rust-std from rust-src with local rust built from the same rust-src.

For example,

• I have a full toolchain installed locally and built from rust-stc in f8297e3
• I'd like to build a dist tarball of rust-std with the toolchain I just built in the same f8297e3 commit.

Command used

A stripped down version of our configuration is roughly this:

./congfigure
	# we have this historically to avoid building LLVM
	--codegen-backends= 
	--enable-locked-deps
	--set rust.codegen-units-std=1
	--set rust.remap-debuginfo
	--debuginfo-level-std=1
	--dist-compression-formats=xz
	--release-channel=stable
	--enable-vendor
	--enable-local-rust
	--enable-local-rebuild
	--set "target.$TARGET_TRIPLE.linker=cc"
	--set "target.$TARGET_TRIPLE.ar=ar"
	--set "target.$TARGET_TRIPLE.ranlib=ranlib"
	--prefix "$INSTALL_DIR"
	--sysconfdir "etc"
	--build "$TARGET_TRIPLE"
	--target "$CROSS_TARGET"
	# other corss-target-specific flags…

python x.py dist --stage 0 rust-std

Expected behaviour

Before the bootstrap redesign, it worked without recompiling LLVM and stage1 compiler artifacts.
Now you need to rebuild them all, which is a bit undesired.

Proposed solution

Not sure. Maybe check whether build.local-rebuild is set to true in the bootstrap sanity check?

HEAD

1.91.0 or anything after 5ce678a

[weihanglo]

Since the commit is in #145887, let me ping @GuillaumeGomez and @Kobzol :)

(Not thing bad in that PR. It is pretty reasonable. I myself used the wrong hack 😅)

[Kobzol]

If you have an existing LLVM and you don't want to rebuild it, the default behavior should be to set llvm-config. Have you tried that?

[weihanglo]

I have the llvm-tools rustup component built along with the local-rust toolchain, though I don't think llvm-tools has the llvm-config executable.

[Kobzol]

Yeah, it doesn't. If it's really just about one sanity check that breaks, we could ignore it for local-rebuild.

Hmm. Long-term, I'd like to support empty codegen backends, because I'd like bootstrap to work without even attempting to build (or use) LLVM, which is not really a thing today. If you remove the check from #145887 and use empty codegen backends, does it work for you? How does bootstrap even locate the right LLVM to use, if you don't build/download LLVM nor set llvm-config? Or it doesn't have to, because you only care about the stdlib, right?

[weihanglo]

use empty codegen backends, does it work for you?

I used to do that and it worked (See the provided ./configure snippet --codegen-backends=)

Or it doesn't have to, because you only care about the stdlib, right?

Yes, we just build std at stage 0. I don't expect this become officially supported feature though :)

[weihanglo]

Wrong. See #148835 (comment)

Reverting 5ce678a + codegen-backend = [] didn't work. Bumped into "Standard library has to be built for dist" which was added in a62488b.

I am really sorry that I have a cursed workflow 😞.

[weihanglo]

Wrong. See #148835 (comment)

Reverting 5ce678a + codegen-backend = [] didn't work. Bumped into "Standard library has to be built for dist" which was added in a62488b.

Worked with this extra patch:

--- a/src/bootstrap/src/core/build_steps/dist.rs
+++ b/src/bootstrap/src/core/build_steps/dist.rs
@@ -818,10 +818,8 @@ fn run(self, builder: &Builder<'_>) -> Option<GeneratedTarball> {
             return None;
         }

-        // It's possible that std was uplifted and thus built with a different build compiler
-        // So we need to read the stamp that was actually generated when std was built
-        let stamp =
-            builder.std(build_compiler, target).expect("Standard library has to be built for dist");
+        let _ = builder.std(build_compiler, target);
+        let stamp = build_stamp::libstd_stamp(builder, build_compiler, target);

         let mut tarball = Tarball::new(builder, "rust-std", &target.triple);
         tarball.include_target_in_component_name(true);

[weihanglo]

I was wrong in the previous two commits (too many patches ongoing…).

This patch seems working.

diff --git a/src/bootstrap/src/core/sanity.rs b/src/bootstrap/src/core/sanity.rs
index 04cf63f1c6d..a3ff0385fcc 100644
--- a/src/bootstrap/src/core/sanity.rs
+++ b/src/bootstrap/src/core/sanity.rs
@@ -139,6 +139,9 @@ pub fn check(build: &mut Build) {

     // We need cmake, but only if we're actually building LLVM or sanitizers.
     let building_llvm = !build.config.llvm_from_ci
+        && !build.config.local_rebuild
         && build.hosts.iter().any(|host| {
             build.config.llvm_enabled(*host)
                 && build

No need to revert anything else. Let me wait for a few hours and see if the full optimization pipeline.

If it eventually works, mind me opening a pull request?

[Kobzol]

I haven't touched this part of bootstrap for a few weeks, but I'm relatively sure that let stamp = build_stamp::libstd_stamp(builder, build_compiler, target); shouldn't work, as that will create an empty build stamp, without it having stdlib artifacts that should be copied to the disted archive.

The last patch that you send with the local rebuild seems fine to me.

[weihanglo]

Opened #148883