bootstrapping rustc for a custom target no longers works since 1.48, assembles stages for build not host

[danc86]

I'm working with Yocto's meta-rust layer, trying to build rustc 1.49. I'm building on a regular x86_64 Linux host, but we use a custom JSON target specification due to rust-lang/cargo#3349. So we have x86_64-unknown-linux-gnu (real Rust triple) cross-bootstrapping for x86_64-linux (custom target which is essentially the same as x86_64-unknown-linux-gnu but with a different name):

target = ["x86_64-linux"]
host = ["x86_64-linux"]
build = "x86_64-unknown-linux-gnu"

With rust 1.47, x.py build --stage 2 src/rustc gives us the desired bootstrapping behaviour:

Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Assembling stage2 compiler (x86_64-linux)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-linux / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-linux / x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-linux / x86_64-linux)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-linux / x86_64-linux)

(Note that it also insisted on building a compiler for the build triple x86_64-unknown-linux-gnu as well as the host x86_64-linux, which is unnecessary but not harmful. It seems this has been addressed in bootstrap since 1.47.)

The end result with 1.47 above is that the build/x86_64-linux/stage2 directory contains an assembled rustc for the host x86_64-linux with libstd for x86_64-linux (and x86_64-unknown-linux-gnu).

However, with 1.48 and 1.49 the bootstrap behaviour has changed in a way that I still don't grok. When I run x.py build --stage 2 it seems both stage0 and stage1 are built for x86_64-unknown-linux-gnu. There is no stage1 built for the host x86_64-linux which is what I would expect.

Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-linux)
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-linux)
Assembling stage2 compiler (x86_64-linux)
Building rustdoc for stage2 (x86_64-linux)

As a result, this step: Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-linux) doesn't actually do what it says. The build/x86_64-linux/stage2 directory ends up containing a compiler, but its rustlib directory is empty:

$ find build/x86_64-linux/stage2
build/x86_64-linux/stage2/
build/x86_64-linux/stage2/lib
build/x86_64-linux/stage2/lib/rustlib
build/x86_64-linux/stage2/lib/rustlib/src
build/x86_64-linux/stage2/lib/rustlib/src/rust
build/x86_64-linux/stage2/lib/rustlib/x86_64-linux
build/x86_64-linux/stage2/lib/rustlib/x86_64-linux/lib
build/x86_64-linux/stage2/lib/rustlib/x86_64-linux/codegen-backends
build/x86_64-linux/stage2/lib/libstd-0689b0d4825be2a9.so
build/x86_64-linux/stage2/lib/libtest-c874a8d9a6c5eeb1.so
build/x86_64-linux/stage2/lib/libchalk_derive-882f59f2b68bc54d.so
build/x86_64-linux/stage2/lib/librustc_driver-3d1b959398c548de.so
build/x86_64-linux/stage2/lib/librustc_macros-47465b30dcf2dd0e.so
build/x86_64-linux/stage2/lib/libtracing_attributes-72e8112b29a353da.so
build/x86_64-linux/stage2/bin
build/x86_64-linux/stage2/bin/rustc
build/x86_64-linux/stage2/bin/rustdoc

That step seems to be copying into build/x86_64-unknown-linux-gnu/stage2 instead.

Complete build config and target spec for reference:
config.toml
x86_64-linux.json

[Mark-Simulacrum]

I suspect this is a similar problem to #77352, and the solution of x.py dist / x.py install I recommended there is the one I'd recommend here as well. If that doesn't work for some reason for your case I'd be interested in hearing why.

[ColinFinck]

@danc86 Just chiming in here to report that I have also been trying to upgrade meta-rust from 1.47.0 to 1.49.0 recently and I'm now stuck at the very same problem.

There were a few other obstacles on the road due to moved files and bugs like dd682cb
See https://github.com/ColinFinck/meta-rust/commits/update-to-rust-1.49.0 for the current status of my meta-rust upgrade.

[danc86]

Thanks for the pointer @Mark-Simulacrum, it does indeed seem the same as #77352, I didn't find that issue when I was searching. I will try x.py install.

In your comments on that issue, you seem to be suggesting it's just the assembly step which is no longer happening. But if I'm understanding the output I pasted above, it seems like there is a significant change in the bootstrapping behaviour. Previously, stage0 was cross-built x86_64-unknown-linux-gnu -> x86_64-linux and then stage1 host was x86_64-linux. Now, stage0 is not cross-built and stage1's host is x86_64-unknown-linux-gnu (build not host). Do you think that might be a problem? It's a little hard to tell in this particular contrived case because the x86_64-unknown-linux-gnu and x86_64-linux-gnu are identical except for their rpath which is not significant for building the compiler anyway.

[Mark-Simulacrum]

Yeah, there's potentially some oddity there. It doesn't sound like it's doing what I would expect in your case. That said, I believe the new root behavior is broadly correct (though maybe exposes preexisting internal inaccuracies/bugs). I don't think it should affect dist/install.

[kaniini]

This also affected rust 1.47 when we tried to do cross compilation for rustc on Alpine. Using x.py dist worked around the issue.

[jameshilliard]

Anyone know what's preventing #3349 from being fixed properly? The linker issue this is working around seems to be blocking pyo3 from working on buildroot right now as well.