./x.py test deletes rustc_private artifacts from the sysroot

[RalfJung]

After doing ./x.py --keep-stage 0 build src/rustc, I can use build/$ARCH/stage2 as Rust toolchain through rustup, and it works just fine. However, if I now do ./x.py --keep-stage 0 test ..., the toolchain breaks -- for example, compiling miri fails with error[E0463]: can't find crate for `rustc`. I have to do ./x.py --keep-stage 0 build src/rustc again (which finishes pretty quickly) to fix this -- but that should not be necessary.

[eddyb]

cc @alexcrichton @Mark-Simulacrum

[jyn514]

@RalfJung are you still running into this issue?

[eddyb]

--keep-stage is/was fundamentally unsound, and I recall it also having implementation bugs on top of that.

Looking back, I'm not really surprised this could happen, I've seen worse.

[RalfJung]

are you still running into this issue?

As of a few weeks ago, definitely.

I am not sure if this is truly related to keep-stage -- the issue is that ./x.py --keep-stage 0 test does more than it is supposed to do, and --keep-stage should only reduce the things that are done. But of course there might be strange interactions.

[RalfJung]

Indeed the issue reproduces without keep-stage:

./x.py build compiler/rustc --stage 2
./x.py test src/test/ui --stage 2 --test-args xyz

Now the stage 2 rustc toolchain is broken:

error[E0463]: can't find crate for `rustc_apfloat`
  --> src/lib.rs:10:1
   |
10 | extern crate rustc_apfloat;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
   |
   = help: maybe you need to install the missing components with: `rustup component add rust-src rustc-dev llvm-tools-preview`

[eddyb]

I've seen running tests result in ./x.py check restart from core, so maybe there is a common issue here.

[jyn514]

That sounds like #76661.

[jyn514]

@RalfJung are you still running into this issue now that bootstrap locks the build directory?

[RalfJung]

Yes, this has nothing to do with concurrency.

[jyn514]

Ok, I was able to reproduce this - the issue is that the sysroot is losing .rlibs somehow. Here is the crate that's loaded after build compiler/rustc:

  name: rustc_apfloat
  cnum: 20
  hash: 1a9ab672a0b086cb
  reqd: Explicit
   rlib: /home/jnelson/rust-lang/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_apfloat-c81c8ebd40dcd26d.rlib

and here is the sysroot after running tests:

$ ls /home/jnelson/rust-lang/rust2/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/
libaddr2line-3733a061cc96d94e.rlib          libpanic_unwind-e619fc71d5e5938c.rlib
libadler-058f7deceffe8ae6.rlib              libproc_macro-c025c630933f960d.rlib
liballoc-694b740f5092604c.rlib              librustc_demangle-6170a356858b7694.rlib
libcfg_if-eaba38c7a7d8afc1.rlib             librustc_std_workspace_alloc-3fa5eddea5055dd2.rlib
libcompiler_builtins-8c3ba16d6db014ad.rlib  librustc_std_workspace_core-e626afde28ee4760.rlib
libcore-95c79210edd9be3d.rlib               librustc_std_workspace_std-83203b3bd446dd68.rlib
libgetopts-4fc96e7a1264828e.rlib            libstd-8ff1f66553aed7b6.rlib
libgimli-29d33f415b86107e.rlib              libstd-8ff1f66553aed7b6.so
libhashbrown-59789cebd18f640a.rlib          libstd_detect-64e62bce3f419596.rlib
liblibc-8373763bd57009a7.rlib               libtest-1bdded4d8b665692.rlib
libLLVM-14-rust-1.64.0-nightly.so           libtest-1bdded4d8b665692.so
libmemchr-55016efe9ecf287f.rlib             libunicode_width-c86dc2a289d37e66.rlib
libminiz_oxide-7f86ace49e82a758.rlib        libunwind-5f745570100ed92f.rlib
libobject-28545d362f2f4a0e.rlib             self-contained
libpanic_abort-abdf8849139cd581.rlib

I think what's going on is that x.py deletes everything in build/stage2 when creating the sysroot for the tests, which would make this is a duplicate of #34279. It might be easier to fix since the files being deleted are controlled by bootstrap? I don't know an easy way to tell whether the generated artifacts are up-to-date (I don't think we should do this if the artifacts are out of date for the reasons described in #34279 (comment)).

[RalfJung]

This is unrelated to x.py test it turns out. E.g. this makes it impossible to have both a rustc and rustdoc binary in build/x86_64-unknown-linux-gnu/stage2/bin/: after ./x.py build compiler/rustc --stage 2, the rustc binary exists. Then I run ./x.py build src/tools/rustdoc --stage 2 which creates the rurstdoc binary but deletes rustc. If I now build rustc again, that will delete rustdoc.

[Mark-Simulacrum]

You need x.py build src/tools/rustdoc compiler/rustc --stage 2; we clean and re-create sysroots on each run today.

There's a discussion somewhere about this (can't find it now) but the short is that I feel this is better than maybe-stale artifacts and don't trust us to not miss deleting a stale artifact.

[jyn514]

@Mark-Simulacrum https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/Performing.20.60.2E.2Fx.2Epy.20dist.60.20results.20in.20a.20broken.20compiler.20at.20.2E.2E.2E

[RalfJung]

@Mark-Simulacrum unfortunately that fails due to #90244

Building rustbuild
    Finished dev [unoptimized] target(s) in 0.05s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.15s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.23s
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.15s
Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized + debuginfo] target(s) in 0.23s
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage3 compiler (x86_64-unknown-linux-gnu)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp")', lib.rs:1434:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:01

[Mark-Simulacrum]

That thread has "Yeah, the error is just annoying; it doesn't actually break anything." -- is that wrong? i.e., that this fails but actually still copies things appropriately.

That's primarily why it's not been prioritized in my eyes.

[RalfJung]

I think in this case it means x.py stops after the rustc build step, which means it never gets to the rustdoc build step.

So, it doesn't break anything for single-build-step x.py invocations, but might be a real problem for multi-build-step x.py invocations. I always thought x.py build X Y is the same as x.py build X && x.py build Y, but your previous comment says that is not the case, and then the stamp issue becomes an actual blocker.

[jyn514]

I'm going to close this as a duplicate of #34279, which has some more discussion.