Prevent miri from being distributed if tests are failing

[oli-obk]

fixes #60301

r? @RalfJung

[RalfJung]

Any reason you do this after the tempdir is already created?

[RalfJung]

Also, uh, this is in the Clippy impl...? Shouldn't the same be added for Miri then?

[RalfJung]

I was also about to ask what this testing method tests, but CI already says it does not even exist. ;)

== TestPass is anyway clearer, IMO.

[oli-obk]

yea... I just did this via copy paste in the github interface. Working with a local build now to get this actually working

[RalfJung]

I don't know bootstrap at all, I feel someone who does should review this.

Cc @rust-lang/infra

[rust-highfive]

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:02:44]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:02:45] error[E0425]: cannot find value `stage` in this scope
[00:02:45]     --> src/bootstrap/dist.rs:1277:57
[00:02:45]      |
[00:02:45] 1277 |             println!("skipping Dist Miri stage{} ({})", stage, target);
[00:02:45] 
[00:02:47] error[E0609]: no field `toolstate` on type `config::Config`
[00:02:47]     --> src/bootstrap/dist.rs:1276:28
[00:02:47]      |
[00:02:47]      |
[00:02:47] 1276 |         if !builder.config.toolstate.miri.testing() {
[00:02:47]      |
[00:02:47]      |
[00:02:47]      = note: available fields are: `ccache`, `ninja`, `verbose`, `submodules`, `fast_submodules` ... and 98 others
[00:02:48] error: aborting due to 2 previous errors
[00:02:48] 
[00:02:48] Some errors have detailed explanations: E0425, E0609.
[00:02:48] For more information about an error, try `rustc --explain E0425`.
---
[00:02:49]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:02:50] error[E0425]: cannot find value `stage` in this scope
[00:02:50]     --> src/bootstrap/dist.rs:1277:57
[00:02:50]      |
[00:02:50] 1277 |             println!("skipping Dist Miri stage{} ({})", stage, target);
[00:02:50] 
[00:02:52] error[E0609]: no field `toolstate` on type `config::Config`
[00:02:52]     --> src/bootstrap/dist.rs:1276:28
[00:02:52]      |
[00:02:52]      |
[00:02:52] 1276 |         if !builder.config.toolstate.miri.testing() {
[00:02:52]      |
[00:02:52]      |
[00:02:52]      = note: available fields are: `ccache`, `ninja`, `verbose`, `submodules`, `fast_submodules` ... and 98 others
[00:02:53] error: aborting due to 2 previous errors
[00:02:53] 
[00:02:53] Some errors have detailed explanations: E0425, E0609.
[00:02:53] For more information about an error, try `rustc --explain E0425`.
---
[00:02:55]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:02:56] error[E0425]: cannot find value `stage` in this scope
[00:02:56]     --> src/bootstrap/dist.rs:1277:57
[00:02:56]      |
[00:02:56] 1277 |             println!("skipping Dist Miri stage{} ({})", stage, target);
[00:02:56] 
[00:02:58] error[E0609]: no field `toolstate` on type `config::Config`
[00:02:58]     --> src/bootstrap/dist.rs:1276:28
[00:02:58]      |
[00:02:58]      |
[00:02:58] 1276 |         if !builder.config.toolstate.miri.testing() {
[00:02:58]      |
[00:02:58]      |
[00:02:58]      = note: available fields are: `ccache`, `ninja`, `verbose`, `submodules`, `fast_submodules` ... and 98 others
[00:02:59] error: aborting due to 2 previous errors
[00:02:59] 
[00:02:59] Some errors have detailed explanations: E0425, E0609.
[00:02:59] For more information about an error, try `rustc --explain E0425`.
---
[00:03:02]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:03:03] error[E0425]: cannot find value `stage` in this scope
[00:03:03]     --> src/bootstrap/dist.rs:1277:57
[00:03:03]      |
[00:03:03] 1277 |             println!("skipping Dist Miri stage{} ({})", stage, target);
[00:03:03] 
[00:03:05] error[E0609]: no field `toolstate` on type `config::Config`
[00:03:05]     --> src/bootstrap/dist.rs:1276:28
[00:03:05]      |
[00:03:05]      |
[00:03:05] 1276 |         if !builder.config.toolstate.miri.testing() {
[00:03:05]      |
[00:03:05]      |
[00:03:05]      = note: available fields are: `ccache`, `ninja`, `verbose`, `submodules`, `fast_submodules` ... and 98 others
[00:03:06] error: aborting due to 2 previous errors
[00:03:06] 
[00:03:06] Some errors have detailed explanations: E0425, E0609.
[00:03:06] For more information about an error, try `rustc --explain E0425`.
---
[00:03:10]    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
[00:03:11] error[E0425]: cannot find value `stage` in this scope
[00:03:11]     --> src/bootstrap/dist.rs:1277:57
[00:03:11]      |
[00:03:11] 1277 |             println!("skipping Dist Miri stage{} ({})", stage, target);
[00:03:11] 
[00:03:13] error[E0609]: no field `toolstate` on type `config::Config`
[00:03:13]     --> src/bootstrap/dist.rs:1276:28
[00:03:13]      |
[00:03:13]      |
[00:03:13] 1276 |         if !builder.config.toolstate.miri.testing() {
[00:03:13]      |
[00:03:13]      |
[00:03:13]      = note: available fields are: `ccache`, `ninja`, `verbose`, `submodules`, `fast_submodules` ... and 98 others
travis_time:end:14b42bc0:start=1559987031636884105,finish=1559987226868178960,duration=195231294855
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 1.
travis_time:start:0af5abd6
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:2c16f557:start=1559987227627291099,finish=1559987227642407050,duration=15115951
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00faab68
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:11021554
travis_time:start:11021554
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:3d014b16
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[kennytm]

Tests aren't run in dist builds, and we can't run the tests for cross-compiled toolchains. I doubt this could be fixed at all.

[RalfJung]

Not being able to ensure that the tools we distribute pass tests seems like a severe problem to me.

[oli-obk]

aw, that's too bad.

Tests aren't run in dist builds, and we can't run the tests for cross-compiled toolchains.

So we are building everything twice? Once in the dist builders and once in the tool builders where we also run tests?

Could we deduplicate that for non-cross dist builds?

[kennytm]

@oli-obk

We are building Miri 1 + N times: 1 test and N dist targets. This is fine since all jobs run in parallel.

Deduplicating means we need to restructure the CI pipeline from "run everything in parallel" to some sort of DAG and share the built artifacts, which we won't consider in the short term (at least not before the Azure experiment becomes stable).

[RalfJung]

We run tests for all tier 1 targets though, right? At least I have seen tool builders for Linux, macOS, Windows. So more like 3 + N? At least for those in principle we could have the test result affect the distribution.

However, from what I gather those are currently not done on the same builder. And that's probably for the better actually, the "tools" builder sets --cfg miri to be more Miri-friendly and we don't want that in the release artifacts.

[oli-obk]

Right, so we'll stay with the status quo. What we could do is make miri parse its corresponding toolstate and warn the user that it may be broken

[RalfJung]

Uh, just because we currently can't fix this bug doesn't mean that the current behavior is reasonable, does it? I'd rather keep this open TBH.

[RalfJung]

Oh sorry, this was the PR, not the bug. Ignore me.^^

[kennytm]

@oli-obk Sorry yes you're right, should be 2 + N 😅 (Linux and Windows x86_64; we don't have enough capacity to test for Mac yet)