Unexpected non-determinism with thread::scope (but only on some targets)

[RalfJung]

The following test should always pass with -Zmiri-fixed-schedule:

use std::thread;

fn main() {
    thread::scope(|s| {
        s.spawn(|| {
            thread::yield_now();
            std::process::abort();
        });
        s.spawn(|| {
            std::process::exit(0);
        });
    });
}

We will always run the first thread, yield, go to the second thread, and exit, therefore never reaching the abort (which would fail the test). On x86_64-unknown-linux-gnu, i686-unknown-linux-gnu, and armv7-unknown-linux-gnueabihf, this indeed behaves as expected.

However, on mips-unknown-linux-gnu, the test sometimes (for around 33% of the seeds) does hit the abort and fail. I have no idea how that is possible. This behavior also only very recently started with a rustup covering this upstream diff: rust-lang/rust@1eb0657...3ff30e7. That includes rust-lang/rust#148765 which is described as merely moving code around but maybe something changed somewhere...? It also includes rust-lang/rust#144465 which is more likely to be the culprit as that mips target does not have 64bit atomics. Cc @joboet @orlp

[RalfJung]

Specifically, this commit introduces yield_now into the guts of the Rust threading code if compare_exchange_weak fails, which interferes with our usual desire to control the schedule for Miri tests.

That also explains why -Zmiri-deterministic-concurrency makes the issue disappear -- that flag also disables spurious failures in compare_exchange_weak so we never hit the extra yield_now.

So with that commit, it's impossible to control the schedule without fully disabling all concurrency-related non-determinism. That's not great for testing, we sometimes want to fix just the schedule precisely so that we can test the rest of the non-determinism. :/

[RalfJung]

@orlp From a Miri perspective the easiest fix here would be to use compare_exchange instead of compare_exchange_weak. I'm not sure what the real-world tradeoffs are regarding using one or the other; I do see some spin loops in std use the "strong" version though.

[orlp]

I think it would be perfectly fine if that compare_exchange_weak was changed to compare_exchange. It's not a performance sensitive path at all, it is only called once during thread init to increment a counter.

[joboet]

I think I found a better fix: rust-lang/rust#149476.