Add x.py option to --force-rerun compiletest tests

[syvb]

This can be used like ./x.py test src/test/ui/abi/ --force-rerun, and is useful when verifying that newly blessed tests don't change between test runs (such as due to being dependent on the current time or memory layout or RNG), without needing to change the test file or find the right file in build to remove.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[Mark-Simulacrum]

useful when verifying that newly blessed tests don't change between test runs (such as due to being dependent on the current time or memory layout or RNG)

I am ... a little surprised that this is something people are doing; can you share more details perhaps? I would've expected to see that under some kind of reproducible builds style environment, where the test invocation is extracted and run separately under various users, times, environment variables, etc.

It seems like there's been a lot of positive feedback from folks though based on the GitHub emojis, so I'd like to hear more but not strictly opposed to landing this. I do want to make sure that the use case is being correctly addressed, however. It may for example make more sense for x.py clean src/test/ui to delete the test artifacts and allow tests to rerun, without needing to propagate options into compile test to force a lack of caching.

[syvb]

@Mark-Simulacrum Often I see things in test .stderr files that looks like it is random (e.g. because it kinda looks like part of a memory address), so I manually re-run that test to check if it really is. It's not a substitute for testing on multiple platforms and environments, but it's easier to test by repeating it on the same machine first. Currently what I do when I want this is to find the right file in build to delete (I often forget where it is and have to look it up), then re-run the test I want.

[Mark-Simulacrum]

Hm. Interesting! I don't think I've ever seen anything that looks like a memory address or similar in a stderr file, so I'd be interested in further details. I'm going to go ahead and r=me on this, but mark it as S-waiting-on-team because I want to hear if this is a common problem -- it might be that we should (for example) automatically run tests twice in --bless mode if we've saved a new version of the input and report differences to the user.

[syvb]

@Mark-Simulacrum One thing that kinda looks like a memory address I've seen is things like this. 70c9 isn't long enough to be an address, but it does look like it might be part of one, or somehow volatile. (but it is not volatile, and doesn't change between runs)

= note: required because it appears within the type `Opaque(DefId(0:39 ~ generator_print_verbose_1[70c9]::make_gen2::{opaque#0}), [std::sync::Arc<std::cell::RefCell<i32>>])`

[jyn514]

I use this for debugging what the compiler is doing, not for actually running tests. I set RUSTDOC_LOG when running tests a lot and it can be difficult to replicate UI tests through rustup, especially when they use aux-build files.

[Mark-Simulacrum]

@bors r+

Seems OK, looks like we have several usage patterns at least.

[bors]

📌 Commit b7e9b1a has been approved by Mark-Simulacrum