compiletest: ./x test <test-name>.stderr (or existing non-.rs file) should warn or error

[jieyouxu]

Noticed by @WaffleLapkin, if you write

./x test tests/ui/parser/bad-let-else-statement.stderr

then compiletest will silently pass but run 0 tests, because bad-let-else-statement.stderr is a tracked stderr file associated with the actual .rs ui test but not a test itself. This might be surprising or give a false sense of security to the person who's running the test suite thinking that the test passed, whereas it actually didn't even run.

• If the path doesn't exist, compiletest will report the non-existent test error.
• However, if the path does exist, but it's not an actual test (whatever consitutes a "test" file under the relevant test suite), compiletest currently seems to silently ignore and pass.

We should probably warn or error if the path exists (and has an extension) but is not a valid test file.

[veera-sivarajan]

@rustbot claim

[jieyouxu]

Unfortunately as mentioned in #127175, because compiletest is a wrapper around libtest, but the actual test filtering logic is in libtest, compiletest does not have info from libtest as to how many tests actually run, so this isn't a simple one-liner fix.

[jieyouxu]

What we can do when we rework compiletest test discovery and directive handling is to not naively rely on libtest to do the filtering (libtest doesn't even to prefix matching, it just does something whacky like substring contains, which I found extremely unintuitive), otherwise we don't have that info in compiletest. In general, I want compiletest to have better control over how tests are discovered, prepared and packaged to libtest, especially surrounding when tests are ignored/filtered out.