Description
I'd like to add an option to the CLI of the standard test runner so that it stops running tests after the first one fails. This would be useful in situations where you just want to find whether any test fails. I don't propose to change the default.
My immediate motivation is that cargo-mutants just needs to see whether any test fails, and it's a waste of time to continue running tests after one failure has been found.
More generally this is a feature many test runners have and that people seem to find useful. For example, failing fast is the default in nextest: https://nexte.st/docs/running/#failing-fast, Bazel has --test_runner_fail_fast
, and pytest has --exitfirst
. And of course cargo test
fails fast by default at the test target level.
Today, cargo test
will fail fast by default at the test target level: if any tests fail, it won't run any more. However, within the test target, there's no way to fail fast. This can be confusing, but it would be disruptive to change it now.
I discovered that the logic for this actually already exists, it's just not exposed in the CLI. #142807 adds an option. With that change, you can run cargo test -- --fail-fast -Zunstable-options
and it will stop after the first test fails.
When multiple threads are used the tests are run in nondeterministic order, and so in a tree with multiple failing tests, with this option on, it's nondeterministic which tests will get run before the process stops. I don't think that's surprising to people who just want to know of any one failure, and the order can be made predictable by running on a single thread.
I have read that people would like to move away from the current libtest architecture and so apparently there has been a soft feature freeze for some months or years. However since this is a small change in the implementation and shouldn't introduce any compatibility concerns I hope it could still be considered.
Crates can work around the absence of this feature in libtest by setting harness = false
and using Nextest or some other harness, but that's a large transition and I think it would be nice to have it in the standard library: at least, that would help cargo-mutants get better performance on most crates.
cc @rust-lang/testing-devex
Metadata
Metadata
Assignees
Labels
Type
Projects
Status