Skip to content

FR: Add a --fail-fast option to libtest #142859

Open
@sourcefrog

Description

@sourcefrog

I'd like to add an option to the CLI of the standard test runner so that it stops running tests after the first one fails. This would be useful in situations where you just want to find whether any test fails. I don't propose to change the default.

My immediate motivation is that cargo-mutants just needs to see whether any test fails, and it's a waste of time to continue running tests after one failure has been found.

More generally this is a feature many test runners have and that people seem to find useful. For example, failing fast is the default in nextest: https://nexte.st/docs/running/#failing-fast, Bazel has --test_runner_fail_fast, and pytest has --exitfirst. And of course cargo test fails fast by default at the test target level.

Today, cargo test will fail fast by default at the test target level: if any tests fail, it won't run any more. However, within the test target, there's no way to fail fast. This can be confusing, but it would be disruptive to change it now.

I discovered that the logic for this actually already exists, it's just not exposed in the CLI. #142807 adds an option. With that change, you can run cargo test -- --fail-fast -Zunstable-options and it will stop after the first test fails.

When multiple threads are used the tests are run in nondeterministic order, and so in a tree with multiple failing tests, with this option on, it's nondeterministic which tests will get run before the process stops. I don't think that's surprising to people who just want to know of any one failure, and the order can be made predictable by running on a single thread.

I have read that people would like to move away from the current libtest architecture and so apparently there has been a soft feature freeze for some months or years. However since this is a small change in the implementation and shouldn't introduce any compatibility concerns I hope it could still be considered.

Crates can work around the absence of this feature in libtest by setting harness = false and using Nextest or some other harness, but that's a large transition and I think it would be nice to have it in the standard library: at least, that would help cargo-mutants get better performance on most crates.

cc @rust-lang/testing-devex

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-libtestArea: `#[test]` / the `test` libraryC-feature-requestCategory: A feature request, i.e: not implemented / a PR.T-libs-apiRelevant to the library API team, which will review and decide on the PR/issue.T-testing-devexRelevant to the testing devex team (testing DX), which will review and decide on the PR/issue.

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions