add --max-fail=N to stop after N failures

[MatthewMckee4]

Summary

This change generalizes karva's --fail-fast / --no-fail-fast pair into a new --max-fail=N flag, bringing karva closer to nextest's failing-fast semantics. The motivation is simple: stopping after the first failure is often too aggressive in CI, and running every test to completion is often too lenient. A bounded budget in between — "stop once N tests have failed" — is exactly what nextest offers, and it is what issue #551 asked for.

The new flag accepts a positive integer. Passing --max-fail=3 tells karva to keep scheduling tests until three have failed and then shut down. The same value can be set in karva.toml under [test] as max-fail = 3. Omitting the field leaves the limit unset, which means every test runs — karva's historical default. Everything is wired through the normal TestOptions -> TestSettings path, and the worker processes receive the resolved limit via --max-fail=<n> so the main binary and workers stay in lockstep.

The pre-existing --fail-fast and --no-fail-fast flags keep working. --fail-fast is equivalent to --max-fail=1, and --no-fail-fast clears any limit that was set in karva.toml, restoring the "run everything" behavior. If both forms are present on the CLI, --max-fail wins — this matches nextest's "takes precedence" rule and means users who migrate can drop the old flags at their own pace without any behavior surprises. Every integration and snapshot test that used --fail-fast continues to pass unchanged.

Example

$ karva test --max-fail=2
    Starting 4 tests across 1 worker
        FAIL [TIME] test_max_fail::test_first_fail
        FAIL [TIME] test_max_fail::test_second_fail

diagnostics:

error[test-failure]: Test `test_first_fail` failed
...
error[test-failure]: Test `test_second_fail` failed
...

     Summary [TIME] 2 tests run: 0 passed, 2 failed, 0 skipped

Note that test_third_fail and test_fourth_skipped are never scheduled: once two failures accumulate, the runner stops walking the discovered tree and the summary reflects only what actually ran.

Defaults

Nextest's default is --max-fail=1. Karva's default has historically been "run everything", and this PR intentionally keeps that default to avoid breaking the dozens of existing integration tests that rely on seeing every failure. Users who want nextest-style behavior can opt in with --max-fail=1 or set it globally in karva.toml. Users who set a limit in config and want to override it for a single run can pass --no-fail-fast on the command line. This is called out both in docs/cli.md and in docs/configuration.md, which are regenerated by cargo run -p karva_dev generate-all alongside this PR.

Implementation Notes

The core enforcement lives in PackageRunner, which holds a Cell<u32> of failure counts and compares it against the configured MaxFail after every test variant. The previous boolean fail_fast checks in execute_module and execute_package have been replaced with a single max_fail_reached() helper, so there is only one place that decides "stop scheduling." The worker writes the existing fail-fast signal file once its local budget is exhausted, which causes sibling workers to wind down through the already-existing WorkerManager::wait_for_completion polling loop.

MaxFail itself is a thin newtype wrapper around Option<NonZeroU32> in karva_metadata, with None meaning "no limit" and Some(n) meaning "stop after n failures." Serde's built-in transparent handling covers the TOML form (max-fail = 5 becomes Some(5), omitting the key is None), and clap parses --max-fail directly as NonZeroU32, so there is no custom FromStr or visitor code to maintain.

Closes #551

Test Plan

• just test — full suite passes (824 tests)
• uvx prek run -a — clean
• Integration tests cover the three meaningful cases:
  • basic::test_max_fail_stops_after_n_failures — --max-fail=2 against four failing tests, asserting exactly two run
  • basic::test_no_fail_fast_runs_every_test — --no-fail-fast runs every test even when some fail
  • basic::test_max_fail_one_is_equivalent_to_fail_fast — --max-fail=1 matches the historical fail-fast behavior
• configuration::test_max_fail_from_config exercises the karva.toml form
• Existing test_fail_fast, test_fail_fast_across_modules, test_fail_fast_true, test_fail_fast_false, and test_cli_fail_fast_overrides_config continue to pass, confirming the legacy flags behave identically

[codspeed-hq]

Merging this PR will degrade performance by 10.95%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 (👁 1) regressed benchmark

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
👁	WallTime	karva_benchmark	22.3 s	25 s	-10.95%

---

_{Comparing feat/max-fail (beaa5c5) with main (e536e42)}