feat(cli): --repeat to run each test n times - #1041
Merged
Merged
Conversation
--retry mitigates flakiness after it has already burned a CI run. There was no way to go looking for it: proving a test deterministic meant running the suite N times by hand and diffing the output. --repeat wraps the existing retry loop rather than replacing it, so repeat is the outer loop and retry the inner one: each iteration gets its full retry budget before the next iteration starts. The test is reported once with the aggregate outcome, so the counters and every report see one entry, and the assertion counts are the deciding iteration's rather than the sum of all of them. Iterating stops at the first failing iteration, since the test is already going to be reported failed. The failure message names the iteration, which is the only way the count is actionable when the test appears once. 0, a negative value and a non-numeric value are usage errors rather than silent no-ops: 0 passes the non-negative check the other integer settings use, so it needs its own guard (#873 class of bug). Closes #1013
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1013
--retrymitigates flakiness only after it has already burned a CI run. There was no way to go looking for it: proving a test deterministic meant running the suite N times by hand and diffing the output.💡 Changes
--repeat <n>runs each selected test n times, so--repeat 50 --filter flaky_candidateis the natural "hammer this one test" invocation.--retrythe inner one: each iteration gets its full retry budget before the next starts. Documented and covered by a test.--repeat 0, negative and non-numeric values exit non-zero with a usage error rather than becoming silent no-ops.