Introducing the "shallow" torture mode #4699
Closed
Conversation
When set, shallow mode limits runtests -t to make no more than NUM fails per test case. If more are found, it will randomly discard entries until the number is right. The random seed can also be set. This is particularly useful when running MANY tests as then most torture failures will already fail the same functions over and over and make the total operation painfully tedious.
As a first step to enable it to run over a more diverse set of tests in a reasonable time.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
With a new option to
runtests.pl, we can limit the number of failures to cause for each test case and should then possible be able to run over more tests in the CI and elsewhere.When running over a large amount of tests, torture mode tests the same function failures (in the common code paths) a very large number of times. That then makes them so slow that we can never really run them much and thus they decrease in value.