Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing the "shallow" torture mode #4699

Closed
wants to merge 2 commits into from
Closed

Conversation

@bagder
Copy link
Member

bagder commented Dec 10, 2019

With a new option to runtests.pl, we can limit the number of failures to cause for each test case and should then possible be able to run over more tests in the CI and elsewhere.

When running over a large amount of tests, torture mode tests the same function failures (in the common code paths) a very large number of times. That then makes them so slow that we can never really run them much and thus they decrease in value.

bagder added 2 commits Dec 10, 2019
When set, shallow mode limits runtests -t to make no more than NUM fails
per test case. If more are found, it will randomly discard entries until
the number is right. The random seed can also be set.

This is particularly useful when running MANY tests as then most torture
failures will already fail the same functions over and over and make the
total operation painfully tedious.
As a first step to enable it to run over a more diverse set of tests in
a reasonable time.
@bagder bagder added the tests label Dec 10, 2019
@bagder bagder mentioned this pull request Dec 10, 2019
7 of 7 tasks complete
@bagder bagder closed this in 7c1bd03 Dec 11, 2019
@bagder bagder deleted the bagder/tests-shallow-torture branch Dec 11, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Mar 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.