-
Notifications
You must be signed in to change notification settings - Fork 646
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clarify the docs for the "invocations" config option #591
Comments
Invocations = 3 will always run 3 times. |
I will update the docs to reflect this |
Ah, thanks. I was reading the current "Useful if you have a non-deterministic test and you want to run that particular test a set number of times" exactly the other way around: Instead of guarding against "false positives", i.e. a test succeeded although it shouldn't, I though the purpose is to give the test another change to succeed if it failed before. |
Let's take an example of a test that fails:
The "non-deterministic" |
The aim is for those tests that fail every now and then and you have to keep rerunning them to get a failure. By putting invocations = 20 you can flush them out sooner. |
Users weren't completely sure of how invocations worked. It was possible to think that you wouldn't need all invocations to pass for a test to succeed. This commit clarifies in the documentation that for a test to pass, all it's invocations must pass, otherwise it will fail. Fixes #591
Users weren't completely sure of how invocations worked. It was possible to think that you wouldn't need all invocations to pass for a test to succeed. This commit clarifies in the documentation that for a test to pass, all it's invocations must pass, otherwise it will fail. Fixes #591
From reading this it's unclear to me whether a test with e.g.
invocations = 3
will always be run 3 times, or only tried up to 3 times until a test run succeeds.Also, supposing the test is always run 3 times in my example, what is the behavior if some runs fail and some succeed? Is it enough for a single run to succeed to make the test succeed, or do e.g. the majority of runs have to succeed in order to make the test succeed?
The text was updated successfully, but these errors were encountered: