Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify the docs for the "invocations" config option #591

Closed
sschuberth opened this issue Jan 18, 2019 · 5 comments
Closed

Clarify the docs for the "invocations" config option #591

sschuberth opened this issue Jan 18, 2019 · 5 comments
Assignees
Labels
enhancement ✨ Suggestions for adding new features or improving existing ones.

Comments

@sschuberth
Copy link
Member

From reading this it's unclear to me whether a test with e.g. invocations = 3 will always be run 3 times, or only tried up to 3 times until a test run succeeds.

Also, supposing the test is always run 3 times in my example, what is the behavior if some runs fail and some succeed? Is it enough for a single run to succeed to make the test succeed, or do e.g. the majority of runs have to succeed in order to make the test succeed?

@LeoColman
Copy link
Member

Invocations = 3 will always run 3 times.
If 1 out of 3 fails, the test will fail. All must succeed for the test to be a pass.

@LeoColman LeoColman self-assigned this Jan 18, 2019
@LeoColman LeoColman added the enhancement ✨ Suggestions for adding new features or improving existing ones. label Jan 18, 2019
@LeoColman
Copy link
Member

I will update the docs to reflect this

@sschuberth
Copy link
Member Author

All must succeed for the test to be a pass.

Ah, thanks. I was reading the current "Useful if you have a non-deterministic test and you want to run that particular test a set number of times" exactly the other way around: Instead of guarding against "false positives", i.e. a test succeeded although it shouldn't, I though the purpose is to give the test another change to succeed if it failed before.

@LeoColman
Copy link
Member

Let's take an example of a test that fails:

class Foo : FreeSpec() {

  val counter = AtomicInteger(0)

  init {

    "Foo".config(invocations = 5) {
      val currValue = counter.incrementAndGet()

      if (currValue > 3) {
        fail("Oh no!")
      }
    }

  }
}

The "non-deterministic" counter.increment() will fail at some of the invocations, therefore the test will fail

@sksamuel
Copy link
Member

All must succeed for the test to be a pass.

Ah, thanks. I was reading the current "Useful if you have a non-deterministic test and you want to run that particular test a set number of times" exactly the other way around: Instead of guarding against "false positives", i.e. a test succeeded although it shouldn't, I though the purpose is to give the test another change to succeed if it failed before.

The aim is for those tests that fail every now and then and you have to keep rerunning them to get a failure. By putting invocations = 20 you can flush them out sooner.

LeoColman added a commit that referenced this issue Jan 18, 2019
Users weren't completely sure of how invocations worked. It was possible to think that you wouldn't need all invocations to pass for a test to succeed.

This commit clarifies in the documentation that for a test to pass, all it's invocations must pass, otherwise it will fail.

Fixes #591
LeoColman added a commit that referenced this issue Jan 18, 2019
Users weren't completely sure of how invocations worked. It was possible to think that you wouldn't need all invocations to pass for a test to succeed.

This commit clarifies in the documentation that for a test to pass, all it's invocations must pass, otherwise it will fail.

Fixes #591
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement ✨ Suggestions for adding new features or improving existing ones.
Projects
None yet
Development

No branches or pull requests

3 participants