Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better docs for stopping immediately after failure #3244

Closed
brandon-leapyear opened this issue Mar 1, 2022 · 1 comment · Fixed by #3251
Closed

Better docs for stopping immediately after failure #3244

brandon-leapyear opened this issue Mar 1, 2022 · 1 comment · Fixed by #3251
Labels
enhancement it's not broken, but we want it to be better

Comments

@brandon-leapyear
Copy link
Contributor

I'm writing integration tests that I'd like randomized, but don't particular care about being exhaustive, because the tests can take a few seconds to run. So I set max_examples to 1 to stop after the first success, and phases to [Phase.generate] to avoid shrinking after a failure. But I noticed that hypothesis still kept rerunning the tests; bumping the verbosity shows multiple "Trying example" logs, which didn't match the expected behavior I thought hypothesis did.

After scouring the docs, I found the report_multiple_bugs setting, and after disabling it, it seems to correctly stop after the first failure. I think my first point of confusion is why report_multiple_bugs is True by default, as it's not the default behavior for other frameworks like Haskell's QuickCheck or Javascript's fast-check. But separately, it seems to me that there is a hidden phase here, in addition to shrinking, that retries runs to find bugs, that should be possible to be disabled, just like the shrinking phase

@Zac-HD Zac-HD added docs documentation could *always* be better enhancement it's not broken, but we want it to be better and removed docs documentation could *always* be better labels Mar 1, 2022
@Zac-HD
Copy link
Member

Zac-HD commented Mar 7, 2022

I think it makes sense to stop generating immediately on failure if shrinking is disabled, even if report_multiple_failures=True, to better match user expectations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement it's not broken, but we want it to be better
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants