You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm writing integration tests that I'd like randomized, but don't particular care about being exhaustive, because the tests can take a few seconds to run. So I set max_examples to 1 to stop after the first success, and phases to [Phase.generate] to avoid shrinking after a failure. But I noticed that hypothesis still kept rerunning the tests; bumping the verbosity shows multiple "Trying example" logs, which didn't match the expected behavior I thought hypothesis did.
After scouring the docs, I found the report_multiple_bugs setting, and after disabling it, it seems to correctly stop after the first failure. I think my first point of confusion is why report_multiple_bugs is True by default, as it's not the default behavior for other frameworks like Haskell's QuickCheck or Javascript's fast-check. But separately, it seems to me that there is a hidden phase here, in addition to shrinking, that retries runs to find bugs, that should be possible to be disabled, just like the shrinking phase
The text was updated successfully, but these errors were encountered:
Zac-HD
added
docs
documentation could *always* be better
enhancement
it's not broken, but we want it to be better
and removed
docs
documentation could *always* be better
labels
Mar 1, 2022
I think it makes sense to stop generating immediately on failure if shrinking is disabled, even if report_multiple_failures=True, to better match user expectations.
I'm writing integration tests that I'd like randomized, but don't particular care about being exhaustive, because the tests can take a few seconds to run. So I set
max_examplesto 1 to stop after the first success, andphasesto[Phase.generate]to avoid shrinking after a failure. But I noticed that hypothesis still kept rerunning the tests; bumping the verbosity shows multiple "Trying example" logs, which didn't match the expected behavior I thought hypothesis did.After scouring the docs, I found the
report_multiple_bugssetting, and after disabling it, it seems to correctly stop after the first failure. I think my first point of confusion is whyreport_multiple_bugsis True by default, as it's not the default behavior for other frameworks like Haskell's QuickCheck or Javascript's fast-check. But separately, it seems to me that there is a hidden phase here, in addition to shrinking, that retries runs to find bugs, that should be possible to be disabled, just like the shrinking phaseThe text was updated successfully, but these errors were encountered: