Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving Docs regarding shrinking and finding multiple errors #3741

Closed
sherry30 opened this issue Sep 14, 2023 · 1 comment
Closed

Improving Docs regarding shrinking and finding multiple errors #3741

sherry30 opened this issue Sep 14, 2023 · 1 comment
Labels
docs documentation could *always* be better question not sure it's a bug? questions welcome

Comments

@sherry30
Copy link

Hi,
There are a few things which are unclear to me after going through the Docs.
I think these things should be explicitly mentioned to make things clear.

  1. Does hypothesis immediately start the shrinking process after it finds 1 failure or does it continue to run the test in search of new failures and then shrinks all the different failures at the end?
  2. Under this section, it says that it might run fewer times if a failure is found early. That is a bit confusing since that could mean that it stops after it finds 1 failure. If it doesn't stop after 1 failure then how does it determine when to stop (considering it found a bug early and will stop before reaching max examples)?
  3. While its shrinking, is it possible to find new failures?

Maybe these things can also be added to the Details and advanced features section so it'll be easier for the users to get a clear idea about the work flow of hypothesis.

It is possible that some of these things are already mentioned in the docs and I missed them, so sorry about that in advance
Thanks

@Zac-HD Zac-HD added question not sure it's a bug? questions welcome docs documentation could *always* be better labels Sep 15, 2023
@Zac-HD
Copy link
Member

Zac-HD commented Sep 15, 2023

  1. We have distinct phases for generation and shrinking, both of which can discover new bugs.
  2. After we find a bug, we might keep generating for a while before switching to the shrink phase. Here's the heuristic.
  3. Yes, it's possible to find new failures while shrinking - in fact this is a well-known effect and discussed in e.g. Multi-bug discovery during generation #847.

We've avoided documenting these behaviors so that they're explicitly not part of the API compatibility guarantee, and we're therefore able to make improvements which would otherwise be breaking changes much more easily.

@Zac-HD Zac-HD closed this as completed Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs documentation could *always* be better question not sure it's a bug? questions welcome
Projects
None yet
Development

No branches or pull requests

2 participants