Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add status reason to UnsatisfiedAssumption rejections #3830

Merged
merged 10 commits into from
Jan 11, 2024

Conversation

tybug
Copy link
Member

@tybug tybug commented Jan 7, 2024

closes #3827.

Note that what we originally discussed in the issue has the knock-on effect of adding to the events of that test case, which shows up in e.g. statistics mode reporting:

from hypothesis import *
from hypothesis.strategies import *

@given(integers())
def test_f(n):
    assume(n > 2)
sandbox.py::test_f:

  - during reuse phase (0.00 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 1 passing examples, 0 failing examples, 0 invalid examples

  - during generate phase (0.04 seconds):
    - Typical runtimes: < 1ms, of which < 1ms in data generation
    - 99 passing examples, 0 failing examples, 96 invalid examples
    - Events:
      * 49.23%, invalid because: failed to satisfy assume()

  - Stopped because settings.max_examples=100

I'd call this desirable, though perhaps a tad confusing as it doesn't state which assume statement was unsatisfied. But I wonder if there was a good reason this hasn't been done yet.

cc @hgoldstein95

Copy link
Member

@Zac-HD Zac-HD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is being nitpicky, but this looks great to me - merge once it's passing!

hypothesis-python/RELEASE.rst Outdated Show resolved Hide resolved
hypothesis-python/tests/cover/test_statistical_events.py Outdated Show resolved Hide resolved
Comment on lines 26 to 27
def __init__(self, *, reason=None):
self.reason = reason
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently we have a test for whether we have a test for all error classes with custom init methods!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_exceptions_are_picklable actually caught a bug due to this, which is that UnsatisfiedAssumption was no longer picklable after making reason a kwarg. To be honest, reason seems sensible as a posarg instead, so I turned it into one (which consequently fixed this issue).

I'm not particularly familiar with pickling issues like this, so maybe there is a more obvious solution.

@tybug tybug enabled auto-merge January 10, 2024 18:27
@tybug
Copy link
Member Author

tybug commented Jan 10, 2024

Ouch. inspect.stack can get expensive, as seen by 91.48s call hypothesis-python/tests/cover/test_searchstrategy.py::test_example_raises_unsatisfiable_when_too_filtered.

from hypothesis import *
from hypothesis.strategies import *

@given(integers().filter(lambda x: False))
@settings(suppress_health_check=list(HealthCheck))
def f(n):
    pass

try:
    f()
except Exception:
    pass
image

@tybug tybug disabled auto-merge January 10, 2024 19:29
@tybug
Copy link
Member Author

tybug commented Jan 10, 2024

Switched to inspect.currentframe().f_back.f_code.co_name which has a negligible impact.

@tybug
Copy link
Member Author

tybug commented Jan 10, 2024

I would hit the merge button here (you did tell me to!), but I changed some internals since, so I'm going to hold off. Free to yell at me if you think that was a bad call 🙂

@Zac-HD
Copy link
Member

Zac-HD commented Jan 11, 2024

That's all about trusting your judgement, and if you want another review I'm happy to do so. In this case it all looks good, so merging now!

@Zac-HD Zac-HD merged commit 1227013 into HypothesisWorks:master Jan 11, 2024
47 checks passed
@tybug tybug deleted the assume-status-reason branch January 11, 2024 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider splitting observability report status for Status.OVERRUN and Status.INVALID
2 participants