Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reporting of failing examples #3529

Closed
Zac-HD opened this issue Dec 11, 2022 · 0 comments · Fixed by #3540
Closed

Improve reporting of failing examples #3529

Zac-HD opened this issue Dec 11, 2022 · 0 comments · Fixed by #3540
Labels
enhancement it's not broken, but we want it to be better legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Dec 11, 2022

It's been a while since we revisited the way that Hypothesis reports failing examples, but I think it's time:

  • We'll need to revisit reporting in order to support Explaining failing examples - by showing which arguments (don't) matter #3411, and it'd be nice to get that ready before the engine changes both to break the work into smaller chunks, and to get improvements into users' hands as soon as possible.
  • I'd like to encourage more use of the @example() decorator, perhaps with .via() on Python 3.9+, and reporting failures in that format seems like an effective way to do so - and makes reproducing a mere matter of copy-pasting when the database doesn't make it fully automatic.

For both of these reasons, using the __repr__ of custom objects is unsatisfying, because they often - and by default - don't consist of executable code which would return an equivalent object. Happily, our friends over at CrossHair recently solved this problem in a way I think we can imitate: represent custom objects by (recursively) representing the call that we executed in order to create them!

You can see their implementation here; in short we can store a tree of fragments (via get_pretty_function_description() etc.) in a dict keyed off the ID of the object, and spanning the lifetime of that example. For anything not in the dict, i.e. where the function call was not executed by Hypothesis builds() or .map() (or flatmap, or recursive, or etc.), we'll fall back to our existing pretty-printing code. One notable divergence: I'll want to store the start and end span of the underlying buffer that generated each fragment, since that's a key component of #3411.

@Zac-HD Zac-HD added enhancement it's not broken, but we want it to be better legibility make errors helpful and Hypothesis grokable labels Dec 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement it's not broken, but we want it to be better legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant