-
Notifications
You must be signed in to change notification settings - Fork 579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Observability features to support deeper understanding + future Tyche integration #3797
Conversation
9e7429b
to
0865e60
Compare
a73e083
to
c72dd55
Compare
Not that my review is a blocker by any means, but I'll take a look at this tomorrow or saturday! Just wanted to let you know I'd seen the review request. |
Thanks, @tybug! I'm not in any particular hurry on this one, because @hgoldstein95 and I haven't quite finalized the format or interface yet. I think we're pretty close, but it'd be nice to see a ready-to-ship version of Tyche based on this PR before we ship it, to avoid needing breaking changes shortly after release 😅 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some minor comments, and basically an approve otherwise. I wouldn't take this as an endorsement of correctness - but I did at least read every line in the diff 🙂
I'm super excited to see where this goes, by the way! I would love to have better observability of my own property tests, particularly input distribution.
78f6956
to
202408b
Compare
202408b
to
239c836
Compare
Merging this now, because I keep talking to people with use-cases where I'd suggest it, and we seem to have stabilized the format and interface without any recent issues 😁 |
I've been corresponding with @hgoldstein95 for a while now, and last week we met up in person too. After talking about and hacking on https://github.com/tyche-pbt/ (paper) - a neat in-editor frontend to see what Hypothesis is doing - we hit on the concept of observability for PBT.
As well as reporting the minimal failing example, let's log every test case - including argument values, test outcome, runtimes, code coverage, events and target values, etc. If all you need is the minimal example, great - but if you need more, you can treat (more of) debugging as a data analysis problem!
In a future version, we're aiming for automatic integration with the Tyche vs code extension and will encourage other editors to add support - e.g. PyCharm devs were interested in Tarantula-style coverage visualization when I mentioned the idea to them at PyCon '23.
Status of this PR: fleshed-out demo, waiting for some decisions + docs + upstream work but pretty compelling
InterestingOrigin
string form