Observability features to support deeper understanding + future Tyche integration

[Zac-HD]

I've been corresponding with @hgoldstein95 for a while now, and last week we met up in person too. After talking about and hacking on https://github.com/tyche-pbt/ (paper) - a neat in-editor frontend to see what Hypothesis is doing - we hit on the concept of observability for PBT.

As well as reporting the minimal failing example, let's log every test case - including argument values, test outcome, runtimes, code coverage, events and target values, etc. If all you need is the minimal example, great - but if you need more, you can treat (more of) debugging as a data analysis problem!

In a future version, we're aiming for automatic integration with the Tyche vs code extension and will encourage other editors to add support - e.g. PyCharm devs were interested in Tarantula-style coverage visualization when I mentioned the idea to them at PyCon '23.

---

Status of this PR: fleshed-out demo, waiting for some decisions + docs + upstream work but pretty compelling

• release notes and documentation
  • ship schema file
  • include schema in the docs with nice typography (e.g. descriptions off to the right / in ⓘ expandos)
• grep for FIXME and TODO comments
• lots of tests
  • InterestingOrigin string form
  • check that our testcase json blobs work reliably
• get a release out which doesn't mention or explicitly integrate with Tyche. Then as followup:
  • detect when running in VSCode tests without Tyche and recommend it - see master...Zac-HD:tyche-notes
  • include the client-side logic for Tyche in Hypothesis itself so users can just install the vsc extension
  • email Pycharm devs about including their own integration

[tybug]

Not that my review is a blocker by any means, but I'll take a look at this tomorrow or saturday! Just wanted to let you know I'd seen the review request.

[Zac-HD]

Thanks, @tybug! I'm not in any particular hurry on this one, because @hgoldstein95 and I haven't quite finalized the format or interface yet. I think we're pretty close, but it'd be nice to see a ready-to-ship version of Tyche based on this PR before we ship it, to avoid needing breaking changes shortly after release 😅

[tybug]

Some minor comments, and basically an approve otherwise. I wouldn't take this as an endorsement of correctness - but I did at least read every line in the diff 🙂

I'm super excited to see where this goes, by the way! I would love to have better observability of my own property tests, particularly input distribution.

[Zac-HD]

Merging this now, because I keep talking to people with use-cases where I'd suggest it, and we seem to have stabilized the format and interface without any recent issues 😁