Alternative internals for fully automatic shrinking + replay of nonserializable cases + interactive data generation

[Zac-HD]

Hi there! I'm the lead maintainer of Hypothesis, the Python property-based testing library, and just saw your talk on JCheck.jl. It's so exciting to see someone working on PBT for Julia!

I'm opening this issue in the hope that I can help you skip past the 2--6 years that it took us to work out a better internal model for PBT, since it looks like you've hit the same port-quickcheck approach that I'd describe as a tempting trap 😬. It's reasonably easy to get something working at first, but seriously hurts both shrinking and replay (and therefore usability) in the longer term. So my proposal is, in short: port the single-file https://github.com/DRMacIver/minithesis to Julia and rebase JCheck on top of that!

• Shrinking is implemented once in the library internals; then every generator is composed out of primitives such that the composed generator also shrinks automatically.
  • This is much easier for users and avoids all kinds of bugs - see https://hypothesis.works/articles/integrated-shrinking/
• Saving and replaying the "choice sequence" is easy (it's a sequence of integers), and suffices to replay inputs from any strategy
• You can generate non-serializable things like "which function to call" and that just works too, even if the options are determined based on earlier parts of the test
• Coverage-based fuzzing integration is also remarkably easy; just get the fuzzer to generate the choice sequence!
• Directed search ("targeted PBT") becomes remarkably easy, with any number of user-supplied objective functions.
• And minithesis is a single file with <900 lines of code, including comments and docs and whitespace, so it's easy to port.

I hope this is helpful! If you'd like to talk in more detail I'd be very happy to have a call sometime too 🙂

[ps-pat]

Thank you for your message. I'm gonna be off the grid for the next few days, but I'd definitely like to have a chat with you about the suggestions you made soon!