Pull out timing observations, more jsonable arguments

[Zac-HD]

Plus a note in the docs inspired by discussion on tyche-pbt/tyche-extension#3.

Cc @hgoldstein95 for timing metadata per your email.

• Report generate:{argname} separately for each argument (and interactive draw). If we update the too_slow healthcheck message to include observability info (or suggest turning it on), this will fix Too-slow health check doesn't say what was too slow #434!

[Zac-HD]

Hmm. With a bit of work on the internals, we could instead report the draw time for each argument, corresponding to the entries in the arguments mapping (i.e. each argument to @given() and each interactive draw from st.data(), plus a residual from e.g. st.functions()). @hgoldstein95, do you think that'd be useful?

[tybug]

pulling out timing data into its own key makes a lot of sense, and reporting per-argument draw times would be even better! In a perfect world I would love to observe the timings of every component strategy draw as well 🙂 (I mentioned as much to @hgoldstein95 in our interview).

[Zac-HD]

pulling out timing data into its own key makes a lot of sense, and reporting per-argument draw times would be even better! In a perfect world I would love to observe the timings of every component strategy draw as well 🙂

Component strategy draws are hard, because we don't have human-readable labels. We could add compute and collect those, but we'd then have the problem that most component strategy draws are internals rather than a meaningful component in the user-facing API/mental model. I remain pretty nervous about introducing too much complexity, and improving life for maintainers and power users at the expense of novices.

Splitting out per-argument timing is definitely doable though, and that is certainly easy to interpret. Keys execute_test, and generate:{argname} for each argname key in the arguments mapping 😁

[tybug]

I'll be forced to get familiar with our reporting code at some point, but I'm not there yet, so consider this less of a code review and more of a "ensure I generally understand what's going on" review 🙂

That said, I'm on board with the design here and the choice of label names etc. 👍 just one comment about the layout of the reported draw timings.

[tybug]

personally, I'd expect this to read left to right: slowest draws on the left, and fastest on the right. I have no scientific backing for this besides an opinion, though.