Strategies from type descriptions

[averagehat]

Referring to the typing module. This would make it easy to generate structured mock data. The recent PEP thing got me interested, as well as python/mypy which @Daenyth clued me in on.

• NamedTuple
• Automatic function arguments
  The first is basically done (some work here), although not all possible types are supported.

The second is similarly straight forward in python3. But I'm not sure if it's possible to get the function annotations from in py2. I'd welcome info on this if anyone knows.

[Zac-HD]

I've been discussing this with @DRMacIver by email, copied below for anyone interested and posterity.

[Zac-HD]

Hi David

I've been looking at (this issue) over the weekend, and I think I could put together a solid extras module for it. I'd be targeting extras because it's only practical on 3.5+. There are also some cases where there's no reasonable output, and I'd like to build something solid but with clearly defined boundaries - ie automatic, not magic. Or at least not terribly black magic...

This could be useful in several ways. I imagine it would make Hypothesis a little more popular with mypy enthusiasts, but more interesting to me is the things you could build on top. For example, a generic test which asserts that two functions have identical output (oracle pattern), two functions invert each others output (round-trip), check idempotence, etc - a library of predefined properties that a user can just drop their functions into.

Before I go through the less-exciting bits of handling all the fiddly edge cases, docs, tests etc, would you accept such a pull?

[Zac-HD]

(reply email by @DRMacIver)

"It's complicated."

I'm not opposed to such a pull request, but there are a huge number of gotchas and difficult things that I think the feature would require to be useful. So I'm not saying no, but I am warning you that between my requirements and the intrinsic difficulty of the task it's going to be a fairly big undertaking.

Here is my minimum set of requirements for such a feature (note: All names are made up on the spot and you should feel free to choose your own):

1. The mechanism for converting types to strategies must be extensible to include new types (this requires taking a stance on using some sort of external dispatch mechanism, which so far I've avoided doing. functools.singledispatch looks the least controversial but also like it might not be very good)
2. This includes for types that are not under your control.
3. There must be a way of getting the strategy for a type object that is independent of using type annotations
4. There must be a way of overriding bits of the inferred strategy. e.g. suppose I have a named tuple Foo with fields a, b, c I should be able to do strategy_from_type(Foo, b=st.integers(1, 2)) without having to specify the other fields.
5. given must not use a type annotation as a strategy without some form of explicit instruction to do so (e.g. my_param_name=auto()).

There might be others I'm currently forgetting!

The fundamental reason for these requirements this: Type based property-based testing is an attractive nuisance. It's convenient but badly limited, and works nicely for like 80% of the use case and then completely screws you over for the remaining 20%. Hypothesis's strategy library is much more flexible than anything you could possibly do with type based approaches. Anything which makes it even slightly inconvenient for people to switch from using type based strategies to explicitly specified ones is in the end a net negative for the end user.

On top of this:

1. Hypothesis currently uses getargspec internally for everything. You are probably going to need to rebuild a lot of this on top of signature to properly propagate annotation information. This will require appropriate care and feeding of backports (though I think there is a reasonably functional signature backport)
2. given's argument juggling logic is awful and you're going to have to touch that
3. Many of the things you will need to make this work properly will rely on undocumented or underdocumented features of the typing module.
4. I don't really think the typing module is well thought out or has had its edge cases properly explored, so I bet you'll find some fun stuff lurking in there

If despite these caveats you still want to do it, I'd be happy to help with any questions you might etc. from fairly early on in the process. I do think this feature is probably inevitable at some point, it's just a bit scary and I'm not in any rush.

[Zac-HD]

(me again)

I think we have different things in mind, which is complicating the discussion: you seem to be talking about an upgrade to given which can infer omitted inputs to the test function. My proposal would be (slightly) less ambitious: a helper function which takes a type and returns a strategy to generate values of that type, and a helper function to look up a strategy for each argument of an annotated function then draw, call, and assert the return type is correct.

This would automatically meet requirements R3 and R5. R1 and R2 are basically the same thing; conceptually a parameter for a supplementary lookup dict of type: strategy would suffice - in practice most of typing.* requires checking with issubtype and worse, but I'm confident that an acceptable API is possible. For R4, a similar overriding mechanism would work - again, getting a clean and orthogonal API will take work but I have some inelegant-but-functional ideas already so I'm pretty sure it's possible.

Challenges... C1 and C2 I think are bypassed by the "lookup with helper functions" approach; I don't expect people to use this on Hypothesis internals (which would need to be type-annotated first...). C3 and C4... I agree. The typing module is young, there's no consistent way to get the element type of container types, backward compatibility is terrible, etc.

I think this is all manageable - if a lot of work to cover all the edges and provide a nice interface - but workable if I can stick it in an extra that requires 3.5+ and avoid doing too much to the internals. Does this scope seem reasonable to you?

[Zac-HD]

(reply email by @DRMacIver)

On to the design issues. I think we have different things in mind, which is complicating the discussion: you seem to be talking about an upgrade to given which can infer omitted inputs to the test function. My proposal would be (slightly) less ambitious: a helper function which takes a type and returns a strategy to generate values of that type,

That seems much more reasonable, yes, and is largely the subset of this feature that I think is a good idea, so I'm keen. :-)

and a helper function to look up a strategy for each argument of an annotated function then draw, call, and assert the return type is correct.

Can you outline a use case for this? Would this be used instead of given? Or within given? Either way this seems to either fall afoul of making it difficult to transition between type based and strategy based test definitions.

Also what do you have in mind for asserting that the return type is correct? Are you just planning to look at function annotations or dynamically check the resulting data?

This would automatically meet requirements R3 and R5. R1 and R2 are basically the same thing; conceptually a parameter for a supplementary lookup dict of type: strategy would suffice - in practice most of typing.* requires checking with issubtype and worse, but I'm confident that an acceptable API is possible. For R4, a similar overriding mechanism would work - again, getting a clean and orthogonal API will take work but I have some inelegant-but-functional ideas already so I'm pretty sure it's possible.

Challenges... C1 and C2 I think are bypassed by the "lookup with helper functions" approach; I don't expect people to use this on Hypothesis internals (which would need to be type-annotated first...).

It probably would be nice to get things type annotated, especially the strategies module. Might help with the "check the return type is correct" part too. Up to you if you want to tackle that though.

C3 and C4... I agree. The typing module is young, there's no consistent way to get the element type of container types, backward compatibility is terrible, etc.

I think this is all manageable - if a lot of work to cover all the edges and provide a nice interface - but workable if I can stick it in an extra that requires 3.5+ and avoid doing too much to the internals. Does this scope seem reasonable to you?

It does. I'd quite like it if you could look into how much work it is to make this work with the backport of the typing module - if the answer is little to none then it would be nice to make it work across all the Python versions Hypothesis supports. If the answer is lots then I have no problem restricting it to 3.5+ only. Even if you do, it might be worth shimming the typing module into some sort of hypothesis.internal.compat_typing to deal with the inevitable breaking API changes that the CPython team will foist on us (provisional APIs in the standard library *shakes tiny fist*)

[Zac-HD]

Now we're out of emails territory, and I'm only writing as myself. Hopefully that wasn't too hard to follow!

Can you outline a use case for [strategies from annotated functions]? Would this be used instead of given? Or within given? Either way this seems to either fall afoul of making it difficult to transition between type based and strategy based test definitions.

It's a lookup function that returns a strategy, just like the type-based lookup (and hopefully merged). So you call it with a function, and get a strategy - inline in @given(st.example(my_func)), assign to a variable, whatever. Essentially it's treating annotated functions as a form of Callable[[*arg_types], ret_type] where we have more precise constraints on the return value; this may also apply for custom classes with an annotated __init__ method.

The inference / manual transition is no better or worse than for purely type-based lookup - plenty of work but manageable.

Also what do you have in mind for asserting that the return type is correct? Are you just planning to look at function annotations or dynamically check the resulting data?

Each time the search strategy draws, it asserts that the return value is an instance of the annotated type before returning it. I'm not certain how failures show up (ie at draw time), but I think it's a health problem for the strategy to return values of the wrong type.

It probably would be nice to get things type annotated, especially the strategies module. ... Up to you if you want to tackle that though.

I think that's a logically separate pull - and this work will be large enough as it is.

I'd quite like it if you could look into how much work it is to make this work with the backport of the typing module - if the answer is little to none then it would be nice to make it work across all the Python versions Hypothesis supports. If the answer is lots then I have no problem restricting it to 3.5+ only. Even if you do, it might be worth shimming the typing module into some sort of hypothesis.internal.compat_typing to deal with the inevitable breaking API changes in CPython

Hmm. I think it's probably possible to make this work with the backport, but without type annotations (3.5+, 3.3 for any annotations IIRC) it's going to be substantially less useful. Let's park this for now - I agree that it would be nice, but I'll develop a working version for 3.5 first and then revisit backports.

Same principle for a compatibility shim - it may well be necessary or just desirable at some point, but I'll get a MVP without it first.

[DRMacIver]

It's a lookup function that returns a strategy, just like the type-based lookup (and hopefully merged).

Right. I misunderstood what you meant here. That sounds fine.

A thing that might be worth considering is instead of making this a separate thing, add it into the builds strategy so that any missing arguments with annotations are automatically filled in - this is a backwards compatible change because everywhere this works would previously have been an error.

Each time the search strategy draws, it asserts that the return value is an instance of the annotated type before returning it.

I suspect you will find this finicky - a lot of return types cannot be checked at runtime (e.g function types), or are expensive to check (e.g. parametrised lists checking the element types). I've no objection in principle to it doing this though.

I think that's a logically separate pull
Let's park this for now - I agree that it would be nice, but I'll develop a working version for 3.5 first and then revisit backports.
Same principle for a compatibility shim - it may well be necessary or just desirable at some point, but I'll get a MVP without it first.

These are all fine by me.

[Zac-HD]

add [argument strategy inference] into the builds strategy so that any missing arguments with annotations are automatically filled in

I like the idea, but would be nervous about inference in builds only working when the extra is available (or dependencies, whatever they end up being). Coming at it from the other direction though, if you can override the inference for specific arguments it almost replaces builds (depending on how type/strategy registration and dispatch ends up working).

I suspect you will find [checking return types] finicky or expensive

It turns out that isinstance(['a'], List[int]) == True, so apparently CPython just gives up on this. My options appear to be (a) don't check at all; (b) follow runtime behaviour which is cheap and easy; or (c) expensive and finicky. It's better to make users check their invariants than do it badly for them, so I prefer (c) but will fall to (a) if a robust implementation is impractical.

[Zac-HD]

I now have a WIP draft, so it's probably time to think about how this might be merged. I see two main and one optional parts to the work.

1. A function to resolve some type to the corresponding SearchStrategy. This is largely implemented in the linked branch. Open design questions include:

   • Should this be available on Python2, without the typing module? I lean towards yes, though it will be relatively crippled for later stages.
   • What should this function be called? Provisional name is resolve
   • What module should it be defined in? Provisionally an extra, but see the first bullet.
   • The above points presume that this should be publicly visible; it's of situational usefulness only but I think still worth exposing.
   • Should irresolvable types raise an exception, or return nothing()? The former provides nicer error messages (compare a healthcheck failing), but the latter allows for easy composition.

   Process: as usual for a large chunk of work, which is coming along nicely. Could be merged in ~weeks to a few months. I'll open a pull soon(ish), once the initial design seems to be working and the location is sorted out.

2. Using resolve, exploit type annotations in user code. Suggested process - wait until it becomes clear while working through resolve.

   • Could be an extras module, or could be an upgrade to builds(...)
   • Will probably require or at least entail some changes to Hypothesis' argspec introspection
   • Needs to support mixing of inference and user-supplied details for some parameters, both for power of the API and to support the final point

3. Provide a library of properties that user functions can be dropped into. No process as it's motivational only for now.

   • examples from property testing literature:
     • never_crashes(dut, allowed_exceptions=())
     • round_trip(*funcs, initial: lambda x: x, eq_by=operator.eq)
     • oracle(oracle, dut, eq_by=operator.eq)
   • Will probably require something derived from inspect.Signature (thinking about binding args) and SearchStrategy, and a generic way to execute an example from this (something like the latter may already exist internally?)

@DRMacIver, any feedback you have on 1 would be welcome so I can work on things useful for a pull; everything else should fall out nicely as we go from there.