Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop silently dropping examples when the @example() decorator is applied to itself #3319

Closed
Zac-HD opened this issue Apr 30, 2022 · 0 comments · Fixed by #3334
Closed

Stop silently dropping examples when the @example() decorator is applied to itself #3319

Zac-HD opened this issue Apr 30, 2022 · 0 comments · Fixed by #3334
Labels
legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Apr 30, 2022

Suppose you wanted to write a function which added explicit @hypothesis.example()s to a function for the cross-product of a few sequences. A correct implementation would look something like:

import itertools
from hypothesis import example

def cross_product_examples(**kwargs):
    def inner(f):
        params, values = zip(*kwargs.items())
        for value_set in itertools.product(*values):
            kw = dict(zip(params, value_set))
            f = example(**kw)(f)
    return inner

But if you forget the inner function, you'd end up with something like this:

def buggy_cross_product_examples(**kwargs):
    # This function is broken in a nastily subtle way: everything will appear
    # to work, *except* that only the last-defined example will ever be run!
    params, values = zip(*kwargs.items())

    decorator = lambda f: f
    for value_set in itertools.product(*values):
        kw = dict(zip(params, value_set))
        # Here's the bug: we're applying the @example() decorator to *another decorator*,
        # not to the test function.  This silently discards everything from previous loop
        # iterations, and so we'll only ever run the last-defined example.
        decorator = example(**kw)(decorator)

    return decorator

To fix this, we should modify the definition of @example() (below), so that the hypothesis_explicit_examples list is defined outside of accept, and added as an attribute on both test and accept.

def example(*args: Any, **kwargs: Any) -> Callable[[TestFunc], TestFunc]:
"""A decorator which ensures a specific example is always tested."""
if args and kwargs:
raise InvalidArgument(
"Cannot mix positional and keyword arguments for examples"
)
if not (args or kwargs):
raise InvalidArgument("An example must provide at least one argument")
def accept(test):
if not hasattr(test, "hypothesis_explicit_examples"):
test.hypothesis_explicit_examples = []
test.hypothesis_explicit_examples.append(Example(tuple(args), kwargs))
return test
return accept

For a complete pull request ("PR"), you'll also need to add a test which checks that examples are not lost when you chain together e.g. example("outer")(example("inner"))(f), as well as writing a RELEASE.rst and adding your name to the AUTHORS.rst list.

@Zac-HD Zac-HD added the legibility make errors helpful and Hypothesis grokable label Apr 30, 2022
This was referenced May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant