Order of given and example decorators matters

[tommilligan]

In the documentation of hypothesis.given, it is explicitly stated:

It doesn’t matter whether you put the example decorator before or after given. Any permutation of the decorators in the above will do the same thing.

However, example must come before given. The below test should fail, however it passes without running the explicit example:

# test_bad.py

from hypothesis import Phase, Verbosity, example, given, settings, strategies

@given(strategies.booleans())
@example(False)
@settings(phases=[Phase.explicit], verbosity=Verbosity.debug)
def test_booleans(boolean: bool) -> None:
    assert boolean

Output:

$ pytest -vvv -s
========================================================= test session starts ==========================================================
platform linux -- Python 3.6.8, pytest-5.2.1, py-1.8.0, pluggy-0.13.0 -- /home/tom/reinfer/env/bin/python3.6
cachedir: .pytest_cache
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/tom/Documents/personal/hypothesis-example-not-run/.hypothesis/examples')
rootdir: /home/tom/Documents/personal/hypothesis-example-not-run
plugins: cov-2.8.1, forked-1.1.1, benchmark-3.2.2, xdist-1.30.0, hypothesis-4.42.3, jobserver-1.0.0, flaky-3.6.1, timeout-1.3.3
collected 1 item

test_bad.py::test_booleans PASSED

========================================================== 1 passed in 0.06s ===========================================================

The following change makes the test fail, as expected:

diff --git a/test_bad.py b/test_bad.py
index 2fd0521..7027831 100644
--- a/test_bad.py
+++ b/test_bad.py
@@ -1,8 +1,8 @@
 from hypothesis import Phase, Verbosity, example, given, settings, strategies


-@given(strategies.booleans())
 @example(False)
+@given(strategies.booleans())
 @settings(phases=[Phase.explicit], verbosity=Verbosity.debug)
 def test_booleans(boolean: bool) -> None:
     assert boolean

---

Platform: Ubuntu 18.04
Python: 3.6.8
Hypothesis: 4.42.3

[tommilligan]

Happy to work on a fix for this, if you can point me in the right direction of where to get started.

[tommilligan]

This was introduced in hypothesis 3.52.0, in commit 92b2dfd

[Zac-HD]

Ah, I think I know what's going on: when @settings is the innermost decorator we return a wrapper that raises the error if called; and @given knows how to retrieve the original test function. If you apply @example in between, it adds the data to that shim... and we forget to copy it back later.

So if you look at how those bits work in core.py a solution should emerge... Happy to give more detailed tips if that would help.