Initial support for stateful testing

[Liam-DeVoe]

Closes #38. Depends on #50 for a clean merge.

This requires a small amount of refactoring Hypothesis-side, including setting a new runTest._hypothesis_state_machine_class = cls attribute. I don't have a great grasp of the stateful testing internals, so I may be overcomplicating things? It seems that what Hypofuzz really wants to access is the underlying @given which drives stateful test - and to get to that we need to rejig some code.

Here's the Hypothesis-side companion diff, which I can PR to Hypothesis:

diff --git a/hypothesis-python/src/hypothesis/stateful.py b/hypothesis-python/src/hypothesis/stateful.py
index f1dfa5205..ae8ccd36c 100644
--- a/hypothesis-python/src/hypothesis/stateful.py
+++ b/hypothesis-python/src/hypothesis/stateful.py
@@ -83,14 +83,7 @@ class TestCaseProperty:  # pragma: no cover
         raise AttributeError("Cannot delete TestCase")
 
 
-def run_state_machine_as_test(state_machine_factory, *, settings=None, _min_steps=0):
-    """Run a state machine definition as a test, either silently doing nothing
-    or printing a minimal breaking program and raising an exception.
-
-    state_machine_factory is anything which returns an instance of
-    RuleBasedStateMachine when called with no arguments - it can be a class or a
-    function. settings will be used to control the execution of the test.
-    """
+def get_state_machine_test(state_machine_factory, *, settings=None, _min_steps=0):
     if settings is None:
         try:
             settings = state_machine_factory.TestCase.settings
@@ -237,8 +230,21 @@ def run_state_machine_as_test(state_machine_factory, *, settings=None, _min_step
         state_machine_factory, "_hypothesis_internal_use_reproduce_failure", None
     )
     run_state_machine._hypothesis_internal_print_given_args = False
+    return run_state_machine
+
 
-    run_state_machine(state_machine_factory)
+def run_state_machine_as_test(state_machine_factory, *, settings=None, _min_steps=0):
+    """Run a state machine definition as a test, either silently doing nothing
+    or printing a minimal breaking program and raising an exception.
+
+    state_machine_factory is anything which returns an instance of
+    RuleBasedStateMachine when called with no arguments - it can be a class or a
+    function. settings will be used to control the execution of the test.
+    """
+    state_machine_test = get_state_machine_test(
+        state_machine_factory, settings=settings, _min_steps=_min_steps
+    )
+    state_machine_test(state_machine_factory)
 
 
 class StateMachineMeta(type):
@@ -437,6 +443,7 @@ class RuleBasedStateMachine(metaclass=StateMachineMeta):
                 run_state_machine_as_test(cls, settings=self.settings)
 
             runTest.is_hypothesis_test = True
+            runTest._hypothesis_state_machine_class = cls
 
         StateMachineTestCase.__name__ = cls.__name__ + ".TestCase"
         StateMachineTestCase.__qualname__ = cls.__qualname__ + ".TestCase"

[Zac-HD]

Correct, because Hypothesis is responsible for creating the TestCase object, and we don't require Pytest - so it can't need parameters filled in.

[Zac-HD]

This will be very nice to have, and the Hypothesis-side refactoring looks good to me too (with one note above).

[Liam-DeVoe]

will look to merge this after #53; the timing of the hypothesis-side releases means we need compatibility with bufferless hypothesis first

[Liam-DeVoe]

Error only in pytest 7 🤔 will have to figure out what pytest internals changed here

crashed in test_wl561fdj.py::NumberModifierTest::runTest 'function' object has no attribute '__self__'

[Liam-DeVoe]

I gave up because 1) this seems like it could be a rabbit hole to track down, and 2) the requirements to trigger this are rare (pytest 7 + stateful + custom database)

[Zac-HD]

Ugh, I really don't like this but I think it's probably the best option we practically have.

In the medium term, I'd like us to track all the tests-we-don't-collect, and have a dashboard page which lists them and explains why each was skipped (as well as optional terminal reporting? hmm).

[Liam-DeVoe]

yup I'd like that as well, should be doable nearish term. (our dashboard will probably turn into the standard "sidebar on the left which lists pages", and one page is a test overview or similar).

I will come back to this bug at some point, there are just many better things to do in the meantime!

[Liam-DeVoe]

I think I found the relevant pytest change: pytest-dev/pytest#12089 (since pytest 8.2.0). unsure if a fix will be easy

[Zac-HD]

let's make sure we pick frameworks etc which can be responsive / mobile-friendly; the dominant usecase will be on localhost but scaling beyond that is a goal.