Alternative API for stateful tests in Python

[Stranger6667]

Even though now all stateful tests are separate pytest items it might be better to run all of them in a single test function. Something like this:

@schema.parametrize()
def test_api(case):
    for step in case.get_stateful_tests():
        response = step.call()
        step.validate_response(response)

It might be much easier to implement since there will be no need to integrate it with pytest itself. Also - there will be no mangled execution statistic in the pytest output because it is a single test, there will be no problems with changing hypothesis settings in subsequent steps, etc

[Zac-HD]

Have you seen REST-ler? It doesn't look like it's open source, but there are some nice ideas about inferring dependencies and order of calls that could enhance stateful mode 🙂

Building schemathesis' stateful mode on top of Hypothesis' RuleBasedStateMachine would be really nice - especially with {A,W}SGI mode so you can reset the app between test runs. Longer-term that would even allow coverage-guided fuzzing for webapps...

[Stranger6667]

Yep, I actually created this issue after reading that paper :) The current implementation with pytest is quite clunky - it modifies some internals between test executions. I hope that this implementation will be re-iterated in the next major release to incorporate ideas from REST-let & #520.

Building schemathesis' stateful mode on top of Hypothesis' RuleBasedStateMachine would be really nice

I tried it a couple of times but decided to go with a more straightforward approach - incorporating RuleBasedStateMachine will require some significant refactorings, but I think it worth it in the long run :)

Longer-term that would even allow coverage-guided fuzzing for webapps...

That sounds really cool! I ran some experiments with Hypothesis' target, and it was promising :) But it is still a huge area to research

[Stranger6667]

As a sidenote

HypoFuzz looks awesome to me :) Wondering what bugs were found in hypothesis-jsonschema as mentioned in the text above the dashboard :)

[Zac-HD]

• Unhandled OverflowError when checking multipleOf with large numbers python-jsonschema/jsonschema#743
• {'not': {'anyOf': [{'type': 'integer'}, {'type': 'number'}]}} should canonicalise to {'type': ['null', 'boolean', 'string', 'array', 'object']}, but produces an intermediate schema (which does) instead. Similar to the known issue Failing test_canonicalises_to_equivalent_fixpoint python-jsonschema/hypothesis-jsonschema#64, if less serious.
• Something about numeric bounds that I've had trouble reproducing; could be a bug in my test harness.

[Stranger6667]

Implemented in #805 🎉