[BUG] Stateful testing restarts too soon and API is not explored

[fscarlato]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker

Describe the bug
Stateful testing seems a really powerful for automating the API exploration, but it's difficult to make it work. It just stops too soon and filters lot of data.
I'll attacch a simple case where i get a subset of the schema, configured with links, and run it.

I run the test with pytest -s --log-cli-level=INFO --junitxml=report.xml -v test.py so the output is shown as the tests goes, as you can see the machine is restarted a lot of times, and the actual examples wich are tried are just a few between each run.

Test Output

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 PUT /api-portale/api/v1/organizations/{organization_id}
INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts
INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts
INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:55 RESTART

INFO root:state-machine-ad-hoc-service.py:63 GET /api-portale/api/v1/contacts/{contact_id}
INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/organizations
INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/contacts
INFO root:state-machine-ad-hoc-service.py:63 PUT /api-portale/api/v1/organizations/{organization_id}
INFO root:state-machine-ad-hoc-service.py:63 POST /api-portale/api/v1/organizations
INFO root:state-machine-ad-hoc-service.py:55 RESTART

To Reproduce

I've attacched the openapi yaml, this is the test code, note that rising up the max_examples and stateful_step_count doesn't improve the behaviour.

import schemathesis
import pytest

from hypothesis.stateful import initialize, run_state_machine_as_test
from hypothesis import settings, reproduce_failure, Phase, Verbosity, HealthCheck

from schemathesis.models import Case
from schemathesis.utils import NOT_SET

import logging

logger = logging.getLogger(__file__)
logger.setLevel(logging.INFO)
    
path = "schema-path.yaml"
base_url = "http://localhost:8080/api-portale/api/v1/"

schema = schemathesis.from_path(
    path,
    base_url = base_url
)

auth_headers = { 'X-Cart-username': 'gestore' }


OrganizationWorkflow = schema.clone(operation_id=['addContact', 'getContact', 'createOrganization', 'updateOrganization']).as_state_machine()

class SimpleWorkflow(OrganizationWorkflow):

    def setup(self):
        logging.info("RESTART\n\n")
    #end

    def get_call_kwargs(self, case):
        return {"headers": auth_headers }
    #end

    def before_call(self, case : Case):  
        logging.info(case.operation.verbose_name)
    #end

    def validate_response(self, response, case):
        case.validate_response(response, checks=(schemathesis.checks.status_code_conformance,))
    #end

#end

test_case_1 = SimpleWorkflow.TestCase

Expected behavior
The full machine gets explored by schemathesis. This is a simple case, but rarely it can follow the link createOrganization -> updateOrganization because with these few tries it cannot build randomly a valid Organization.

Furthermore, when I raise the max_examples or the stateful_step_count, the tests stops with:
hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data.

Nothing really changes if I suprress this healthCheck, the state machine still stops too soon.

openapi.yaml.txt

[fscarlato]

Hello, I Just wanted to add that for now we solved this by using the CLI tool with python hooks, and the API exploration goes far more better in respect to the state machine version.
I read somewhere in the issues that the CLI version is moving towards a state machine implementation. In this case this would be a regression since the CLI has a better behavior..

[Stranger6667]

Thank you for opening this and sorry for the delay!

It is indeed true and caused by the internal implementation of Hypothesis state machine. I know where to make change in the Hypothesis codebase, but there is no public API to control it from the outside. Probably it could be just copy-pasted with the change here, but I need to explore it more

[Stranger6667]

Hi @fscarlato! Schemathesis 3.18.3 contains the changes I mentioned (but you need to update Hypothesis to 6.68.1. Schemathesis now should explore the state machine much better. Let me know if there is anything else in stateful testing I can help with