[BUG] hypothesis.errors.InvalidArgument: test has already been decorated with a settings object.

[dkbarn]

Checklist

• [ X ] I checked the FAQ section of the documentation
• [ X ] I looked for similar issues in the issue tracker
• [ X ] I am using the latest version of Schemathesis

Describe the bug

After upgrading from schemathesis 3.19.5 to 3.26.0, several tests are now failing with this error message:

hypothesis.errors.InvalidArgument: test_graphql_schema has already been decorated with a settings object.

To Reproduce

Here is an example test that produces the above error:

import copy
import falcon
import hypothesis
from hypothesis import HealthCheck
import pytest
import schemathesis

from .util import GRAPHQL_HEADERS, create_database

def graphql_schema_from_asgi(db_setup_factory):
    setup = db_setup_factory()
    # this slows things down but we need this fixture to not be async
    # so schemathesis.from_pytest_fixture can use it correctly
    falcon.async_to_sync(create_database, setup.sql_engine_manager)

    endpoint = "/v1/graphql"
    return schemathesis.graphql.from_asgi(endpoint, setup.app, headers=GRAPHQL_HEADERS)

graphql_schema = schemathesis.from_pytest_fixture("graphql_schema_from_asgi")

@graphql_schema.parametrize()
@hypothesis.settings(
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
    response = case.call_asgi()
    case.validate_response(response)

The problem seems to be the fact that we are decorating our test with both the @graphql_schema decorator generated by schemathesis and the @hypothesis.settings decorator. But we need to do this, in order to customize the behavior of hypothesis for our test.

Expected behavior

The expected behaviour is that a test which passes without issue in 3.19.5 of schemathesis would continue to work in 3.26.0

Environment

- OS: EL7
- Python version: 3.8
- Schemathesis version: 3.26.0

Additional context

[Stranger6667]

Hi! This is weird, I think we should have tests for it.

What is your pytest version?

[Stranger6667]

And pytest-subtests version would be useful to know too

[dkbarn]

Sure thing, the test venv has:
pytest 8.1.1
pytest-subtests 0.7.0

(venv) bash-4.2$ poetry show pytest
 name         : pytest                                      
 version      : 8.1.1                                       
 description  : pytest: simple powerful testing with Python 

dependencies
 - colorama *
 - exceptiongroup >=1.0.0rc8
 - iniconfig *
 - packaging *
 - pluggy >=1.4,<2.0
 - tomli >=1

required by
 - pytest-asyncio >=7.0.0
 - pytest-benchmark >=3.8
 - pytest-cov >=4.6
 - pytest-datadir >=5.0
 - pytest-mock >=6.2.5
 - pytest-regressions >=6.2.0
 - pytest-subtests >=7.0
 - schemathesis >=4.6.4,<9

(venv) bash-4.2$ poetry show pytest-subtests
 name         : pytest-subtests                                 
 version      : 0.7.0                                           
 description  : unittest subTest() support and subtests fixture 

dependencies
 - pytest >=7.0

required by
 - schemathesis >=0.2.1,<0.8.0

[Stranger6667]

I tried to reproduce it with a simpler version but wasn't successful. I also noted that graphql_schema_from_asgi has no @pytest.fixture decorator - its a typo in the original comment, right?

[Stranger6667]

Generally, I don't expect that the exact Python version / Platform would matter here, but will take a closer look.

import hypothesis
import pytest
from hypothesis import HealthCheck
import schemathesis
from test.apps import _graphql


@pytest.fixture
def graphql_schema():
    app = _graphql._fastapi.create_app()
    return schemathesis.graphql.from_asgi("/graphql", app)


schema = schemathesis.from_pytest_fixture("graphql_schema")


@schema.parametrize()
@hypothesis.settings(
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
    response = case.call_asgi()
    case.validate_response(response)

❯ pytest sample.py
======================================= test session starts =======================================
platform darwin -- Python 3.11.7, pytest-8.1.1, pluggy-1.4.0
codspeed: 2.2.0 (callgraph: not supported)
rootdir: /tmp/schemathesis
configfile: pyproject.toml
plugins: syrupy-4.6.1, schemathesis-3.26.0, codspeed-2.2.0, hypothesis-6.98.10, mock-3.10.0, asyncio-0.21.1, trio-0.8.0, doctestplus-0.12.1, anyio-3.7.1, xdist-3.3.1, forked-1.6.0, subtests-0.7.0, httpserver-1.0.6
asyncio: mode=Mode.STRICT
collected 1 item                                                                                  

sample.py ,,,,.

======================================== 1 passed in 3.24s ========================================

How do you run your tests? Maybe there is some CLI arg, that could interfere?

[dkbarn]

You're right, I had left out a key detail from my example code which it turns out is the cause of the error. It seems that the error happens when the schema fixture is parametrized. Here is an example to demonstrate:

import copy
import falcon
import hypothesis
from hypothesis import HealthCheck
import itertools
import pytest
import schemathesis

from .util import GRAPHQL_HEADERS, create_database

INTERFACES = ["v1"]
DB_SCHEMAS = ["db0"]

@pytest.fixture(params=itertools.product(INTERFACES, DB_SCHEMAS))
def graphql_schema_from_asgi(request, db_setup_factory):
    interface, db_schema = request.param
    setup = db_setup_factory(schema=db_schema)
    # this slows things down but we need this fixture to not be async
    # so schemathesis.from_pytest_fixture can use it correctly
    falcon.async_to_sync(create_database, setup.sql_engine_manager)

    endpoint = f"/{interface}/graphql"
    return schemathesis.graphql.from_asgi(endpoint, setup.app, headers=GRAPHQL_HEADERS)

graphql_schema = schemathesis.from_pytest_fixture("graphql_schema_from_asgi")

@graphql_schema.parametrize()
@hypothesis.settings(
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much],
)
def test_graphql_schema(case):
    response = case.call_asgi()
    case.validate_response(response)

[Stranger6667]

Thank you! I can reproduce it now.
Looks like it starts failing if Hypothesis is >=6.84.3, I'll fix it soon

[Stranger6667]

Fixed in 3.26.1

[dkbarn]

Confirmed that the error is now resolved with 3.26.1 - thanks for the super fast turnaround!

[Stranger6667]

Awesome! Happy to help