[BUG] Parametrization incompatibility with falcon ASGI apps

[BrandonWiebe]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker
• I am using the latest version of Schemathesis

Describe the bug

We've recently upgraded from schemathesis 3.19.5, and our tests that involve parametrized schemas from pytest fixtures have begun to hang indefinitely when using case.call_asgi. I noticed that the call_asgi method was just recently deprecated, so I switched the tests to use just case.call instead, and they no longer hang but instead error out due to calling run_wsgi_app with an ASGI app. The issue is present for both OpenAPI and GraphQL schemas.

Digging in to the issue a little bit, I discovered that these issues only present themselves when using falcon as the ASGI framework. When using FastAPI instead, for example, everything works just fine. I've been unable to dig any further and determine whether this is a bug in falcon, so my apologies if this is not the right venue for this bug, but it seems to have something to do with the way the starlette testclient interacts with the falcon app. Specifically, the hanging seems to occur in the testclient's __exit__ method, where it's waiting for everything to shutdown.

To Reproduce

The following test code should demonstrate both the failing test with the new case.call method, and the hanging test with the old case.call_asgi method. In addition, there are tests using the exact same code/schema via a fastapi app that will pass:

import falcon.asgi
import fastapi
import pytest
import schemathesis


async def get_schema():
    schema = {
        "openapi": "3.0.2",
        "info": {"description": ""},
        "paths": {
            "/info": {
                "get": {
                    "summary": "",
                    "parameters": [],
                    "responses": {
                        "200": {"description": ""}
                    },
                },
            },
        },
    }

    return schema


@pytest.fixture
def falcon_app_schema():
    class InfoResource:
        async def on_get(self, req, resp):
            resp.media = await get_schema()
    
    falcon_app = falcon.asgi.App()
    falcon_app.add_route("/info", InfoResource())
    return schemathesis.from_asgi("/info", falcon_app)


@pytest.fixture
def fastapi_app_schema():
    fastapi_app = fastapi.FastAPI()
    fastapi_app.get("/info")(get_schema)
    return schemathesis.from_asgi("/info", fastapi_app)


fastapi_schema = schemathesis.from_pytest_fixture("fastapi_app_schema")
falcon_schema = schemathesis.from_pytest_fixture("falcon_app_schema")


@fastapi_schema.parametrize()
def test_fastapi_call(case):
    response = case.call()
    case.validate_response(response)


@fastapi_schema.parametrize()
def test_fastapi_call_asgi(case):
    response = case.call_asgi()
    case.validate_response(response)


@falcon_schema.parametrize()
def test_falcon_call(case):
    response = case.call()
    case.validate_response(response)


@falcon_schema.parametrize()
def test_falcon_call_asgi(case):
    response = case.call_asgi()
    case.validate_response(response)

Please include a minimal API schema causing this issue:

{'openapi': '3.0.2', 'info': {'description': ''}, 'paths': {'/info': {'get': {'summary': '', 'parameters': [], 'responses': {'200': {'description': ''}}}}}}

Expected behavior

The result of the above tests should be independent of the ASGI framework used.

Environment

- OS: Linux
- Python version: 3.9.15
- Schemathesis version: 3.27.0
- Spec version: Open API 3.0.2

Additional context

Here are the versions of the libraries directly involved with running the tests that I've been using to reproduce the issue:

falcon                    3.1.3
fastapi                   0.110.1
pytest                    8.1.1
pytest-subtests           0.7.0
schemathesis              3.27.0

[Stranger6667]

Thank you so much for reporting this! I’ll take a look at it soon

[Stranger6667]

I think the issue is that Schemathesis relies on starlette too much when working with ASGI :/ it should be more generic

[Stranger6667]

@BrandonWiebe

#2150 will solve the detection issue, then I am going to adjust the test client so it sends the ASGI spec version to the app. As far as I can see, the ASGI spec defaults to 2.0, but Falcon only supports 3.0 and errors on startup in such cases. Sending the right metadata solves the issue - I'll solve it in starlette-testclient and then will update the dependency here in Schemathesis. Expect the fixes in 3.27.1 likely today / tomorrow

[BrandonWiebe]

Confirming that 3.27.1 has resolved this for me. Thanks a bunch, this is much appreciated!

[Stranger6667]

Awesome! Feel free to reach out in case of any other issues! :)