Issues on using schemathesis with pytest-aiohttp

[playpauseandstop]

Hi,

First, thanks for a great library and an effort behind!

I'm happy to try it, but ran into the issue, after tried to integrate it with my aiohttp.web application via next test case:

from pathlib import Path

import schemathesis
from petstore.app import create_app


schema = schemathesis.from_path(
    Path(__file__).parent / "petstore-expanded.yaml"
)


@schema.parametrize()
async def test_petstore(aiohttp_client, case):
    client = await aiohttp_client(create_app())
    response = await client.request(
        case.method, case.formatted_path, headers=case.headers, json=case.json
    )
    assert response.status < 500

Where,

• aiohttp_client is a fixture from pytest-aiohttp
• petstore.app:create_app factory for creating web.Application

This test case results in,

E       fixture 'case' not found
>       available fixtures: aiohttp_client, aiohttp_raw_server, aiohttp_server, aiohttp_unused_port, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, fast, loop, loop_debug, monkeypatch, no_cover, proactor_loop, pytestconfig, raw_test_server, record_property, record_testsuite_property, record_xml_attribute, recwarn, subtests, test_client, test_server, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, unused_port
>       use 'pytest --fixtures [testpath]' for help on them.

after running pytest as,

python -m pytest /path/to/file/with_test.py

More data that might be helpful,

• Python 3.7.4
• OS: macOS 10.14.4
• pytest: 5.2.1
• hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/playpauseandstop/Projects/rororo/.hypothesis/examples')
• pytest plugins: schemathesis-0.8.1, subtests-0.2.1, aiohttp-0.3.0, hypothesis-4.39.0, cov-2.8.1

---

Maybe, I got the concept behind schemathesis wrong, but I thought that I'm being able to,

• Make the schema instance from the OpenAPI 3.0 schema file, which stores next to test case module
• Wrap my test case with @schema.parametrize() decorator
• Create my web.Application and test aiohttp_client inside the test
• Run the test
• And ensure that schema from the file covered well by my aiohttp.web application

If this not possible, I will happy to know on proposed workflow to test my aiohttp.web application with schemathesis.

[Stranger6667]

Hello @playpauseandstop !
Thank you for your report and precise details, we appreciate it :)
I will take a closer look tomorrow, but from the first glance I can say that we didn't pay attention to coroutine-based tests (yet!), since by default hypothesis expects tests that return None:

In [4]: @given(i=st.integers()) 
   ...: async def test(i): 
   ...:     pass 

In [5]: await test()

...

FailedHealthCheck: Tests run under @given should return None, but test returned <coroutine object test at 0x7f479454d4d0> instead.

But it is possible to run via custom executors - https://hypothesis.readthedocs.io/en/latest/details.html#custom-function-execution

And it is definitely something that we can work on soon :)

As for a quick workaround in your example, it should be possible to use an explicit loop and run_until_complete. Something like this:

@schema.parametrize()
def test_petstore(loop, aiohttp_client, case):
    client = loop.run_until_complete(aiohttp_client(create_app()))
    response = loop.run_until_complete(client.request(
        case.method, case.formatted_path, headers=case.headers, json=case.json
    ))
    assert response.status < 500

I will add more details one I'll have them :)

[playpauseandstop]

@Stranger6667

Thanks for quick answer!

Yeah, I've thought about using test functions instead of coroutines, just was not aware that hypothesis not yet supported the coroutine-based tests (at least without extra configuration).

I'll try to dig deeper early tomorrow and will let you know about results. Thanks for the directions

[Stranger6667]

There is a small conflict with the pytest_pycollect_makeitem hook defined in aiohttp pytest plugin.
After a fix in #122 it should work, the test there is similar to the example you provided. I am not completely sure about corner cases, but I will check today / tomorrow + verify compatibility with pytest-asyncio and then if everything will be ok will do a release :)

[Stranger6667]

Hej @playpauseandstop ! The problem should be solved in the master branch. We will issue a new release tomorrow :) I'll update when it will be done

[playpauseandstop]

Thanks a lot @Stranger6667!

I haven't time today to play with your proposed solution from #121 (comment), so looking forward to try new release instead.