New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conflicts with pytest-asyncio (sample with redis) #8

Closed
vlcinsky opened this Issue Jan 31, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@vlcinsky
Copy link

vlcinsky commented Jan 31, 2018

Having following test file, things behave differently with different pytest plugins installed:

import aioredis
import pytest
from aiohttp import web


def init():
    """Create web.Application instance
    """
    app = web.Application()
    app.router.add_get('/', hello)
    app["redis_url"] = "redis://localhost:6379"
    app.on_startup.append(create_redis_pool)
    app.on_shutdown.insert(0, close_redis_pool)
    return app


async def hello(request):
    """url Handler for hello"""
    return web.Response(text='Hello, world')


async def create_redis_pool(app):
    """Create Redis connection pool"""
    redis_url = app["redis_url"]
    pool = await aioredis.create_redis_pool(redis_url, loop=app.loop)
    app["redis_pool"] = pool


async def close_redis_pool(app):
    """Close Redis connection pool"""
    pool = app["redis_pool"]
    pool.close()
    await pool.wait_closed()


# @pytest.fixture
# def loop(event_loop):
#     """Ensure usable event loop for everyone.
#
#     If you comment this fixture out, default pytest-aiohttp one is used
#     and things start failing (when redis pool is in place).
#     """
#     return event_loop


@pytest.fixture
def app(loop):
    """Application instance.
    """
    return init()


@pytest.fixture
async def cli(test_client, app, loop):
    """HTTP client for our application.
    """
    return await test_client(app)


async def test_hello(cli, loop):
    resp = await cli.get("/")
    assert resp.status == 200
    text = await resp.text()
    assert "Hello, world" in text

only pytest-aiohttp installed: all is fine

When only pytest-aiohttp is installed, the test runs well as it is.

============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.6.3, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /home/javl/devel/tis/dpo-tis-pns, inifile:
collected 1 item

tests/http/test_redisloop.py .                                                                                                                                                                          [100%]

========================================================================================== 1 passed in 0.06 seconds ===========================================================================================

[Process exited 0]

with pytest-asyncio, it fails

When also pytest-asyncio is installed, the test fails with:

self = <ConnectionsPool [db:0, size:[1:10], free:0]>

    async def wait_closed(self):
        """Wait until pool gets closed."""
        await self._close_state.wait()
        assert self._close_waiter is not None
>       await asyncio.shield(self._close_waiter, loop=self._loop)
E       RuntimeError: Task <Task pending coro=<pytest_fixture_setup.<locals>.wrapper.<locals>.setup() running at /home/javl/devel/tis/dpo-tis-pns/.tox/py36/lib/python3.6/site-packages/pytest_asyncio/plugin.p
y:117> cb=[_run_until_complete_cb() at /home/javl/.pyenv/versions/3.6.3/lib/python3.6/asyncio/base_events.py:176]> got Future <Future pending> attached to a different loop

.tox/py36/lib/python3.6/site-packages/aioredis/pool.py:177: RuntimeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[33] > /home/javl/devel/tis/dpo-tis-pns/.tox/py36/lib/python3.6/site-packages/aioredis/pool.py(177)wait_closed()
-> await asyncio.shield(self._close_waiter, loop=self._loop)
   6 frames hidden (try 'help hidden_frames')
(Pdb++)

fix by masking loop fixture

To fix this problem (still using pytest-asyncio plugin), one has to uncomment the loop fixture.

@vlcinsky

This comment has been minimized.

Copy link

vlcinsky commented Jan 31, 2018

@asvetlov

This comment has been minimized.

Copy link
Member

asvetlov commented Feb 1, 2018

I see no pool fixture in your code

@vlcinsky

This comment has been minimized.

Copy link

vlcinsky commented Feb 1, 2018

@asvetlov thanks for heads up, I meant loop (not pool). I fixed the description.

@scottbelden

This comment has been minimized.

Copy link

scottbelden commented Mar 20, 2018

This has also caused me some headaches as well when trying to write tests and having both pytest-aiohttp and pytest-asyncio installed. So far I've been the most successful with redefining the loop fixture to use the event_loop fixture as mentioned here.

@gjcarneiro

This comment has been minimized.

Copy link

gjcarneiro commented May 17, 2018

me too;

At least it seems like I can reasonably get away with using only pytest-aiohttp and just skip pytest-asyncio entirely. So not that big a problem. Just time wasting and confusing for developers.

@scottbelden

This comment has been minimized.

Copy link

scottbelden commented May 17, 2018

While it's nice that many people can get away with only using the pytest-aiohttp plugin, I don't think this is always possible. I can double check if necessary, but I remember a case where I couldn't just use the aiohttp plugin. I think it was a test that didn't use any aiohttp fixtures and was just a simple async test and got some exception.

Sorry for the worst description ever, but if it would be useful I can try to reproduce what I was seeing.

@ChristopheLallement

This comment has been minimized.

Copy link

ChristopheLallement commented Jul 17, 2018

I think I've encouter same issue with a shorted code using ThreadPoolExecutor without redis.
It seems that I can't use loop fixture from pytest-aiohttp while pytest-asyncio > 0.5.0 is installed.

Following source using event_loop fixture from pytest-asyncio is working fine with pytest-asyncio == 0.8.0 installed :

import asyncio
import pytest
from concurrent.futures import ThreadPoolExecutor

def count():
    return  0


@pytest.fixture
async def my_fixture(event_loop):
    executor = ThreadPoolExecutor()    
    await event_loop.run_in_executor(executor, count)


async def test_f(my_fixture):
    pass

Following source with loop fixture from pytest-aiohttp :

  • is working fine if pytest-asyncio is not installed
  • also working fine if pytest-asyncio == 0.5.0 is installed
  • doesn't work if pytest-asyncio == 0.8.0 is installed
import asyncio
import pytest
from concurrent.futures import ThreadPoolExecutor


def count():
    return  0


@pytest.fixture
async def my_fixture(loop):
    executor = ThreadPoolExecutor()    
    await loop.run_in_executor(executor, count)


async def test_f(my_fixture):
    pass
@asvetlov

This comment has been minimized.

Copy link
Member

asvetlov commented Jul 17, 2018

pytest-asyncio and pytest-aiohttp libraries are not compatible.
That's sad but we should live with it.

@asvetlov asvetlov closed this Jul 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment