trio_asyncio.open_loop() in a fixture?

[SillyFreak]

I hoped to be able to do the following for my test cases requiring trio-asyncio:

@pytest_trio.trio_fixture
async def trio_aio_loop():
    async with trio_asyncio.open_loop() as loop:
        yield loop

@pytest.mark.trio
async def test_aio_funcs(trio_aio_loop, autojump_clock):
    @trio_asyncio.aio_as_trio
    async def func():
        await asyncio.sleep(0.1)
        return 1

    assert await func() == 1

However the fixture blows up:

test setup failed
@pytest_trio.trio_fixture
    async def trio_aio_loop():
>       async with trio_asyncio.open_loop() as loop:

tests/test_loop.py:261: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../env/lib/python3.7/site-packages/async_generator/_util.py:34: in __aenter__
    return await self._agen.asend(None)
.../env/lib/python3.7/site-packages/async_generator/_impl.py:366: in step
    return await ANextIter(self._it, start_fn, *args)
.../env/lib/python3.7/site-packages/async_generator/_impl.py:197: in __next__
    return self._invoke(first_fn, *first_args)
.../env/lib/python3.7/site-packages/async_generator/_impl.py:209: in _invoke
    result = fn(*args)
.../env/lib/python3.7/site-packages/trio_asyncio/async_.py:108: in open_loop
    async with trio.open_nursery() as nursery:
.../env/lib/python3.7/site-packages/trio/_core/_run.py:378: in __aenter__
    self._scope = CancelScope._create(deadline=inf, shield=False)
.../env/lib/python3.7/site-packages/trio/_core/_run.py:130: in _create
    task = _core.current_task()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def current_task():
        """Return the :class:`Task` object representing the current task.
    
        Returns:
          Task: the :class:`Task` that called :func:`current_task`.
    
        """
    
        try:
            return GLOBAL_RUN_CONTEXT.task
        except AttributeError:
>           raise RuntimeError("must be called from async context") from None
E           RuntimeError: must be called from async context

.../env/lib/python3.7/site-packages/trio/_core/_run.py:1534: RuntimeError

I assume that has to do something with the description of fixture handling here (I admit not having read it in full). It seems rather intricate, so I have no idea how hard this would be to fix. I think it would be a valuable addition, though.

[touilleMan]

Hi,

You should have a look here:

https://github.com/Scille/parsec-cloud/blob/e1e25366d07425312712c01950440476a7322608/tests/conftest.py#L133-L140

Btw trio-asyncio is tricky because it creates relationship between two event loops, which can end up in hard to track deadlocks. For instance you should avoid using the nursery fitxure along with an asyncio_loop fixture (because there are both disconnected from trio point of view, but once the asyncio_loop is torndown, coroutines belonging to the nursery fixture and using asyncio code won't work anymore without explanation...)

[SillyFreak]

thanks for the hint! could you very shortly explain why it has to be @pytest.fixture instead of @pytest_trio.trio_fixture, even though trio_asyncio.open_loop() is a trio-style async context manager? The shielded cancel scope is separate from that, for the issue you pointed out above, right?

[touilleMan]

@SillyFreak @pytest_trio.trio_fixture is just a small wrapper around @pytest.fixture to flag the test using it as trio (in my tests I use the @pytest.mark.trio decorator to add this trio flag)

[njsmith]

If you have trio-mode enabled, then @pytest.fixture and @pytest_trio.trio_fixture are equivalent when applied to an async def; it doesn't matter which one you use. You can also use @pytest_trio.trio_fixture on synchronous fixtures, or when trio mode isn't enabled, and in that case it makes a difference.

[lordi]

I think we currently can't use trio_asyncio.open_loop at all with pytest_trio, because it would require that the test is ran with trio_asyncio.run instead of trio.run. Is that assumption correct?

[njsmith]

@lordi No, trio_asyncio.run is just a confusing shorthand for doing trio.run and then trio_asyncio.open_loop. So you don't need it; you can just do open_loop yourself. I'm not sure why we have trio_asyncio.run, honestly; trio_asyncio's core is pretty solid but the API still needs some fine tuning.

[lordi]

Ok, thanks, that's good to know.

[jmehnle]

Did anything change since 2019? I'm using this fixture, and it is working just fine:

@pytest.fixture
async def asyncio_loop() -> AsyncIterator[asyncio.events.AbstractEventLoop]:
    '''
    This fixture must be used by any test that tests code involving `asyncio` (as opposed to `trio`)
    code and hence `trio_asyncio` wrappers. Ensure the fixture is active for the entire duration of
    the test. A fixture may depend on this on behalf of dependent tests as long as the fixture is a
    generator (i.e., uses `yield` instead of `return`).

    Unfortunately, relevant tests must depend on this fixture explicitly for these reasons:
    - It cannot be a session fixture, as `pytest-trio` expects async fixtures to be function-scoped.
      <https://pytest-trio.readthedocs.io/en/stable/reference.html#trio-fixtures>
    - It cannot be an autouse fixture, as this would apply it to non-async tests and fail.
      <https://github.com/python-trio/pytest-trio/issues/123>
    '''
    # When a ^C happens, trio sends a `Cancelled` exception to each running task. We must protect
    # this one to avoid deadlock if it is cancelled before another coroutine that uses trio-asyncio.
    with trio.CancelScope(shield=True):  # pyright: ignore[reportCallIssue]
        async with trio_asyncio.open_loop() as loop:
            yield loop

BTW, I ran across #105. Would it make sense to offer another value, say, trio_asyncio, for the trio_run ini option?

[oremanj]

I'm not sure why the original poster ran into trouble, but I agree a fixture like you describe should work fine. I think the shield shouldn't be necessary anymore after some recent improvements to trio-asyncio (there's now a bunch of careful shielding inside open_loop) but I don't think it will do any harm either.

Would it make sense to offer another value, say, trio_asyncio, for the trio_run ini option?

Yeah that seems like a pretty easy win!

[jmehnle]

Please consider #146.