Implemented a test runner system for the pytest plugin

Fixes #107.

docs/testing.rst

@@ -9,24 +9,92 @@ plugin.
 Creating asynchronous tests
 ---------------------------
 
-To mark a coroutine function to be run via :func:`anyio.run`, simply add the ``@pytest.mark.anyio``
-decorator::
+Pytest does not natively support running asynchronous test functions, so they have to be marked
+for the AnyIO pytest plugin to pick them up. This can be done in one of two ways:
+
+#. Using the ``pytest.mark.anyio`` marker
+#. Using the ``anyio_backend`` fixture, either directly or via another fixture
+
+The simplest way is thus the following::
 
     import pytest
 
+    # This is the same as using the @pytest.mark.anyio on all test functions in the module
+    pytestmark = pytest.mark.anyio
+
 
-    @pytest.mark.anyio
     async def test_something():
-        pass
+        ...
+
+Marking modules, classes or functions with this marker has the same effect as applying the
+``pytest.mark.usefixtures('anyio_backend')`` on them.
+
+Thus, you can also require the fixture directly in your tests and fixtures::
+
+    import pytest
+
+
+    async def test_something(anyio_backend):
+        ...
+
+Specifying the backends to run on
+---------------------------------
+
+The ``anyio_backend`` fixture determines the backends and their options that tests and fixtures
+are run with. The AnyIO pytest plugin comes with a function scoped fixture with this name which
+runs everything on all supported backends.
+
+If you change the backends/options for the entire project, then put something like this in your top
+level ``conftest.py``::
+
+    @pytest.fixture
+    def anyio_backend():
+        return 'asyncio'
+
+If you want to specify different options for the selected backend, you can do so by passing a tuple
+of (backend name, options dict)::
+
+    @pytest.fixture(params=[
+        pytest.param(('asyncio', {'use_uvloop': True}), id='asyncio+uvloop'),
+        pytest.param(('asyncio', {'use_uvloop': False}), id='asyncio'),
+        pytest.param('curio'),
+        pytest.param(('trio', {'restrict_keyboard_interrupt_to_checkpoints': True}), id='trio')
+    ])
+    def anyio_backend(request):
+        return request.param
+
+If you need to run a single test on a specific backend, you can use ``@pytest.mark.parametrize``
+(remember to add the ``anyio_backend`` parameter to the actual test function, or pytest will
+complain)::
+
+    @pytest.mark.parametrize('anyio_backend', ['asyncio'])
+    async def test_on_asyncio_only(anyio_backend):
+        ...
+
+Because the ``anyio_backend`` fixture can return either a string or a tuple, there are two
+additional function-scoped fixtures (which themselves depend on the ``anyio_backend`` fixture)
+provided for your convenience:
+
+* ``anyio_backend_name``: the name of the backend (e.g. ``asyncio``)
+* ``anyio_backend_options``: the dictionary of option keywords used to run the backend
 
 Asynchronous fixtures
 ---------------------
 
 The plugin also supports coroutine functions as fixtures, for the purpose of setting up and tearing
-down asynchronous services used for tests::
+down asynchronous services used for tests.
+
+There are two ways to get the AnyIO pytest plugin to run your asynchronous fixtures:
+
+#. Use them in AnyIO enabled tests (see the first section)
+#. Use the ``anyio_backend`` fixture (or any other fixture using it) in the fixture itself
+
+The simplest way is using the first option::
 
     import pytest
 
+    pytestmark = pytest.mark.anyio
+
 
     @pytest.fixture
     async def server():
@@ -35,68 +103,51 @@ down asynchronous services used for tests::
         await server.shutdown()
 
 
-    @pytest.mark.anyio
     async def test_server(server):
         result = await server.do_something()
         assert result == 'foo'
 
-Any coroutine fixture that is activated by a test marked with ``@pytest.mark.anyio`` will be run
-with the same backend as the test itself. Both plain coroutine functions and asynchronous generator
-functions are supported in the same manner as pytest itself does with regular functions and
-generator functions.
 
-Specifying the backend to run on
---------------------------------
+For ``autouse=True`` fixtures, you may need to use the other approach::
 
-By default, all tests are run against the default backend (asyncio). The pytest plugin provides a
-command line switch (``--anyio-backends``) for selecting which backend(s) to run your tests
-against. By specifying a special value, ``all``, it will run against all available backends.
+    @pytest.fixture(autouse=True)
+    async def server(anyio_backend):
+        server = await setup_server()
+        yield
+        await server.shutdown()
 
-For example, to run your test suite against the curio and trio backends:
 
-.. code-block:: bash
+    async def test_server():
+        result = await client.do_something_on_the_server()
+        assert result == 'foo'
 
-    pytest --anyio-backends=curio,trio
 
-Behind the scenes, any function that uses the ``@pytest.mark.anyio`` marker gets parametrized by
-the plugin to use the ``anyio_backend`` fixture. One alternative is to do this parametrization on
-your own::
+Using async fixtures with higher scopes
+---------------------------------------
 
-    @pytest.mark.parametrize('anyio_backend', ['asyncio'])
-    async def test_on_asyncio_only(anyio_backend):
-        ...
+For async fixtures with scopes other than ``function``, you will need to define your own
+``anyio_backend`` fixture because the default ``anyio_backend`` fixture is function scoped::
 
-Or you can write a simple fixture by the same name that provides the back-end name::
+    @pytest.fixture(scope='module')
+    def anyio_backend():
+        return 'asyncio'
 
-    @pytest.fixture(params=['asyncio'])
-    def anyio_backend(request):
-        return request.param
 
-If you want to specify different options for the selected backend, you can do so by passing a tuple
-of (backend name, options dict). The latter is passed as keyword arguments to :func:`anyio.run`::
-
-    @pytest.fixture(params=[
-        pytest.param(('asyncio', {'use_uvloop': True}), id='asyncio+uvloop'),
-        pytest.param(('asyncio', {'use_uvloop': False}), id='asyncio'),
-        pytest.param('curio'),
-        pytest.param(('trio', {'restrict_keyboard_interrupt_to_checkpoints': True}), id='trio')
-    ])
-    def anyio_backend(request):
-        return request.param
-
-Because the ``anyio_backend`` fixture can return either a string or a tuple, there are two
-additional fixtures (which themselves depend on the ``anyio_backend`` fixture) provided for your
-convenience:
-
-* ``anyio_backend_name``: the name of the backend (e.g. ``asyncio``)
-* ``anyio_backend_options``: the dictionary of option keywords used to run the backend
+    @pytest.fixture(scope='module')
+    async def server(anyio_backend):
+        server = await setup_server()
+        yield
+        await server.shutdown()
 
-Using AnyIO from regular tests
-------------------------------
+Technical details
+-----------------
 
-In rare cases, you may need to have tests that run against whatever backends you have chosen to
-work with. For this, you can add the ``anyio_backend`` parameter to your test. It will be filled
-in with the name of each of the selected backends in turn::
+The fixtures and tests are run by a "test runner", implemented separately for each backend.
+The test runner keeps an event loop open during the request, making it possible for code in
+fixtures to communicate with the code in the tests (and each other).
 
-    def test_something(anyio_backend):
-        assert anyio_backend in ('asyncio', 'curio', 'trio')
+The test runner is created when the first matching async test or fixture is about to be run, and
+shut down when that same fixture is being torn down or the test has finished running. As such,
+if no async fixtures are used, a separate test runner is created for each test. Conversely, if
+even one async fixture (scoped higher than ``function``) is shared across all tests, only one test
+runner will be created during the test session.

docs/versionhistory.rst

@@ -60,6 +60,11 @@ This library adheres to `Semantic Versioning 2.0 <http://semver.org/>`_.
   async generators and cancels any leftover native tasks
 - Fixed ``Condition.wait()`` not working on asyncio and curio (PR by Matt Westcott)
 - Added the ``anyio.aclose_forcefully()`` to close asynchronous resources as quickly as possible
+- Added ``open_test_runner()`` which is used by the pytest plugin to ensure that all fixtures are
+  run inside the same event loop as the test itself
+- Removed the ``--anyio-backends`` command line option for the pytest plugin. Use the ``-k`` option
+  to do ad-hoc filtering, and the ``anyio_backend`` fixture to control which backends you wish to
+  run the tests by default.
 
 **1.4.0**
 

setup.cfg

@@ -57,7 +57,7 @@ pytest11 =
     anyio = anyio.pytest_plugin
 
 [tool:pytest]
-addopts = -rsx --tb=short -p no:curio --anyio-backends=all
+addopts = -rsx --tb=short -p no:curio
 testpaths = tests
 filterwarnings = always
 

src/anyio/__init__.py

@@ -100,8 +100,10 @@ def claim_worker_thread(backend) -> typing.Generator[Any, None, None]:
         del _local.current_async_module
 
 
-def _get_asynclib():
-    asynclib_name = sniffio.current_async_library()
+def _get_asynclib(asynclib_name: Optional[str] = None):
+    if asynclib_name is None:
+        asynclib_name = sniffio.current_async_library()
+
     modulename = 'anyio._backends._' + asynclib_name
     try:
         return sys.modules[modulename]

src/anyio/_backends/_asyncio.py

@@ -119,6 +119,25 @@ def get_callable_name(func: Callable) -> str:
 # Event loop
 #
 
+def _maybe_set_event_loop_policy(policy: Optional[asyncio.AbstractEventLoopPolicy],
+                                 use_uvloop: bool) -> None:
+    # On CPython, use uvloop when possible if no other policy has been given and if not
+    # explicitly disabled
+    if policy is None and use_uvloop and sys.implementation.name == 'cpython':
+        try:
+            import uvloop
+        except ImportError:
+            pass
+        else:
+            # Test for missing shutdown_default_executor() (uvloop 0.14.0 and earlier)
+            if (not hasattr(asyncio.AbstractEventLoop, 'shutdown_default_executor')
+                    or hasattr(uvloop.loop.Loop, 'shutdown_default_executor')):
+                policy = uvloop.EventLoopPolicy()
+
+    if policy is not None:
+        asyncio.set_event_loop_policy(policy)
+
+
 def run(func: Callable[..., T_Retval], *args, debug: bool = False, use_uvloop: bool = True,
         policy: Optional[asyncio.AbstractEventLoopPolicy] = None) -> T_Retval:
     @wraps(func)
@@ -134,21 +153,7 @@ async def wrapper():
         finally:
             del _task_states[task]
 
-    # On CPython, use uvloop when possible if no other policy has been given and if not explicitly
-    # disabled
-    if policy is None and use_uvloop and sys.implementation.name == 'cpython':
-        try:
-            import uvloop
-        except ImportError:
-            pass
-        else:
-            if (not hasattr(asyncio.AbstractEventLoop, 'shutdown_default_executor')
-                    or hasattr(uvloop.loop.Loop, 'shutdown_default_executor')):
-                policy = uvloop.EventLoopPolicy()
-
-    if policy is not None:
-        asyncio.set_event_loop_policy(policy)
-
+    _maybe_set_event_loop_policy(policy, use_uvloop)
     return native_run(wrapper(), debug=debug)
 
 
@@ -1331,3 +1336,40 @@ async def wait_all_tasks_blocked() -> None:
                 break
         else:
             return
+
+
+class TestRunner(abc.TestRunner):
+    def __init__(self, debug: bool = False, use_uvloop: bool = True,
+                 policy: Optional[asyncio.AbstractEventLoopPolicy] = None):
+        _maybe_set_event_loop_policy(policy, use_uvloop)
+        self._loop = asyncio.new_event_loop()
+        self._loop.set_debug(debug)
+        asyncio.set_event_loop(self._loop)
+
+    def _cancel_all_tasks(self):
+        to_cancel = all_tasks(self._loop)
+        if not to_cancel:
+            return
+
+        for task in to_cancel:
+            task.cancel()
+
+        self._loop.run_until_complete(
+            asyncio.gather(*to_cancel, loop=self._loop, return_exceptions=True))
+
+        for task in to_cancel:
+            if task.cancelled():
+                continue
+            if task.exception() is not None:
+                raise task.exception()
+
+    def close(self) -> None:
+        try:
+            self._cancel_all_tasks()
+            self._loop.run_until_complete(self._loop.shutdown_asyncgens())
+        finally:
+            asyncio.set_event_loop(None)
+            self._loop.close()
+
+    def call(self, func: Callable[..., Awaitable], *args, **kwargs):
+        return self._loop.run_until_complete(func(*args, **kwargs))

src/anyio/_backends/_curio.py

@@ -1041,3 +1041,19 @@ async def wait_all_tasks_blocked() -> None:
                 break
         else:
             return
+
+
+class TestRunner(abc.TestRunner):
+    def __init__(self, **options):
+        self._kernel = curio.Kernel(**options)
+
+    def close(self) -> None:
+        self._kernel.run(shutdown=True)
+
+    def call(self, func: Callable[..., Awaitable], *args, **kwargs):
+        async def call():
+            # This wrapper is needed because curio kernels cannot run() the asend() method of async
+            # generators because it's neither a coroutine object or a coroutine function
+            return await func(*args, **kwargs)
+
+        return self._kernel.run(call)