Ad-hoc parametrization of tests in pytest_collection_modifyitems

[Anton3]

We've found parametrized fixtures extremely useful and powerful, especially in conjunction with the built-in minimal caching: we can vary the behavior of a whole tree of fixtures depending on test parametrization.

In userver, we use daemon_scoped_mark, a session-scoped parametrized fixture, for service options, then there are fixtures for generating various configs, for starting up the service, checking its health, etc.

This way we have, essentially, a separate scope for fixtures, orthogonal to the normal scopes hierarchy. When the param for daemon_scoped_mark changes between tests, the service is restarted together with all relevant fixtures.

For user's comfort, the whole thing is controlled by @pytest.mark.uservice_oneshot. There is a default daemon, and users can specify options for a custom one per-test or per-module.

This is our whole implementation:

@pytest.fixture(scope='session')
def daemon_scoped_mark(request) -> Optional[Dict[str, Any]]:
    """
    Depend on this fixture directly or transitively to make your fixture a per-daemon fixture.

    For tests marked with `@pytest.mark.uservice_oneshot(...)`, the service will be restarted,
    and all the per-daemon fixtures will be recreated.

    This fixture returns kwargs passed to the `uservice_oneshot` mark (which may be an empty dict).
    For normal tests, this fixture returns `None`.
    """
    return getattr(request, 'param', None)


def pytest_collection_modifyitems(session: pytest.Session, config: pytest.Config, items: List[pytest.Item]):
    for item in items:
        oneshot_marker = item.get_closest_marker('uservice_oneshot')
        if oneshot_marker and isinstance(item, pytest.Function):
            func_item = typing.cast(pytest.Function, item)
            param = oneshot_marker.kwargs
            scope = 'session' if param.get('shared', False) else 'function'
            # The fact that daemon_scoped_mark works through fixture parametrization is an implementation detail.
            # However, since we blatantly patch callspec at such a late stage, test name, nodeid and keywords
            # will not be updated anyway.
            if not hasattr(func_item, 'callspec'):
                func_item.callspec = _pytest.python.CallSpec2()
            func_item.callspec.params[daemon_scoped_mark.__name__] = param
            func_item.callspec.indices[daemon_scoped_mark.__name__] = 0
            # pylint: disable=protected-access
            func_item.callspec._arg2scope[daemon_scoped_mark.__name__] = _pytest.python.Scope(scope)

As you can see, we currently have to use pytest's internals.

The initial implementation used pytest_generate_tests, but there are two requirements that make it unusable:

1. The mark can be used in a manual parametrize:

   @pytest.mark.parametrize('foo, bar',
   [
       pytest.param(
           True,
           True,
           marks=pytest.mark.uservice_oneshot(config_hooks=[custom_hook]),
       ),
       (True, True),
       (False, True),
   ],
   def my_test(service_daemon, ...):
       # ...

   We want pytest to first expand the manual parametrize, then additionally parametrize only the one specialized function node that is marked.

2. We want to hide the param from test output. Parametrization is an internal mechanism for essentially creating a custom fixture scope. Not a public feature that should be visible to the end user.

3. (Bonus) There is a guarantee that we don't spawn new tests this way, which is the intent.

So, what do we want from pytest? A small piece of public API that wraps what we need, so we can instead write:

def pytest_collection_modifyitems(session: pytest.Session, config: pytest.Config, items: List[pytest.Item]):
    for item in items:
        oneshot_marker = item.get_closest_marker('uservice_oneshot')
        if oneshot_marker and isinstance(item, pytest.Function):
            func_item = typing.cast(pytest.Function, item)
            param = oneshot_marker.kwargs
            scope = 'session' if param.get('shared', False) else 'function'
            func_item.set_indirect_param(daemon_scoped_mark.__name__, param, scope=scope, hidden=True)

[Anton3]

I've come to the conclusion that the correct way forward is to implement the missing features in Metafunc.parametrize, not to add a new completely separate API for parametrization. See the linked issues above.