Using Fixtures as Parameters

[jgersti]

Discussed in #11412

^{Originally posted by jgersti September 7, 2023}
In #11284 @RonnyPfannschmidt mentioned that he would like to incorporate pytest-lazy-fixture¹ and building blocks/hooks for pytest-cases² like behaviour into the pytest core.
I have experimented the last month with a pytest-cases² replacement since its author has not been active for some time.
I would like to share some my observations and conclusion to start a discussion about what the scope of the feature should be and what steps need to be taken to implement it.

I will start by giving a brief overview what these plugins do and how they work, followed by my opinion what is in scope for addition to the core of pytest. In case of pytest-cases this description is heavily simplified.

Plugin Overview

pytest-lazy-fixture¹

The plugin provides a single feature: it introduces a LazyFixture object to reference a fixture inside pytest.mark.parametrize and the params argument of pytest.fixture.

Because the referenced fixture maybe parametrized itself or have a parametrized dependency, a pytest_generate_tests hook is used after the core test generation to discover transitive parameters by inspecting each Metafunc.callspecs funcargs and params entries for LazyFixture objects and recursively descending until no further parameters are found.
This is done by recalculating a new fixture closure with the referenced fixture included and calling FixtureManager.pytest_generate_tests with a new (deep) copied Metafunc object with the new fixture closure and finally replacing the callspecs in the old Metafunc are replaced by the newly calculated ones when ascending.
Because the added fixture names in the new fixture closure cannot be passed to the Metafunc object without influencing all calls, they are dropped at this point and in turn are not considered when reordering the tests and higher scoped fixture maybe initialised late and/or multiple times. Also since the additional parametrizations are applied last instead of in order of the dependencies the parameter id order is wrong.

The LazyFixture objects are resolved by using the pytest_run_setup, pytest_fixture_setup and pytest_run_call hooks.
In pytest_runtest_steup item._request._fillfixtures is replaced with a wrapper that inspects item.callspec.params and item.funcargs and resolves found LazyFixtures before calling the original _fillfixture method.
During iteration item.callspec.params is reordered to account for dependency order and scopes.
In pytest_fixture_setup request.param is inspected and if it is a LazyFixture resolved, resp. in pytest_runtest_call each item.funcargs entry is inspected and resolved.

pytest-cases²

The plugin provides an (opiniated³) unified alternative to conventional parametrization. It features a two new decorators for parametrization, parametrize and parametrize_with_cases, a replacement decorator for pytest.fixture and lazily evaluated functions as parameters and fixture reference in form of FixtureRef (parametrize can auto detect fixtures and wrap them into a FixtureRef).
The plugin also provides some more features but to keep the description short I will concentrate on the mentioned featured minus lazy functions I will also not delve into how cases are discovered and just assume we are given a list of functions as cases.

The basic idea is offload parametrization to the new parametrize decorator and use it for test functions and fixtures (as well as cases) to have a unified UX.
This is achieved by wrapping the function that is decorated to manipulate the signature, creating an intermediate fixture that is parametrized and creating a parameter fixture for each parameter (if the parameter is a FixtureRef this parameter fixture is obviously skipped and the referenced fixture is used).
The intermediate fixture depends on all parameter fixtures but during test collection and execution fixtures that are created using the new fixture decorator are selectively disabled.
Indirect parametrization is not allowed if any new feature is used.
The new fixture decorator does not support the params argument but detects parametrization marks and create the parametrization similar to the above mechanism.
It also wraps the decorated function to inject the mechanism for skipping.
Case parametrization decorator takes a list of function (or a class) and applies the fixture decorator to it and forwards FixtureRefs for these to parametrize.

To achieve the proper parametrized test and proper skipping of all the parameters that are not currently active, the plugin replaces FixtureManager.getfixtureclosure and wraps Metafunc.parametrize to inject facades into FuncFixtureInfo.names_closure and Metafunc.callspec.

Note: This is a extremely simplified description that hopefully conveys the major points.

Feature Scope

While a unified parametrization UX would be nice to have it is most definitely out of scope because it would break most existing code bases and would involve quite a bit of black magic behind the curtains.

What I think is in scope is a reimplementation of pytest-lazy-fixture, though I would prefer a name like FixtureRef/FixtureReference because it better conveys the intended usage/meaning, and the proper calculation of the dependency graph tied to the Callspec2 objects instead of the Metafunc object.
I think that test reordering might also need to be touched.

But with a feature that allows fixtures in parametrization and proper dependency calculation writing a plugin the behaves similar to pytest-cases and offers a unified parametrization UX is relative simple. As already stated i have an experimental (internal) implementation for an replacement uses a heavily modified version of pytest-lazy-fixtures under the hood that mostly works but does some sketchy stuff to inject dependencies.

Proposed Changes

Following is a loose and incomplete list of changes/tasks i would propose to tackle this.

• Investigate and implement calculation of the complete and exact⁴ dependency graph instead of the current fixture closure. Then names_closure/fixturenames is the iteration in topological sort order, if this order does not exist the graph has an cycle and is invalid. This would also address function-scoped fixture run before session-scoped fixture #5303, params on Fixture are not always detected during test generation #11350 and maybe Wrong session scoped fixtures parametrization #2844.
  I am aware that this is computationally expensive. But I think it is either this or recursive algorithms further down the line.
• Do not share the FuncFixtureInfo object between all calls to a test and attach it to the callspecs instead of MetaFunc. This is similar to what is included in Support usefixtures with parametrize #11298.
• Extend Metafunc.parametrize to recalculate the dependency graph. For now this would be only to prune the graph in case of direct parametrization.
• Implement LazyFixture/FixtureRef. This can be a simple dataclass with the name of the fixture and an optional field id for parameter id generation.
• Extend/reimplement Metafunc.parametrize to recalculate the dependency graph by adding branches and iterate along it to discover all parametrizations.
  Ideally this is done none-recursive.
• Check if the dependency graphs can be used elsewhere. Reordering? Fixture Setup?
• Check how ordering in deeply parametrized dependencies of higher scopes is impacted.

I would like to reiterate that this post is intended as a starting point for a discussion and not as an definite 'this needs to happen' roadmap.
I would appreciate feedback, comments and any help in making this happen.

Footnotes

1. https://github.com/TvoroG/pytest-lazy-fixture ↩ ↩²

2. https://github.com/smarie/python-pytest-cases ↩ ↩² ↩³

3. my words, not the authors ↩

4. i.e. resolved to a single FixtureDef instead all of them.

↩

[glatterf42]

Would love to see pytest-lazy-fixture become a part of core pytest since the existing outside solutions are not compatible with pytest 8.x (and not maintained) or missing functionality, it seems.

[nicoddemus]

Previous discussion on the topic: #3244.

Now that pytest-lazy-fixture is no longer being maintained (it seems), I definitely agree that this feature should be integrated into the core.

My hunch is that because it can be implemented directly in the internals, the implementation can be simpler than pytest-lazy-fixture itself.

[jgersti]

Unfortunately it is not as simple. Both pytest-cases and pytest-lazy-fixture have bugs with transitive dependencies, because pytest-cases explicitly and pytest-lazy-fixture implicitly assume that the dependencies form a tree. In reality the dependencies from a DAG (directed acyclic graph) and if there is a diamond pattern in the transitive dependencies both plugins can break by either not loading a required fixture, loading too many fixtures, and/or using inappropriate parametrization.

As I tried to convey in the post above and the attached discussion, I think the first step should be fixing/refactoring the dependency calculation and internal depenency representation in pytest itself, so so the dependency graph is explicitly known and it can be manipulated . There is a experimental branch where I implemented an algorithm to build the graph, but i realized quite fast that the current priority of autouse fixtures is difficult to replicate. I did take some notes which tests failed with some guesses as to why, but i am currently unable to find these.

This change more or less equates to ripping out most/all of the fixture dependency and replacing them.

[youtux]

It'd be great also if the lazy fixtures could accept callables, and evaluate them when possible.

We do that in pytest-factoryboy: https://github.com/pytest-dev/pytest-factoryboy/blob/ea91d7e7540340f6d758419eca52f3791310139e/pytest_factoryboy/fixture.py#L513-L516