Idea: @pytest.mark.parametrize on @pytest.fixture

[nyanpasu64]

Miniconda Python 3.6.6, pytest 3.8.0 from conda-forge

STEREO_PATHS = ['stereo 1.wav', 'stereo 2.wav']

@pytest.mark.parametrize("path", STEREO_PATHS)
@pytest.mark.parametrize("cfg_factory", [list, dict])   # not actual params
@pytest.fixture(scope="module")
def stereo_cfg(path, cfg_factory): ...

          @pytest.mark.parametrize("path", STEREO_PATHS)
          @pytest.mark.parametrize("cfg_factory", [n163_cfg, unrounded_cfg])
          @pytest.fixture(scope="module")
          def stereo_cfg(path, cfg_factory):
E       fixture 'path' not found

• @pytest.fixture(params) does not allow me to parameterize one fixture with multiple orthogonal parameters (I want all 4 combinations of path and cfg_factory).

• I find the pytest.fixture(params=[...]) to be extremely inelegant and confusing compared to @pytest.mark.parametrize. This is because of the requirement that the function argument be named request, the actual parameter be named request.param (making the parameter name non-self-documenting and confusing at first).

maybe test functions can accept parametric parameters directly, while fixture functions can only accept fixtures as parameters? if so, that's a problem that should be solved.

---

Workaround: Boilerplate fixtures which wrap lists:

STEREO_PATHS = ['stereo 1.wav', 'stereo 2.wav']

@pytest.fixture(scope="module", params=STEREO_PATHS)
def path(request):
    path = request.param
    return Path('test_waves', path)

@pytest.fixture(scope="module", params=[list, dict])
def cfg_factory(request):
    return request.param

@pytest.fixture(scope="module")
def stereo_cfg(path, cfg_factory): ...

But this adds many boilerplate lines of code. I thought pytest was a boilerplate-free test system.

# packages in environment at /home/jimbo1qaz/miniconda3/envs/wavetable:
#
# Name                    Version                   Build  Channel
atomicwrites              1.2.1                    py36_0  
attrs                     18.2.0           py36h28b3542_0  
blas                      1.0                         mkl  
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24                py36_1    conda-forge
click                     6.7                      py36_0  
dataclasses               0.6                        py_0    conda-forge
intel-openmp              2018.0.3                      0  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.4            py36h4414c95_1  
mkl_random                1.0.1            py36h4414c95_1  
more-itertools            4.3.0                    py36_0  
ncurses                   6.1                  hf484d3e_0  
numpy                     1.15.1           py36h1d66e8a_0  
numpy-base                1.15.1           py36h81de0dd_0  
openssl                   1.0.2p               h470a237_0    conda-forge
pip                       10.0.1                   py36_0  
pluggy                    0.7.1            py36h28b3542_0  
py                        1.6.0                    py36_0  
pytest                    3.8.0                    py36_0    conda-forge
python                    3.6.6                hc3d631a_0  
readline                  7.0                  h7b6447c_5  
ruamel.yaml               0.15.65          py36h470a237_0    conda-forge
scipy                     1.1.0            py36hfa4b5c9_1  
setuptools                40.2.0                   py36_0  
six                       1.11.0                   py36_1  
sqlite                    3.24.0               h84994c4_0  
tk                        8.6.8                hbc83047_0  
waveform-analysis         0.1                       <pip>
wavetable                 0.0.0                     <pip>
wheel                     0.31.1                   py36_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               ha838bed_2  

[smarie]

I implemented a workaround @pytest_fixture_plus in pytest-cases 1.1.0. It behaves exactly like @pytest.fixture except that in addition it supports @pytest.mark.parametrize:

from pytest_cases import pytest_fixture_plus

STEREO_PATHS = ['stereo 1.wav', 'stereo 2.wav']

@pytest_fixture_plus(scope="module")
@pytest.mark.parametrize("path", STEREO_PATHS)
@pytest.mark.parametrize("cfg_factory", [list, dict])
def stereo_cfg(path, cfg_factory):
    ...

It has been tested against pytest 2.x and 3.x. It will remain available until this issue is directly fixed in pytest, if it is one day.

It is also compliant with @cases_data as shown here.

Note: it works by creating sub-fixtures under the hood, just like the ones you showed in your workaround. So thanks for the idea! :)

[smarie]

Update: the execution order was a bit random, I fixed it in 1.2.0, see smarie/python-pytest-cases#22 .

[Sup3rGeo]

I would love to see this implemented, as I see no reason why we cannot use the same way to parameterize fixtures and test functions.

@smarie does your improved fixture accept also parameterizing pair of arguments? like:

@pytest.mark.parametrize("arg1, arg2", [
    ("a", "b"),
    ("c", "d"),
]
@pytest_fixture_plus
def my_fixture(arg1, arg2):
    pass

[smarie]

yes it does support multi-parameters. Let me know (please post an issue) if it does not work for you.
Note: the decoration order matters, @pytest_fixture_plus should be on top so that it is able to "see" the marks. So the correct way is:

@pytest_fixture_plus
@pytest.mark.parametrize("arg1, arg2", [
    ("a", "b"),
    ("c", "d"),
]
def my_fixture(arg1, arg2):
    pass

[Sup3rGeo]

I created an issue in: smarie/python-pytest-cases#26

[RonnyPfannschmidt]

This directly ties into a critical topic - how should marks pass from fixture to test,

there is no simple solution

[smarie]

@Sup3rGeo this was a regression, the issue is now fixed in pytest-cases 1.3.2.

@RonnyPfannschmidt my two cents on this topic for what's worth: if the problem is passing marks, then it might be worth considering to make the parametrization decorator a first-class citizen and not a mark anymore. And/or to provide a parametrization decorator that "feels the same" for users, but is implemented differently whether you parametrize test functions or fixtures. Implementation can then rely on marks if needed, but that's an implementation decision, not user's business.

[Sup3rGeo]

@smarie I would agree 100%, and not only for parametrization, but perhaps also the other concepts that are treated/implemented as marks and should be first-class citizens (skip, skipif, xfail, etc). I discussed a little bit about it in #4081

[RonnyPfannschmidt]

@smarie another issue name-spacing the new parameters - parameters are fixtures with direct or indirect values, so they naturally spill to the test in some way

[smarie]

If I understand your comment correctly, you are saying that what is proposed in pytest-cases and above is not an ideal longterm solution. I fully agree, it is indeed a workaround.

I try to make the generated fixture names explicit and unique (see source):

gen_name = fixture_func.__name__ + "__" + 'X'.join(m.param_names)

But they are still generated fixtures, so that's not entirely transparent. At least it allows us to work while this makes its way in the core pytest design (I understand that some major refactoring would be needed, so there is some interest in having something usable in the meantime).

[smarie]

Following your comment and smarie/python-pytest-cases#28 I changed the design in pytest_fixture_plus: instead of re-generating intermediate fixtures, I collect all parametrization marks and simply create the fixture's param accordingly.

The only problem is that I have to generate the ids for the user (when they are not explicitly provided) and I would like to do it exactly like pytest. When no explicit ids (callable or iterable) is provided, do you simply use str(value) to get the id ? Thanks for confirming !

[RonnyPfannschmidt]

idmaker and its related functions in src/_pytest/python.py sort it out in detail

[smarie]

thanks !

[smarie]

Hi @RonnyPfannschmidt , just to let you know that after making this feature, @Sup3rGeo also made what I think is a great suggestion: providing users with an API to create "parameter" fixtures, that is, parametrized fixtures whose value is the parameter value.

This could naturally be extended to the fact that all test parameters created by @pytest.mark.parametrize are fixtures. That way, everything would be a fixture - parameters are just "simple" fixtures, and pytest.mark.parametrize is just creating function-scope parameter fixtures.

This enables very neat test architectures where you can safely reuse parameters within test functions and fixtures without having to refactor your whole code or do complex request manipulation so that the fixture knows about the parameters.

In my opinion it would be a great evolution for pytest, definitely simplifying a lot of issues I personally encountered during test refactoring. See an example implementation here if you wish to play with it: https://smarie.github.io/python-pytest-cases/#param_fixtures
(this implementation does not replace the mark.parametrize with fixture though, it is just providing the parameter fixture feature)

[RonnyPfannschmidt]

@smarie im currently not on active duty and i suspect it will be quite a long time before i take a look at something like this (as quite frankly getting the details right on that kind of system is just a horrendous mess of details)

[smarie]

No worries, I understand that you have put a lot of efforts to bring pytest to the state it is today and that rest is well deserved before a major revisit of the internal architecture - if it happens.

In the meantime that's absolutely not a problem, since pytest-cases has been validated against several versions of pytest and is in a stable production state. We can certainly live with this workaround for months if not more :)

[Conchylicultor]

Any updates on this ? Mixing the @pytest.mark.parametrize with @pytest.fixture would feel very natural and I was surprise pytest did not support this already.

@pytest.fixture
@pytest.mark.parametrize('v', [0, 1])
def my_fixture(v):
  return v

[tanev]

@Conchylicultor, just mentioning something that works very nicely, but might not be what you need:

@pytest.fixture(name="v")
def my_fixture(request):
    # do some stuff with request.param here
    yield request.param
    # then do some stuff with request.param post-tests

@pytest.mark.parametrize(
    "v", [0, 1], indirect=True,
)
def test_my(v):
    print(f"{v} from test")

...might be of help in such cases, it has its own set of limitations, though.

[smarie]

@Conchylicultor there is a workaround until this is baked in pytest: pip install pytest-cases and then :

from pytest_cases import fixture

@fixture
@pytest.mark.parametrize('v', [0, 1])
def my_fixture(v):
  return v

[Conchylicultor]

@smarie Yes, I'm aware of pytest-case, but I would prefer not having to install extra modules for something as simple. Also, pytest-case isn't available at Google currently.

@tanev, this feature request is about parametrize the fixture, not the test. You solution only works mostly when a single test is parametrized. Also the indirection makes it too "magic".

[RonnyPfannschmidt]

In near future its impossible to do in core

What looks simply has a massive amount of edge cases we currently can't sanely manage, so for now, marks won't work for fixtures

[smarie]

Looks like making pytest-cases available at Google would be easier than refactoring pytest internals ;) Thanks for the precision @RonnyPfannschmidt . I confirm that the only way I found to do this in pytest-cases is to create a single param with the correct cross-product of values, ids and marks. See here in the code of @fixture

[nicoddemus]

What looks simply has a massive amount of edge cases we currently can't sanely manage, so for now, marks won't work for fixtures

Just to add to @RonnyPfannschmidt's response: the main issue about adding this support is not parametrization per-se, but adding mark support for fixtures, which needs some careful consideration of the repercussions of that.

The "core" implementation of parametrization in pytest is pytest_generate_tests, which is a generic way to implement parametrization, for any test, based on anything you can imagine.

This is how the pytest core implements parametrization support out of the box for general use:

1. For tests, a pytest_generate_tests implementation will check if the test is marked with pytest.mark.parametrize and generate extra tests accordingly.

2. For fixtures, another pytest_generate_tests implementation will generate new tests based on the fixtures used by the function.

[Sup3rGeo]

It seems to me that it might be that the mark thing is bringing more complexity than it is needed.

The parametrization of fixtures from @smarie seems to be already what me and other uses typically need and expect from such feature - at the same time, I don't see how it is or should be related to the problem related to passing marks from fixtures to tests.

Generally speaking, I see a completely different usage for general @pytest.mark and parametrization (either test functions or fixtures). I tend to think that using marks for that matter might not be the right solution and parametrization must be a "first class citizen" (even maybe @pytest.parametrize, without marks) instead of being some side-benefit of marks, so we effectively decouple the two problems.

[nicoddemus]

(even maybe @pytest.parametrize, without marks) instead of being some side-benefit of marks, so we effectively decouple the two problems.

You are correct in that it could be treated as separate concerns, I just wanted to address explicitly the title of the issue ("why isn't @pytest.mark.parametrize work on fixures already?").

To elaborate:

@pytest.fixture is just a decorator that essentially appends metadata to the decorated function, which is picked up during collection.

A new @pytest.parametrize decorator could do the same thing: append metadata to the function it decorates, be it either a fixture or a test function, which is then processed by a custom pytest_generate_tests hook.

I'm pretty sure the idea above can be implemented outside the core even, if someone wants to play with it.

---

Having said that, I'm not a big fan of the idea in principle: creating another way to do the same thing is confusing for users and a maintenance problem for the project (both in code to maintain and the possibility of bugs that will arise from that implementation).

Conceptually, I think supporting marks in fixtures makes sense (and in turn make fixture parametrization that much nicer), but we need to think carefully what it means.

For parametrize, the answer is "obvious":

@pytest.mark.parametrize("x", [1, 2])
def fix(x):
    return x * 10

def test(fix):
    assert fix in (100, 200)

But not so much in the general sense:

@pytest.mark.X(hello="world")
def fix():
    return 1

def test(fix):
    assert fix == 1

What happens with "X"? Are marks applied to fixtures are automatically propagated to tests which use it? How about the mark arguments, {"hello": "world"}, how do I access them?

Internally, there's another problem, marks are applied to Nodes (test functions and classes), and fixtures are not part of the collection tree.

All those details need to figure out first.

[Sup3rGeo]

Thanks for the detailed answer @nicoddemus

You are correct in that it could be treated as separate concerns, I just wanted to address explicitly the title of the issue ("why isn't @pytest.mark.parametrize work on fixures already?").

Fair point!

If I understood correctly from your words:

• On one hand, we have the fixture parametrization without marks (@pytest.fixture, with just some metadata for pytest_generate_tests), but with potential confusion for users/maintainability
• On the other hand, we keep fixture parametrization with marks (@pytest.mark.fixture), but have to deal with the bigger, unrelated, more complicated mark propagation problem

Personally I don't think it is even worth thinking about the second point - As in the end there is the user not having a very helpful feature because of some mark behavior he probably would not care at all (e.g. I never had the need for marking a fixture myself).

Although of course it could be that just the implementation of the first point could prove too complicated itself but I guess it is always the case with new features.

[The-Compiler]

@Sup3rGeo I'm confused about you referring to @pytest.fixture and @pytest.mark.fixture - did you mean @pytest.parametrize and @pytest.mark.parametrize there?

[smarie]

Not directly related, but connected: I have created a plugin to design higher-level marks for filtering. It is named pytest-pilot. This is maybe another example of the huge gap that @Sup3rGeo pointed out, between parametrization marks (that are closely related to the core pytest engine itself) and other marks (that, as demonstrated by this extension, are quite "far" from the core of pytest and can be manipulated easily without hacking into the pytest engine. No new debate here, just food for thoughts :) .