Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parametrize based on another parameter #4050

Open
Sup3rGeo opened this issue Sep 28, 2018 · 6 comments
Open

Parametrize based on another parameter #4050

Sup3rGeo opened this issue Sep 28, 2018 · 6 comments
Labels
topic: parametrize related to @pytest.mark.parametrize type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature

Comments

@Sup3rGeo
Copy link
Member

Sup3rGeo commented Sep 28, 2018

Hi,

This feature is related to a concept I think that pytest is missing. Let's say I have this very simple test:

@pytest.mark.parametrize("param", [1,2,3])
def test(param):
    mysetupcode(param)
    myassertfunc(param)

Now, I want to refactor the setup as a fixture, but still have the same parametrization for both the test and the fixture. In that case, I end up having to create a fixture:

@pytest.fixture(params=[1,2,3])
def param():
    # This is not a real fixture because it does not run any code
    return request.param

@pytest.fixture
def mysetupcode(param):
    # ... setup code
    pass

def test(param, mysetupcode):
    pass

So basically the concept that pytest is missing is this param fixture that is not a real fixture as does not run any code, but is just a parameter broadcast entity that is different from parametrizing at the individual fixture/test level.

Another example where this would be useful, with some API suggestions:

datasetA = [data1_a, data2_a, data3_a]
datasetB = [data1_b]
datasetC = [data1_c, data2_c]


# HERE the concept missing. The following line is equivalent to:
# @pytest.fixture(params=[datasetA, datasetB])
# def dataset(request)
#     return request.param
# Also you can think as a pytest.mark.parametrize not tied to any function
pytest.parameterize('dataset', [datasetA, datasetB])

# Fixtures could use it just as well (as it is just like a fixture)
# Actually becomes another way to parametrize
@pytest.fixture
def fixture1(dataset):
    setup(dataset)
    pass

# tests can use this concept as well, as we would with fixtures
def test_one(dataset)

# Following line is almost as if we are parametrizing the parametrize
# So it will be doubly parametrized: 
# test_data[datasetA-data1_a]
# test_data[datasetA-data2_a]
# test_data[datasetA-data3_a]
# test_data[datasetB-data1_b]
# test_data[datasetC-data1_c]
# test_data[datasetC-data2_c]
@pytest.mark.parametrize('data', pytest.parameter('dataset')])
def test_data(data):
    #do test

This feature was discussed in
TvoroG/pytest-lazy-fixture#25
#349 (comment)

I decided to open a new feature request because I believe it is a different use case/implementation than #349, although they might be related. The difference is that we don't need to worry about this that @nicoddemus mentioned:

Under the current design, pytest needs to generate all items during the collection stage. Your examples generate new items during the session testing stage (during fixture instantiation), which is not currently possible.

Because we don't need to run any code to have all the parameters in place at collection time.

@Sup3rGeo Sup3rGeo added type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature topic: parametrize related to @pytest.mark.parametrize labels Sep 28, 2018
@RonnyPfannschmidt
Copy link
Member

covariant parameterize is long-sought ^^

@nicoddemus
Copy link
Member

Gave a quick and dirty stab at this:

# conftest.py
import attr

import pytest

@attr.s
class Covariant:
    name = attr.ib()
    values = attr.ib()

    def make_fixture(self):
        @pytest.fixture(name=self.name, params=self.values)
        def fixture_func(request):
            return request.param

        return fixture_func


def pytest_configure():
    pytest.covariant_param = Covariant


@pytest.hookimpl(hookwrapper=True)
def pytest_collect_file():
    outcome = yield
    modules = outcome.get_result()
    for module in modules:
        if isinstance(module, pytest.Module):
            print(module)
            for name in dir(module.obj):
                member = getattr(module.obj, name)
                if isinstance(member, Covariant):
                    fixture = member.make_fixture()
                    setattr(module.obj, name, fixture)
                    print(fixture)
            module.session._fixturemanager.parsefactories(module)

Usage:

import pytest

dataset = pytest.covariant_param('dataset', [1, 2, 3])

def test_foobar(dataset):
    pass
λ pytest .tmp\covariant\ -v
======================== test session starts ========================
platform win32 -- Python 3.6.6, pytest-3.10.1.dev78+gb83e9780.d20181119, py-1.6.0, pluggy-0.8.0 -- c:\pytest\.env36\scripts\python.exe
...
collected 3 items

.tmp\covariant\test_foo.py::test_foobar[1] PASSED              [ 33%]
.tmp\covariant\test_foo.py::test_foobar[2] PASSED              [ 66%]
.tmp\covariant\test_foo.py::test_foobar[3] PASSED              [100%]

===================== 3 passed in 0.03 seconds ======================

But the idea is to do some post processing after collection, where we look for Covariant instances and replace them with proper fixtures.

It needs some refining to collect from test classes as well, possibly other features like scope and ids, but I believe the underlying implementation is surprisingly solid because we are using fixtures behind the scenes. It might be missing some functionality as I only tested with test_foo.py above.

This can easily be done in an external plugin too.

@Sup3rGeo
Copy link
Member Author

Hey @nicoddemus that is really interesting!
However, although it tackles the first thing I described (the missing concept of a independent parameter), I believe it still would not have the covariant aspect which is basically the "parametrization of second order" (parameters coming from another parameter - hence the covariance). Or maybe I am getting the terms wrong as well!
Anyway we call it, any ideas for this?

@nicoddemus
Copy link
Member

Sorry about that, wrote that right before going to bed.

believe it still would not have the covariant aspect which is basically the "parametrization of second order"

You mean this last example right?

# Following line is almost as if we are parametrizing the parametrize
# So it will be doubly parametrized: 
# test_data[datasetA-data1_a]
# test_data[datasetA-data2_a]
# test_data[datasetA-data3_a]
# test_data[datasetB-data1_b]
# test_data[datasetC-data1_c]
# test_data[datasetC-data2_c]
@pytest.mark.parametrize('data', pytest.parameter('dataset')])
def test_data(data):
    #do test

Perhaps extending the idea further, I think this might be possible to implement:

import pytest

datasetA = pytest.covariant_param('datasetA', ['data1_a', 'data2_a', 'data3_a'])
datasetB = pytest.covariant_param('datasetB', ['data1_b'])
datasetC = pytest.covariant_param('datasetC', ['data1_c', 'data2_c'])

dataset = pytest.covariant_param('dataset', [datasetA, datasetB, datasetC])

def test_foobar(dataset):
    pass

What do you think?

@Sup3rGeo
Copy link
Member Author

If it results in the set of tests described in the comments above the test function, then I think it should do the job!

Probably we would have to iterate a bit to get the interface 100% correct but it seems to be going in the right direction

@smarie
Copy link
Contributor

smarie commented Apr 5, 2019

Just for the sake of traceability, @Sup3rGeo you requested this feature in pytest-cases and it is now available.

Note: when it is officially fixed in pytest the pytest-cases version will fallback on it, but I will keep the current implementation available for older versions of pytest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: parametrize related to @pytest.mark.parametrize type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature
Projects
None yet
Development

No branches or pull requests

4 participants