Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations #3542

Open
kamichal opened this issue Jun 6, 2018 · 6 comments

Comments

@kamichal
Copy link

@kamichal kamichal commented Jun 6, 2018

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it's required for larger scopes to work properly. There is only an advice to do so, but...

@pytestbot

This comment has been minimized.

Copy link

@pytestbot pytestbot commented Jun 6, 2018

GitMate.io thinks possibly related issues are #634 (metafunc.parametrize overwrites scope), #519 (fixture scope is ignored when using metafunc.parametrize()), #570 (indirect=True in parametrize breaks fixture scopes), #244 (parametrize fails when values are unhashable), and #1111 (pytest.mark.parametrize fails with lambdas).

@pytestbot pytestbot added the type: bug label Jun 6, 2018
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Jun 6, 2018

that looks like a bug

note that you can use fun workaround

since you already know your parameters,

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}

@pytest.fixture(scope="session", param=KNOWN_BRANCHES.keys())
def repo_preparation(request):
    """ If it would have a "session" scope then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[request.param] += 1
    return KNOWN_BRANCHES[request.param]

might do (untested)

@kamichal

This comment has been minimized.

Copy link
Author

@kamichal kamichal commented Jun 6, 2018

Thank you Ronny. I know the "workaround" very well.
The point is that I would like to take the tested_branch parameters from parser.addoption.
So, contents of that list is variable in my case, that's why I took pytest_generate_tests hook.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Jun 6, 2018

@kamichal i see - in that case - there is a deeper underlying issue that i wont be investigating in near future - i have no idea if any of the other core devs has other plans

@winklerrr

This comment has been minimized.

Copy link

@winklerrr winklerrr commented Jun 19, 2018

I came across the same bug, I think...
When using metafunc.parametrize(scope=session) in combination with parametrized tests, all of my session scoped fixtures teared down after each test.

This behavior can be demonstrated very vividly with --setup-show:

SETUP    S session_fixture1 # <-- This fixture is generated with pytest_generate_tests
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test1()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2 
TEARDOWN S session_fixture1 # <-- Session scoped fixture gets tear downed
SETUP    S session_fixture1 
SETUP    S session_fixture2 (fixtures used: session_fixture1)
    SETUP    M module_fixture (fixtures used: session_fixture2)
        SETUP    F function_fixture (fixtures used: module_fixture)
            test_file.py::TestClass::my_test2()
        TEARDOWN F function_fixture
    TEARDOWN M module_fixture
TEARDOWN S session_fixture2
TEARDOWN S session_fixture1

Workaround

@kamichal For me, the workaround with indirect=True also works in combination with parameters received from metafunc.config.getoption.

Just define a real fixture with the same name (in the example below my_fixture) and use the request fixture to return the indirect parametrized value:

def pytest_addoption(parser):
    parser.addoption("--my_parser_option")
    
def pytest_generate_tests(metafunc):
    if "my_fixture" in metafunc.fixturenames:
        metafunc.parametrize("my_fixture", metafunc.config.getoption("my_parser_option"), scope="session", indirect=True)

@pytest.fixture(scope="session")
def my_fixture(request):
    return request.param

def test_fixture(my_fixture):
    assert my_fixture == "world"

Then you can call it with pytest --my_parser_option hello, which will result in "hello" != "world"
I hope this helps someone.

Regards
Winklerrr

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Jun 19, 2018

thanks for providing the workaround,
this also means we can have a better way of understanding what is wrong

as its limited to freestanding parameterization

barnabasJ pushed a commit to barnabasJ/testinfra that referenced this issue Jul 20, 2018
There seems to be a bug in pytest where metafunc.parametrize breaks
fixture scope. By creating a fixture and parameterizing it indirectly
scope can be maintained
barnabasJ pushed a commit to barnabasJ/testinfra that referenced this issue Aug 27, 2018
There seems to be a bug in pytest where metafunc.parametrize breaks
fixture scope. By creating a fixture and parameterizing it indirectly
scope can be maintained
philpep added a commit to philpep/testinfra that referenced this issue Sep 12, 2018
Implement workaround for pytest issue [#3542](pytest-dev/pytest#3542)

There seems to be a bug in pytest where metafunc.parametrize breaks
fixture scope. By creating a fixture and parameterizing it indirectly
scope can be maintained
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.