indirect=True in parametrize breaks fixture scopes

[pytestbot]

Originally reported by: Ivan Kalinin (BitBucket: PuPSSMaN, GitHub: PuPSSMaN)

---

The following test:

#!python

import pytest

@pytest.fixture(scope='class')
def my_profile(request):
    print 'Setting up profile with param', getattr(request, 'param', 'Nothing')


@pytest.mark.parametrize('my_profile', ['pewpew', 'ololo'], indirect=True)
class TestClassOne:
    def test_foo(self, my_profile):
        pass

    def test_bar(self, my_profile):
        pass

Outputs this:

#!text

(pytest)pupssman@dirigible:~$ py.test test_1.py  -sv
================================================ test session starts =================================================
platform linux2 -- Python 2.7.4 -- py-1.4.23 -- pytest-2.6.1 -- /home/pupssman/venv/pytest/bin/python
plugins: qabs-yadt, allure-adaptor, contextfixture, capturelog, xdist
collected 4 items 

test_1.py::TestClassOne::test_foo[pewpew] Setting up profile with param pewpew
PASSED
test_1.py::TestClassOne::test_foo[ololo] Setting up profile with param ololo
PASSED
test_1.py::TestClassOne::test_bar[pewpew] Setting up profile with param pewpew
PASSED
test_1.py::TestClassOne::test_bar[ololo] Setting up profile with param ololo
PASSED

============================================== 4 passed in 0.01 seconds ==============================================

Looks like it sets the fixture up for each test method of class independently.
However, the declaration of scope='class' should force py.test to set the fixture up only once for each parameter value for the test class. I.e., the output should be something like that:

#!text

(pytest)pupssman@dirigible:~$ py.test test_1.py  -sv
================================================ test session starts =================================================
platform linux2 -- Python 2.7.4 -- py-1.4.23 -- pytest-2.6.1 -- /home/pupssman/venv/pytest/bin/python
plugins: qabs-yadt, allure-adaptor, contextfixture, capturelog, xdist
collected 4 items 

test_1.py::TestClassOne::test_foo[pewpew] Setting up profile with param pewpew
PASSED
test_1.py::TestClassOne::test_bar[pewpew]
PASSED
test_1.py::TestClassOne::test_foo[ololo] Setting up profile with param ololo
PASSED
test_1.py::TestClassOne::test_bar[ololo]
PASSED

============================================== 4 passed in 0.01 seconds ==============================================

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/570

[pytestbot]

Original comment by BitBucket: mavlyutov, GitHub: mavlyutov:

---

any news bout that?

[vlcinsky]

Another case of the same issue.

The intention was to allow db_connect to be set up and tear down only once per session:

import pytest

DBCFG = {"pgsql": "postgresql://scott:tiger@localhost:5432/mydatabaser",
         "mysql": "mysql://scott:tiger@localhost/foo",
         "oracle": "oracle://scott:tiger@127.0.0.1:1521/sidname"
}


@pytest.yield_fixture(scope="session")
def db_connect(request):
    connect_name = request.param
    print("\nopening db {connect_name}".format(connect_name=connect_name))
    assert connect_name in DBCFG
    yield DBCFG[connect_name]
    print("\nclosing db {connect_name}".format(connect_name=connect_name))


@pytest.fixture(scope="session")
def table_name(request):
    return "tabname-by-fixture {request.param}".format(request=request)


scenarios = [
    ("mysql", "myslq-user"),
    ("mysql", "myslq-groups"),
    ("pgsql", "pgsql-user"),
    ("pgsql", "pgsql-groups"),
    ("oracle", "oracle-user"),
    ("oracle", "oracle-groups"),
]
@pytest.mark.parametrize("db_connect,table_name",
                         scenarios,
                         indirect=["db_connect", "table_name"])
def test_it(db_connect, table_name):
    print("Testing: {} + {}".format(db_connect, table_name))

However, when the tests is run:

$ py.test -sv tests/test_indirect.py
py.test========================================= test session starts ==================================
=======
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/.virtualenvs/stack
/bin/python2
cachedir: tests/.cache
rootdir: /home/javl/sandbox/stack/tests, inifile:
collected 6 items

tests/test_indirect.py::test_it[mysql-myslq-user]
opening db mysql
Testing: mysql://scott:tiger@localhost/foo + tabname-by-fixture myslq-user
PASSED
closing db mysql

tests/test_indirect.py::test_it[mysql-myslq-groups]
opening db mysql
Testing: mysql://scott:tiger@localhost/foo + tabname-by-fixture myslq-groups
PASSED
closing db mysql

tests/test_indirect.py::test_it[pgsql-pgsql-user]
opening db pgsql
Testing: postgresql://scott:tiger@localhost:5432/mydatabaser + tabname-by-fixture pgsql-user
PASSED
closing db pgsql

tests/test_indirect.py::test_it[pgsql-pgsql-groups]
opening db pgsql
Testing: postgresql://scott:tiger@localhost:5432/mydatabaser + tabname-by-fixture pgsql-groups
PASSED
closing db pgsql

tests/test_indirect.py::test_it[oracle-oracle-user]
opening db oracle
Testing: oracle://scott:tiger@127.0.0.1:1521/sidname + tabname-by-fixture oracle-user
PASSED
closing db oracle

tests/test_indirect.py::test_it[oracle-oracle-groups]
opening db oracle
Testing: oracle://scott:tiger@127.0.0.1:1521/sidname + tabname-by-fixture oracle-groups
PASSED
closing db oracle


====================================== 6 passed in 0.01 seconds =======================================

the opening <database> and closing <databas> appears per test case.

[maddy-longo]

Is this still a problem?

[rajiv-vats]

I can still see this issue on latest pytest.

[scythargon]

+1

[nicoddemus]

FWIW the original example now works as expected in the latest version:

.tmp/test_parametrize_bug.py::TestClassOne::test_foo[pewpew] Setting up profile with param pewpew
PASSED
.tmp/test_parametrize_bug.py::TestClassOne::test_bar[pewpew] PASSED
.tmp/test_parametrize_bug.py::TestClassOne::test_foo[ololo] Setting up profile with param ololo
PASSED
.tmp/test_parametrize_bug.py::TestClassOne::test_bar[ololo] PASSED```

This is related to #3551: pytest tries to follow some logic to minimize the number of session/class/module scoped fixture it creates between each test, but it can't always guess correctly. I think we should also update the docs to explain that "session" is not a guarantee that a fixture will be executed only once per session, but is a "best effort" when it involves parametrization with multiple session scoped fixtures being used by the same tests. @ceridwen explains this problem well in #3551 (comment).

[nicoddemus]

Having said the above, I believe we still have a bug because I think @vlcinsky is correct that db_connect should be created/destroyed only 3 times, not 6 like currently.