parametrizing metafunc in pytest_generate_tests does not group tests as expected

[dbak-invn]

conftest.py:

@pytest.fixture(scope='session')
def fw_config(request):
    logging.info(f'fw_config called with {request.param}')

    return request.param


def pytest_generate_tests(metafunc):
    if 'fw_config' in metafunc.fixturenames:
        configs_to_test = ['ABC', 'DEF']
        pattern = metafunc.definition.get_closest_marker('config_pattern')

        if pattern is None:
            pattern = '.*'
        else:
            assert len(pattern.args) == 1
            pattern = pattern.args[0]

        regex = re.compile(pattern)
        configs_to_test = filter(regex.match, configs_to_test)

        metafunc.parametrize(argnames='fw_config', argvalues=configs_to_test, ids=configs_to_test, scope='session', indirect=True)

test file:

class TestGrouping:

    @pytest.mark.config_pattern(r'.*ABC.*')
    def test_abc(self, fw_config):
        time.sleep(4)
        return True

    @pytest.mark.config_pattern(r'.*DEF.*')
    def test_def(self, fw_config):
        time.sleep(4)
        return True
    def test_all(self, fw_config):
        time.sleep(4)
        return True

Since fw_config is session-scoped, I am hoping that the tests will run in the following order:

test_abc[ABC]
test_all[ABC]
test_def[DEF]
test_all[DEF]

And that fw_config will be called only twice.

However, instead, they are called in the following order:

test_abc[ABC]
test_def[DEF]
test_all[ABC]
test_all[DEF]

and fw_config is called four times.

I have been trying for almost a month to get this behavior using various methods and have not been able to achieve it.

I am aware that I can just have test_abc skip() if fw_config is 'DEF', but then I have a skipped test in my output. I want to not run the test at all.

fw_config here is a stand-in for an expensive configuration with a name. Some tests need to run on all configs, and some tests need to run on only a subset.

Is there any way to do this? Am I doing something incorrectly?

I am guessing that there are actually three instances of the fixture being created, and pytest is treating them as being totally separated.

[Zac-HD]

Closing this question as inactive. Maybe try StackOverflow?

[RonnyPfannschmidt]

this is possibly a bug in the test reordering