CI job "py27-pytestlatest-xdist" fails

[youtux]

The job is failing lately, and this can be reproduced (without using xdist) using this particular schedule:

$ tests/feature/test_no_sctrict_gherkin.py::test_background_no_strict_gherkin  tests/feature/test_wrong.py::test_verbose_output

It seems that the setup of test_background_no_strict_gherkin leaks into test_verbose_output. I think this is a pytest issue and I am trying to create a smaller file to reproduce the issue.

[sliwinski-milosz]

Good spot!

The problem here is with pytest.config which keeps using config loaded from testdir during test_background_no_strict_gherkin.

While running test_verbose_output:

(Pdb) pytest.config.inifile
local('/private/var/folders/f3/5s577s4x3ln6fb9g8t4kxbn40000gn/T/pytest-of-milosz.sliwinski/pytest-222/test_background_no_strict_gherkin0/tox.ini')

So solution would be to do the setup for test_background_no_strict_gherkin but then also teardown and bring back the config from before test.

But anyway as we rely on pytest.config which is global variable it is not thread safe.

Maybe we could add something like this:

@pytest.fixture(autouse=True)
def config_reset():
    # setup
    # save default config
    yield
    # teardown
    # load default config

[youtux]

It turns out to be a bug in pytest: pytest-dev/pytest#4495 (comment)