Tests are skipped with pytest-xdist

[azmeuk]

pytest-dependency does not seem to behave well when using pytest-xdist. Some tests are skipped when they should not be. Let's see how to quickly reproduce the bug:

test_foobar.py

import pytest

@pytest.mark.dependency()
def test_a():
    pass

@pytest.mark.dependency(depends=["test_a"])
def test_b():
    pass

console

virtualenv env
./env/bin/pip install pytest
./env/bin/pip install pytest-xdist
./env/bin/pip install pytest-dependency
./env/bin/pytest -n auto test_foobar.py

output

pytest test_foobar.py -n auto
============================= test session starts ==============================
platform linux -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/eloi/dev/dependency-test, inifile:
plugins: xdist-1.18.1, dependency-0.2
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2]
scheduling tests via LoadScheduling
s.
===================== 1 passed, 1 skipped in 0.31 seconds ======================

[RKrahl]

I never used pytest-xdist by now. I need to investigate. Might take some time.

[RKrahl]

I still didn't got a round to investigate this in detail. But after having a short glance on the pytest-xdist documentation, I assume the problem is the following: pytest-xdist features to run the tests in parallel. It seem to do this in parallel processes, each running its own pytest instance. As a result, the dependent test is skipped because its dependency has not been run (in the same parallel instance of pytest). I'm afraid, this is fundamentally incompatible with pytest-dependency, because such a parallelization can only work if the tests are independent from each other. But the very purpose of pytest-dependency is to handle the case that they are not.

If this assumption turn out to be true, the only thing pytest-dependency can do about this problem, is to disable the parallelization in pytest-xdist. I would still need to figure out how to do this.

[RKrahl]

I finally got around to take a closer look. My first assumption from the last comment turns out to be right, the problem is the parallelization in pytest-xdist. This can only work if the test are independent from each other.

There is not much that pytest-dependency can do about this. I will refrain from trying to disable the parallelization in pytest-xdist, as I think plugins should not interfere with each other. The only thing I can do is to document this incompatibility.

[RKrahl]

I documented the incompatibility with the parallelization in pytest-xdist and closed this issue. However, this is not the full story: pytest-xdist 1.19 added distributing mode loadscope and 1.21 added loadfile. Both distributing modes are compatible with pytest-dependency under certain conditions. But these distributing modes are marked experimental in pytest-xdist. I will wait for them to become stable before documenting this option.

[Kinzal15]

@RKrahl Any workaround for above issue?

[CMCDragonkai]

I've tried using --dist=loadfile but then xdist complains and says:

ERROR: MISSING test execution (tx) nodes: please specify --tx

And with --dist=no it doesn't work. The dependencies are being skipped.

[pkoch]

@RKrahl Can we perhaps piggyback off of --dist loadgroup? If we make every connected test belong to their own group, that seems to compatibilize the interaction between the two packages, no?