Flaky, xdist, and fixture exceptions can result in test suites running indefinitely

[dshuga]

This happens if and only if all of the following are true:

• using xdist
• using flaky
• An exception occurs in a fixture
• There are remaining tests left to be assigned to processes, and every process has died due to the exceptions occurring in fixtures.

Sample test script:

import pytest
import flaky


@pytest.fixture
def fixture():
    raise AssertionError


@pytest.mark.usefixtures('fixture')
@flaky.flaky(max_runs=2, min_passes=1)
class TestCase:

    @pytest.mark.foobar
    def test_foo(self):
        pass

    def test_bar(self,):
        pass

    def test_baz(self):
        pass

Sample output:

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n1 test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, flaky-3.3.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_foo

(stays this way indefinitely)

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n3 test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, flaky-3.3.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw1] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw2] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
[gw1] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
[gw2] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3] / gw1 [3] / gw2 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_baz 
test_example.py::TestCase::test_bar 
test_example.py::TestCase::test_foo 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 536, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 555, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 593, in slave_slavefinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_example.py::TestCase::()::test_bar', <SlaveController gw1>)
INTERNALERROR> assert not 'test_example.py::TestCase::()::test_bar'

============================================================== no tests ran in 1.39 seconds ==============================================================

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n1 -p no:flaky test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_foo 
[gw0] ERROR test_example.py::TestCase::test_foo 
test_example.py::TestCase::test_bar 
[gw0] ERROR test_example.py::TestCase::test_bar 
test_example.py::TestCase::test_baz 
[gw0] ERROR test_example.py::TestCase::test_baz 

========================================================================= ERRORS =========================================================================
__________________________________________________________ ERROR at setup of TestCase.test_foo ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
__________________________________________________________ ERROR at setup of TestCase.test_bar ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
__________________________________________________________ ERROR at setup of TestCase.test_baz ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
================================================================ 3 error in 1.49 seconds =================================================================

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ 

Probably the same root cause as #114

[jakeczyz]

Also seeing this. I'm running:
platform linux -- Python 3.6.0, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 metadata: {'Python': '3.6.0', 'Platform': 'Linux-4.4.41-k8s-x86_64-with', 'Packages': {'pytest': '3.0.7', 'py': '1.4.33', 'pluggy': '0.4.0'}, 'Plugins': {'xdist': '1.15.0', 'tap': '2.0', 'metadata': '1.3.0', 'html': '1.14.2', 'flaky': '3.3.0'}}