` testing/acceptance_test.py::test_internal_errors_propagate_to_controller` seems to fail 30% of the time

[trofi]

NixOS distribution frequently rebuilds and reruns tests of packages software. I noticed pytest-xdist very frequently fails test_internal_errors_propagate_to_controller test. Attaching both successful and problematic logs:

• problematic.log
• good.log

Running the build/test in a loop i get about 70% success rate for tests. Could it be a pytest-xdist bug?

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /build/pytest-xdist-2.4.0, configfile: tox.ini, testpaths: testing
plugins: xdist-2.4.0, forked-1.3.0
collected 164 items / 8 deselected / 156 selected

testing/acceptance_test.py .............s..x.......xx................... [ 28%]
......x.................F...................                             [ 57%]
testing/test_dsession.py ........x...x                                   [ 65%]
testing/test_looponfail.py ...........x.ss                               [ 75%]
testing/test_newhooks.py ...                                             [ 76%]
testing/test_plugin.py ..............                                    [ 85%]
testing/test_remote.py x....x......                                      [ 93%]
testing/test_workermanage.py .......x.s                                  [100%]

=================================== FAILURES ===================================
_________________ test_internal_errors_propagate_to_controller _________________

testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_internal_errors_propagate_to_controller0')>

    def test_internal_errors_propagate_to_controller(testdir):
        testdir.makeconftest(
            """
            def pytest_collection_modifyitems():
                raise RuntimeError("Some runtime error")
            """
        )
        testdir.makepyfile("def test(): pass")
        result = testdir.runpytest("-n1")
>       result.stdout.fnmatch_lines(["*RuntimeError: Some runtime error*"])
E       Failed: nomatch: '*RuntimeError: Some runtime error*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0'
E           and: 'rootdir: /build/pytest-of-nixbld/pytest-0/test_internal_errors_propagate_to_controller0'
E           and: 'plugins: xdist-2.4.0, forked-1.3.0'
E           and: 'gw0 I'
E           and: 'gw0 [1]'
E           and: ''
E           and: 'INTERNALERROR> Traceback (most recent call last):'
E           and: 'INTERNALERROR>   File "/nix/store/09sk6r94sgyvb5695y5gz5jl1d97iff4-python3.9-pytest-6.2.5/lib/python3.9/site-packages/_pytest/main.py", line 269, in wrap_session'
E           and: 'INTERNALERROR>     session.exitstatus = doit(config, session) or 0'
E           and: 'INTERNALERROR>   File "/nix/store/09sk6r94sgyvb5695y5gz5jl1d97iff4-python3.9-pytest-6.2.5/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main'
E           and: 'INTERNALERROR>     config.hook.pytest_runtestloop(session=session)'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__'
E           and: 'INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec'
E           and: 'INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_manager.py", line 327, in traced_hookexec'
E           and: 'INTERNALERROR>     return outcome.get_result()'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result'
E           and: 'INTERNALERROR>     raise ex[1].with_traceback(ex[2])'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_result.py", line 33, in from_call'
E           and: 'INTERNALERROR>     result = func()'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_manager.py", line 324, in <lambda>'
E           and: 'INTERNALERROR>     lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall'
E           and: 'INTERNALERROR>     return outcome.get_result()'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result'
E           and: 'INTERNALERROR>     raise ex[1].with_traceback(ex[2])'
E           and: 'INTERNALERROR>   File "/nix/store/6gpm9yb90cflindawcxm7ba0g0l6liaj-python3.9-pluggy-1.0.0/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall'
E           and: 'INTERNALERROR>     res = hook_impl.function(*args)'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/dsession.py", line 112, in pytest_runtestloop'
E           and: 'INTERNALERROR>     self.loop_once()'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/dsession.py", line 135, in loop_once'
E           and: 'INTERNALERROR>     call(**kwargs)'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/dsession.py", line 256, in worker_collectionfinish'
E           and: 'INTERNALERROR>     self.sched.schedule()'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/scheduler/load.py", line 257, in schedule'
E           and: 'INTERNALERROR>     self._send_tests(next(nodes), 1)'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/scheduler/load.py", line 269, in _send_tests'
E           and: 'INTERNALERROR>     node.send_runtest_some(tests_per_node)'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/workermanage.py", line 284, in send_runtest_some'
E           and: 'INTERNALERROR>     self.sendcommand("runtests", indices=indices)'
E           and: 'INTERNALERROR>   File "/nix/store/3gwnbq9193x53g7hk2vzmn3ffjkba1v6-python3.9-pytest-xdist-2.4.0/lib/python3.9/site-packages/xdist/workermanage.py", line 300, in sendcommand'
E           and: 'INTERNALERROR>     self.channel.send((name, kwargs))'
E           and: 'INTERNALERROR>   File "/nix/store/9a3r84xmg0ppjl1zf7q0jas8f4mksgdw-python3.9-execnet-1.9.0/lib/python3.9/site-packages/execnet/gateway_base.py", line 728, in send'
E           and: 'INTERNALERROR>     raise IOError("cannot send to {!r}".format(self))'
E           and: 'INTERNALERROR> OSError: cannot send to <Channel id=1 closed>'
E           and: ''
E           and: '============================ no tests ran in 0.20s ============================='
E       remains unmatched: '*RuntimeError: Some runtime error*'

/build/pytest-xdist-2.4.0/testing/acceptance_test.py:1148: Failed

[nicoddemus]

Hi @trofi,

Thanks for writing. Unfortunately seems something related to execnet, which unfortunately is in maintenance only mode.

[Luflosi]

I just ran into this same problem on NixOS. The test_max_worker_restart_tests_queued test also failed. When building for a second time, all tests passed.