All tests fail on one node and run never completes, around 20% of the time

[dgolombek]

We have an annoyingly high rate (~20%) of a test suite run using pytest-xdist having all tests on one worker fail, then the collection never completing:

$ poetry run pytest --durations=5 -v -W ignore::DeprecationWarning tests/integration/mycor -n auto
============================= test session starts ==============================
platform linux -- Python 3.11.11, pytest-8.3.4, pluggy-1.5.0 -- /home/runner/work/dave-energy/dave-energy/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/dave-energy/dave-energy
configfile: pytest.ini
plugins: xdist-3.6.1, asyncio-0.26.0, pytest_httpx-0.29.0, Faker-35.0.0, respx-0.20.2, ddtrace-2.21.7, timeout-2.3.1, typeguard-2.13.3, anyio-3.7.1, time-machine-2.16.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
created: 4/4 workers
4 workers [2215 items]

scheduling tests via LoadScheduling

tests/integration/foo.py::test_bar
[gw2] [  0%] ERROR tests/integration/foo.py::test_bar
tests/integration/foo.py::test_baz
[gw2] [  0%] ERROR tests/integration/foo.py::test_baz
[gw3] [  1%] PASSED tests/integration/quux.py::test_quux
tests/integration/quux.py::test_quux

The tests run until 85-92% completion, then hang forever.

Oddly (and almost certainly relatedly), I only see logs for gw2 and gw3 in this run -- I never see gw0 or gw1. In the runs that do pass, I see all 4 workers. I've added psutil, but that didn't help. I tried dropping to pytest-xdist 3.4.0, to see if an older version would help, but no luck. Using --dist loadfile did not help either. I'm using execnet 2.1.1.

Please let me know if there's other information I can provide, or flags I can add to provide more details -- unfortunately I can't reproduce locally, only in CI -- which is Linux vs Mac, and X86 vs ARM.

[RonnyPfannschmidt]

Getting th3 logdiles when PYTEST_DEBUG is set in the env would be the next step

[dgolombek]

Thanks, I should've done that before, I'd gotten scared by the hundreds of megabytes of logs I usually see with it.

That exposed (unsurprisingly) that there was an error during a session-scoped fixture that I wasn't seeing before.

It looks like fixing that solved the rest of our problems.

It looks like the failure in gw1 session fixture creation caused problems with gw0 and gw2 -- in the logs, for those two workers, all I see is:

2025-05-15T19:50:30.0360776Z       pytest_testnodeready [hook]
2025-05-15T19:50:30.0360925Z           node: <WorkerController gw2>
...
2025-05-15T19:50:44.2679464Z           node: <WorkerController gw2>
...
2025-05-15T19:50:44.6371754Z           finish pytest_fixture_setup --> gw2 [hook]

whereas for gw3, which had all tests pass, I see the equivalent of above, and then

2025-05-15T19:50:44.6395168Z           finish pytest_fixture_setup --> ***localhost:5432/pytest-gw3.denergy.61dd2cdc468c40d686a895c68b8a50ee [hook]
...
And a lot more references

Not sure if that's worth trying to handle or not. Thanks for the suggestion, and the great framework.

[RonnyPfannschmidt]

Thanks for the followup

Am i right to presume your issue is resolved?

[dgolombek]

Correct, our issue is resolved, thanks! The framework could probably be more robust in detecting failures in session-level fixtures, but the problem was entirely on our side.