pytest xdist appears to throttle running parallel tests towards end of test run

[giantbrite]

I've noticed a bizarre pattern while using pytest and xdist to run out selenium webdriver tests in parallel. We are running on a Linux box on Jenkins. At first I noticed this in a large batch of 1200+ tests on 75 subprocesses like this:

py,test -n 75 -v --reruns=0

For the first 5 minutes or so, we had great parallelization as expected; all 75 workers were active. As soon as a test finished another test was dispatched to an available worker. However, towards the end of the run the parallel tests appeared to slowly recede back to 1 active worker; then it jumped back up in a burst of 70+ workers going in parallel, the last batch of tests, until the tests were finished.

There is nothing special going on in our setup.cfg:

[pytest]
python_files=*.py

I have been examining this behavior over the past week and trying different combinations. We are using pytest 2.7.3, xdist 1.13.1, py 1.4.30, execnet 1.3.0, pytest-bdd-2.15.0. Thinking it could be the fault of pytest-rerunfailures plugin, I removed that and the same behavior occurred.

What's puzzling about this is that it happened with every combination of test runs I tried. Finally I tried running just 1 python class of tests (3 selenium test cases: test1, test2, test3) with -n 10. Here is what happened:

0:00 test1 was launched
0:30 test1 finished with PASS
0:31 test2 was launched
0:31 test3 was launched
1:00 test2 finished
1:00 test3 finished

So do you see the pattern I have observed? It is the same whether it is 3 tests, 80 tests, or 1200 tests.

When # of tests to run > # of subprocesses, pytest-xdist feeds the next test to run to the process as soon as it is available to run it.

When # of tests to run <= # of subprocesses, pytest-dist appears to wait for all currently running tests to finish, then it launches the remaining set of tests on the available workers to run it.

In the case of my 1200+ tests, this delay was quite noticeable. It appeared to run 1125 tests quickly in parallel on 75 subprocesses; when there were 75 or less tests to run, it waited almost 2 minutes for the currently running tests to finish; when the last test finished it ran all the remaining tests on 75 workers. As you can see, that time it waited and did not run tests on workers as soon as they finished were troubling. I did check our test framework, even went to the trouble of commenting out our tearDown method to make sure any calls there weren't dragging things out.

Has anyone else reported this? Is there some mysterious switch or setting I don't know about?

[RonnyPfannschmidt]

This is a known issue that originates from the need to have next item in order to ensure setupstate cacheing working

[hpk42]

are you running with a lot of higher than function-level fixtures? (what ronny refers to is that the test run protocol for a single test requires to know which test comes next in order to determine which fixtures to teardown). there is currently no way to switch/change the behaviour from the command line IIRC.

[sri85]

I am facing the same issue that @giantbrite mentioned .Is there any work around for it?

[RonnyPfannschmidt]

No, a larger refactoring in xdist is needed before this can be tackled

[RonnyPfannschmidt]

this one is basically superseeded by pytest-dev/pytest-xdist#40