Use pytest-xdist

[Gallaecio]

In summary:

• Add pytest-xdist to tests/requirements*.txt and remove versions from pytest* packages.
• In the tox configuration, add -n auto for all environments running pytest.
• Because Windows does not seem to support pytest-xdist cleanly, create a new tox environment specifically for Windows that does not use pytest-xdist (no -n auto).

[codecov]

Codecov Report

Merging #3707 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3707   +/-   ##
=======================================
  Coverage   85.42%   85.42%           
=======================================
  Files         169      169           
  Lines        9635     9635           
  Branches     1433     1433           
=======================================
  Hits         8231     8231           
  Misses       1156     1156           
  Partials      248      248

[kmike]

Hey! I'm surprised tests pass, I was thinking they won't work out of box 👍

What do you think about adding pytest-xdist to tox, documenting how to use it, but not having -n auto by default, or at least allowing to override it? In some cases using all CPUs may be not desirable, as it may use too much RAM, and make tests slower, not faster; also, with pytest-xdist sometimes it is harder to stop tests cleanly. It could also make things slower if you're just running a few tests, selected by -k, as setup of these processes may take longer than just running everything in a single process.

[Gallaecio]

Then, what about simply installing pytest-xdist and documenting its usage?

[kmike]

I think in this example tests are executed for all environments; there is a different way to parallelize it (https://tox.readthedocs.io/en/latest/example/basic.html#parallel-mode). I'm not sure what is better, to run individual environments with several processes, or to run environments in parallel in this case. What do you think about having e.g. tox -e py36 -- scrapy tests -n auto as an example?

[Gallaecio]

You can actually use both in combination, as it turns out, but I guess it’s better to cover them separately, and let developers do what works best for them.

[kmike]

@Gallaecio Thanks, the PR looks great!

[kmike]

Thanks @Gallaecio!