WIP use multiprocessing as backend (NEVER MERGE)

[nicoddemus]

This is just a proof of concept, far from finished. I can run simple tests, and even pytest's own test suite runs almost completely (10 or so failures and it hangs at the end, highlighting that this obviously is a WIP).

Opening this just to start the dialog about creating a pytest-xdist2 or so, which uses multiprocessing to run local workers and other libraries such as mitogen for remote execution.

cc @RonnyPfannschmidt

[nicoddemus]

This is required because now output in worker processes actually goes to stdout, so print() and -s works as expected in this setup.

[RonnyPfannschmidt]

this is a good pointer to what needs abstraction to get multiple back-ends supported

[isaulv]

I ran some simple tests with this branch and the -v option doesn't work.

[isaulv]

I know at this stage this is a simple change to use multiprocessing as a backend (And remove execnet language), but in my simple tests I found the following issues:
If a subprocess crashes, it hangs the master process.
Not a true queue in distributing tests. a long process blocking should be the last to report complete.

But I am excited for this change and would love to test it as changes are made to make it better :)

[RonnyPfannschmidt]

@Dude-x this is only a poc to test feasibility, the real implementation will come as a new python package due to massive breaking changes

[isaulv]

@RonnyPfannschmidt good to know it will be a new package. And it will probably a slow evolution. I am still excited though.

[nicoddemus]

@Dude-x this is only a poc to test feasibility, the real implementation will come as a new python package due to massive breaking changes

Exactly, but thanks @Dude-x for taking the time to report those problems anyway. 👍

[ionelmc]

I'm currently smashing this: http://www.nooooooooooooooo.com/

Please don't switch to multiprocessing. Why would you even consider switching?!

I mean, look at this https://bugs.python.org/issue38227

[RonnyPfannschmidt]

Execnet is unmaintained and potentially broken

[nicoddemus]

Closing as stale/served its purpose.

[ionelmc]

Is there some summary/writeup of the current/future xdist situation?

[RonnyPfannschmidt]

@ionelmc unfortunately no

[nicoddemus]

No write up, but what we learned from this branch:

1. Seems definitely possible.
2. We would need to break all hooks, as they deal with execnet objects directly.
3. Probably better to reduce scope too: drop all the remote execution and fork support.

Because of those reasons we thought it would be better to have a separate project.

I meant to start that new project last year, which I suppose will take a good time to do it, but I've been really short on time with everything else to take this on.

So from my part this is "on hold" for the time being, I'm afraid.

[RonnyPfannschmidt]

My project about enabling a general api in here is also on hold

[ugomancz]

Hi, I know it's been quite a while since this topic was on, but I was wondering whether it is still on the table for pytest-xdist2 or something similar to be developed. The execnet backend is causing issues for me at work and this seemed like the perfect solution.

[RonnyPfannschmidt]

it currently in limbo as the people able to implementit re preoccupied