CI(pytest): Run tests marked need_solo_run separately

[echoix]

One probable explanation of some flaky pytest failures is that a small number of tests use more resources since they expect they would be the only module running. Or, for example, testing with multiple number of processes, while already multiple pytest runners are executing tests.

This ended up stalling the other tests, and we regularly encountered failures to different tests files.

The solution here is identifying which tests have this behaviour, and should be run separately from the tests run in parallel, with a single runner. Inspired by pytest’s docs on markers, the needs_solo_run marker name was chosen and applied to two tests (10 items).
The slow marker from pytest’s docs is kept as I intend to mark some tests that are way slower than expected with it.

With these two steps running with parallel (pytest-xdist) and without, we end up with a somewhat constant time execution, no timeouts, but having to do the test collection phase twice (currently about 10 more seconds). It also allows to reduce the timeout to 30 seconds without any problem, as in any case, NO TESTS should take near 30 seconds to setup and execute, especially if they are unit tests.

[echoix]

I've been reanalyzing the problem here.

[echoix]

Edit: Since the real fix to deadlocks/forking of multiprocessing+subprocess is somewhat substantial, I'm loosening back the timeout requirements. This still improves the situation, but some parts of the test fixtures need to be reworked to be able to use the "spawn" multiprocessing startup mode, that will be default in 3.14. Spawn also requires the code to work correctly with pickling, which currently fails with our code base. Spawn startup method for multiprocessing most probably seems like the real fix for some of the issues. Fork (still the default on linux) isn't available for Windows, and problematic on macOS (and not default since an emergency change for 3.8).

There is one usage in a fixture, "run_in_subprocess", (that isn't subprocesss, but multiprocessing.Process), that makes multiple tests fail when using the spawn startup.

So some more changes are needed to make these pytest work for Windows and macOS, which will be required at some point, and for linux, for python 3.14 support.

The original intent of the run_in_subprocess fixture can be replaced once #3694 is merged, which requires #3002 for python 3.8+ support offered by Jammy.

[echoix]

But this is still an improvement, as two problematic functions that use more processes aren't stalling the tests running in parallel. It should be merged when possible.

[wenzeslaus]

This is great. Thank you for the effort figuring this out!