libregrtest: add option to run every test case in separate process

[serhiy-storchaka]

Feature or enhancement

Proposal:

libregrtest can run every test module in a separate process, and can even run several test modules in parallel. It has options to simply output all test modules or all test cases.

I propose to add an option to run every test case in a separate process. Test cases from the same module should still run sequentially, because they can use the same resource (for example create a server at fixed port).

This will slow down the tests several times, but it is not for regular running in the CI, it is for manual running once every months or for manual running selected test module while writing new tests or debugging test flakiness. It can help to solve the following issues:

1. Order dependency. Some tests are only passed because they run in specific order, and the order of tests in a module is the same, only the order of test modules is randomized. If run the test in a separate process, it is always the first, and cannot depend on others.
2. Environment changes and leaks. Some tests change the environment (e.g. leak temporary files or do not wait for terminating a thread or a subprocess), and it is not noticed because subsequent tests have cleaned after it. If run the test in a separate process, it is always the last, and nobody cleans after it.

cc @vstinner @gpshead

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[serhiy-storchaka]

For now I can do this by using the following command:

PYTHON=./python; $PYTHON -m test -uall --list-tests 2>/dev/null | while read -r test
do
    $PYTHON -m test -uall --list-cases $test 2>/dev/null | xargs -n1 $PYTHON -m test -vuall $test -m
done

But it is slow (because it does not parallel different test modules), the output is difficult to analyze, and it cannot be run on Windows.

[vstinner]

See also closed PR gh-99637 by @gpshead.