Create a way to always run tests in subprocesses within regrtest

[elibendersky]

BPO	18906
Nosy	@ncoghlan, @vstinner, @ezio-melotti, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-25.10:23:03.043>
created_at = <Date 2013-09-02.17:17:16.997>
labels = ['type-feature', 'tests']
title = 'Create a way to always run tests in subprocesses within regrtest'
updated_at = <Date 2019-06-25.10:23:03.039>
user = 'https://bugs.python.org/elibendersky'

bugs.python.org fields:

activity = <Date 2019-06-25.10:23:03.039>
actor = 'vstinner'
assignee = 'eli.bendersky'
closed = True
closed_date = <Date 2019-06-25.10:23:03.043>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2013-09-02.17:17:16.997>
creator = 'eli.bendersky'
dependencies = []
files = []
hgrepos = []
issue_num = 18906
keywords = []
message_count = 10.0
messages = ['196793', '196810', '196811', '196812', '196814', '196833', '196844', '221991', '251963', '346501']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'ezio.melotti', 'Arfrever', 'eli.bendersky', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18906'
versions = ['Python 3.5']

[elibendersky]

There were numerous discussions and issues in the past about the cross-test dependencies that sometimes exist because some tests need to muck with import caches (both on the Python and C level). Some examples:

http://mail.python.org/pipermail/python-dev/2013-January/123409.html
http://mail.python.org/pipermail/python-dev/2013-August/127766.html

Issue bpo-15651 is an example of a hack that has to stay in the code to allow tests to pass. Other things are also difficult - such as simulating situations where some optional modules don't exist (like pyexpat).

----

I suggest to add a capability to regrtest that will allow certain tests to request always being run inside a subprocess. Most of the infrastructure is already in place, since this is what regrtest uses to run with -jN today.

An alternative would be to add some test.support utility to help with this, but then test execution would not be uniform between -j1 and -jN and also some things would be difficult to track, such as -R executions (currently all -R runs of a single test have to be within a single process for the refcount tracking to work).

[elibendersky]

A question that comes up is how should a module signal to regrtest that it needs to be run in a subprocess?

The most natural approach is to have a special attribute set in the module's global dict (for example: __REGRTEST_SUBPROCESS__ = True); however, there's a slight problem with this approach - regrtest has to import the module to see this attribute, and the module may do some work in its top-level code (commonly, imports) that already needs to be done within a subprocess.

One solution is to use a different approach, such as a separate file alongside the test file (i.e. test_foo.regrtest_subprocess alongside test_foo.py). This is quite ugly but "safe".

A different solution is to require modules that want to be executed in a subprocess to not do any top-level work; modules can already define a custom test_main function that regrtest discovers - we can require for this feature to work that modules do all their work through this function, rather than in the global scope.

[ncoghlan]

An easier hack is likely just a new "always run in subprocess" container
with submodule names in regrtest.py. It's not elegant, but it will work.

[ncoghlan]

Although the "well, don't do that then" alternative also sounds reasonable,
and better localises the information about how the test should run.

[elibendersky]

An easier hack is likely just a new "always run in subprocess" container
with submodule names in regrtest.py. It's not elegant, but it will work.

True, that's also an option. I had it in mind in the beginning, but it's
too hacky for my tastes :-) Not doing state-changing stuff in the global
scope is a good practice anyway, so the restriction should not be too horrible.

Do we have some sort of conventions of outside-discoverable module
attributes (like the __REGRTEST_SUBPROCESS__ proposed above)? I.e. in terms
of naming, type, expected values?

[serhiy-storchaka]

The most natural approach is to have a special attribute set in the module's global dict (for example: __REGRTEST_SUBPROCESS__ = True); however, there's a slight problem with this approach - regrtest has to import the module to see this attribute, and the module may do some work in its top-level code (commonly, imports) that already needs to be done within a subprocess.

The main regrtest process can run auxilary child process which imports all test modules and says main process which of them have __REGRTEST_SUBPROCESS__=True.

It will be even better if the main process runs child process for testing all tests so when any test crashes it is possible to report this and respawn child process to continue testing other tests.

[elibendersky]

On Tue, Sep 3, 2013 at 2:32 AM, Serhiy Storchaka <report@bugs.python.org>wrote:

Serhiy Storchaka added the comment:

> The most natural approach is to have a special attribute set in the
module's global dict (for example: __REGRTEST_SUBPROCESS__ = True);
however, there's a slight problem with this approach - regrtest has to
import the module to see this attribute, and the module may do some work in
its top-level code (commonly, imports) that already needs to be done within
a subprocess.

The main regrtest process can run auxilary child process which imports all
test modules and says main process which of them have
__REGRTEST_SUBPROCESS__=True.

It will be even better if the main process runs child process for testing
all tests so when any test crashes it is possible to report this and
respawn child process to continue testing other tests.

Well, if we go *that* way, my initial proposal would be to just always run
every test in a subprocess. Kind of what happens today with -jN, just also
for -j1. Since most people, and I assume bots, run -jN anyway, they already
see each test executed in a subprocess. Some folks didn't feel good about
it because the stress testing "all in one process" provides is apparently
desired.

Your proposal complicates the flow significantly, IMHO. I'd just run each
test in its own subprocess and be done with it.

[BreamoreBoy]

I believe that bpo-9517 may be relevant if we decide to take the subprocess route.

[vstinner]

I created the issue bpo-25285: "regrtest: run tests in subprocesses with -j1 on buildbots".

[vstinner]

Since 2013, the Python test suite evolved a lot. First, we fixed hundreds of bugs where tests leaked various kinds of resources: processes, threads, file descriptors, memory, etc. Second, python3 -m test -j0 now runs each test file in a subprocess. Maybe we should emit a warning when tests are run sequentially, without -j0. Maybe it should be an opt-in option, rather than the default. But I prefer to close this old issue. Please open a new issue if you want to make further changes in regrtest.