multiprocess: catch explicitly SystemExit in the Server class

[vstinner]

BPO	43867
Nosy	@brianquinlan, @pitrou, @vstinner, @applio, @pablogsal
PRs	bpo-43867: multiprocessing Server catchs SystemExit #25441

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-16.17:43:28.393>
created_at = <Date 2021-04-16.14:41:31.153>
labels = ['library', '3.10']
title = 'multiprocess: catch explicitly SystemExit in the Server class'
updated_at = <Date 2021-04-16.17:43:28.392>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-16.17:43:28.392>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-16.17:43:28.393>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-04-16.14:41:31.153>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43867
keywords = ['patch']
message_count = 3.0
messages = ['391205', '391207', '391240']
nosy_count = 5.0
nosy_names = ['bquinlan', 'pitrou', 'vstinner', 'davin', 'pablogsal']
pr_nums = ['25441']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43867'
versions = ['Python 3.10']

[vstinner]

In bpo-43843, I modified libregrtest to log a warning if a thread raises an uncaught exception. test_concurrent_futures now logs two warnings.

s390x Debian 3.x:
https://buildbot.python.org/all/#/builders/49/builds/1124

AMD64 RHEL7 3.x:
https://buildbot.python.org/all/#/builders/15/builds/1074

0:03:18 load avg: 4.78 [302/427] test_concurrent_futures passed (2 min 46 sec) -- running: test_tokenize (37.8 sec), test_peg_generator (59.2 sec)
Warning -- Uncaught thread exception: SystemExit
Warning -- Uncaught thread exception: SystemExit

The problem can be seen with test_ressources_gced_in_workers():

$ ./python -m test test_concurrent_futures -m test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest.test_ressources_gced_in_workers -v
(...)
test_ressources_gced_in_workers (test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest) ... 
Warning -- Uncaught thread exception: SystemExit
Warning -- Uncaught thread exception: SystemExit
(...)
Tests result: SUCCESS

I propose to explicitly catch this SystemExit. We can take the opportunity to call c.close() explicitly: see attached PR.

[vstinner]

Oh sorry, I was confused. While the issue is seen in test_concurrent_futures, my PR 25441 changes the multiprocessing module.

[vstinner]

New changeset 7c29ae1 by Victor Stinner in branch 'master':
bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)
7c29ae1