multiprocessing.Pool: emit ResourceWarning

[vstinner]

BPO	35424
Nosy	@vstinner, @pablogsal, @miss-islington
PRs	bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction #10974 bpo-35424: test_multiprocessing: join 3 pools #10986 [3.7] bpo-35424: test_multiprocessing: join 3 pools (GH-10986) #10987 [3.6] bpo-35424: test_multiprocessing: join 3 pools (GH-10986) #10988 bpo-35424: Fix test_multiprocessing_main_handling #11223 [3.7] bpo-35424: Fix test_multiprocessing_main_handling (GH-11223) #11227

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-20.19:37:22.702>
created_at = <Date 2018-12-06.01:00:34.427>
labels = ['3.8', 'tests']
title = 'multiprocessing.Pool: emit ResourceWarning'
updated_at = <Date 2018-12-20.19:37:22.701>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-12-20.19:37:22.701>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-20.19:37:22.702>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2018-12-06.01:00:34.427>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35424
keywords = ['patch']
message_count = 10.0
messages = ['331201', '331220', '331223', '331230', '331231', '331823', '332097', '332100', '332249', '332251']
nosy_count = 3.0
nosy_names = ['vstinner', 'pablogsal', 'miss-islington']
pr_nums = ['10974', '10986', '10987', '10988', '11223', '11227']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35424'
versions = ['Python 3.8']

[vstinner]

Since 2 years, I'm fixing frequently "dangling thread" and "dangling process" warnings on buildbots. These bugs are really hard to reproduce. They usually require to get access to a specific buildbot, simulate a specific workload, and get the proper timing to get the warning.

I propose to emit a ResourceWarning in multiprocessing.Pool destructor if the pool has not been cleaned properly. I'm not sure in which cases a warning should be emitted.

Attached PR is a WIP implementation.

[vstinner]

While testing PR 10974, I found two tests which emits ResourceWarning. I wrote 10986 to fix them.

[vstinner]

New changeset 388c8c2 by Victor Stinner in branch 'master':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
388c8c2

[miss-islington]

New changeset b7c67c4 by Miss Islington (bot) in branch '3.7':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
b7c67c4

[miss-islington]

New changeset e44b5b2 by Miss Islington (bot) in branch '3.6':
bpo-35424: test_multiprocessing: join 3 pools (GH-10986)
e44b5b2

[vstinner]

See this discussion:
[Python-Dev] Usage of the multiprocessing API and object lifetime
https://mail.python.org/pipermail/python-dev/2018-December/155946.html

[vstinner]

New changeset 6cdce3d by Victor Stinner in branch 'master':
bpo-35424: Fix test_multiprocessing_main_handling (GH-11223)
6cdce3d

[vstinner]

New changeset c74e7c4 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-35424: Fix test_multiprocessing_main_handling (GH-11223) (GH-11227)
c74e7c4

[vstinner]

New changeset 9a8d1d7 by Victor Stinner in branch 'master':
bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction (GH-10974)
9a8d1d7

[vstinner]

The first implementation of my PR also emitted ResourceWarning when .join() hasn't been called, but the merged PR only emit a warning if .close() and .terminated() have not been called: if the pool is still running.