Create a way to check that the parent process is alive for deamonized processes

[tomMoral]

BPO	36888
Nosy	@pitrou, @vstinner, @applio, @tomMoral, @miss-islington
PRs	bpo-36888: Add multiprocessing.parent_process() #13247 bpo-36888: Increase timeout in test_parent_process #14286 [3.8] bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286) #14382

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-25.22:27:51.849>
created_at = <Date 2019-05-11.15:36:32.843>
labels = ['3.8', 'type-feature', 'library']
title = 'Create a way to check that the parent process is alive for deamonized processes'
updated_at = <Date 2019-06-25.22:27:51.848>
user = 'https://github.com/tomMoral'

bugs.python.org fields:

activity = <Date 2019-06-25.22:27:51.848>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-25.22:27:51.849>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-05-11.15:36:32.843>
creator = 'tomMoral'
dependencies = []
files = []
hgrepos = []
issue_num = 36888
keywords = ['patch']
message_count = 7.0
messages = ['342199', '342945', '342946', '345321', '346558', '346560', '346563']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'davin', 'tomMoral', 'miss-islington']
pr_nums = ['13247', '14286', '14382']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36888'
versions = ['Python 3.8']

[tomMoral]

In the std lib, the semaphore_tracker and the Manager rely on daemonized processes that are launched with server like loops. The cleaning of such processes is made complicated by the fact that there is no cannonical way to check that the parent process is alive.

I propose to add in context a parent_process function that would give access to a Process object representing the parent process. This way, we could benefit from sentinel to improve the clean up of this process that can be left dangling in case of hard stop from the main interpreter.

[pitrou]

New changeset c09a9f5 by Antoine Pitrou (Thomas Moreau) in branch 'master':
bpo-36888: Add multiprocessing.parent_process() (GH-13247)
c09a9f5

[pitrou]

Waiting for the next PR now :-)

[vstinner]

This new test is not reliable, it failed on x86 Gentoo Refleaks 3.8:
https://buildbot.python.org/all/#/builders/223/builds/9

Please fix the test or revert the change. The CI must remain green ;-)

running: test_multiprocessing_spawn (31 min 51 sec), test_multiprocessing_forkserver (3 min 7 ms)
3:20:42 load avg: 6.89 [415/423/1] test_multiprocessing_spawn failed (31 min 30 sec) -- running: test_multiprocessing_forkserver (3 min 7 sec)
beginning 6 repetitions
123456
....Process Process-1588:1:
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 326, in _test_report_parent_status
    wconn.send("alive" if parent_process().is_alive() else "not alive")
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
  Before: set()
  After:  {<weakref at 0xb6d7e840; to 'Process' at 0xb4573878>} 
test test_multiprocessing_spawn failed -- Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process

/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_b32d1a2f': [Errno 2] No such file or directory: '//psm_b32d1a2f'
warnings.warn('resource_tracker: %r: %s' % (name, e))

(...)

Re-running test_multiprocessing_spawn in verbose mode

======================================================================
FAIL: test_parent_process (test.test_multiprocessing_spawn.WithProcessesTestProcess)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process

[vstinner]

New changeset 594d9b9 by Victor Stinner (Pierre Glaser) in branch 'master':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
594d9b9

[miss-islington]

New changeset 4adc38e by Miss Islington (bot) in branch '3.8':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
4adc38e

[vstinner]

Thanks Pierre Glaser. Let's see if it's enough to make the test more reliable on buildbots ;-)