Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a way to check that the parent process is alive for deamonized processes #81069

Closed
tomMoral mannequin opened this issue May 11, 2019 · 7 comments
Closed

Create a way to check that the parent process is alive for deamonized processes #81069

tomMoral mannequin opened this issue May 11, 2019 · 7 comments
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@tomMoral
Copy link
Mannequin

tomMoral mannequin commented May 11, 2019

BPO 36888
Nosy @pitrou, @vstinner, @applio, @tomMoral, @miss-islington
PRs
  • bpo-36888: Add multiprocessing.parent_process() #13247
  • bpo-36888: Increase timeout in test_parent_process #14286
  • [3.8] bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286) #14382
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-06-25.22:27:51.849>
    created_at = <Date 2019-05-11.15:36:32.843>
    labels = ['3.8', 'type-feature', 'library']
    title = 'Create a way to check that the parent process is alive for deamonized processes'
    updated_at = <Date 2019-06-25.22:27:51.848>
    user = 'https://github.com/tomMoral'

    bugs.python.org fields:

    activity = <Date 2019-06-25.22:27:51.848>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-06-25.22:27:51.849>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2019-05-11.15:36:32.843>
    creator = 'tomMoral'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36888
    keywords = ['patch']
    message_count = 7.0
    messages = ['342199', '342945', '342946', '345321', '346558', '346560', '346563']
    nosy_count = 5.0
    nosy_names = ['pitrou', 'vstinner', 'davin', 'tomMoral', 'miss-islington']
    pr_nums = ['13247', '14286', '14382']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue36888'
    versions = ['Python 3.8']

    @tomMoral
    Copy link
    Mannequin Author

    tomMoral mannequin commented May 11, 2019

    In the std lib, the semaphore_tracker and the Manager rely on daemonized processes that are launched with server like loops. The cleaning of such processes is made complicated by the fact that there is no cannonical way to check that the parent process is alive.

    I propose to add in context a parent_process function that would give access to a Process object representing the parent process. This way, we could benefit from sentinel to improve the clean up of this process that can be left dangling in case of hard stop from the main interpreter.

    @tomMoral tomMoral mannequin added 3.9 only security fixes 3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 11, 2019
    @pitrou
    Copy link
    Member

    pitrou commented May 20, 2019

    New changeset c09a9f5 by Antoine Pitrou (Thomas Moreau) in branch 'master':
    bpo-36888: Add multiprocessing.parent_process() (GH-13247)
    c09a9f5

    @pitrou
    Copy link
    Member

    pitrou commented May 20, 2019

    Waiting for the next PR now :-)

    @pitrou pitrou removed the 3.9 only security fixes label May 20, 2019
    @pitrou pitrou closed this as completed May 20, 2019
    @vstinner
    Copy link
    Member

    This new test is not reliable, it failed on x86 Gentoo Refleaks 3.8:
    https://buildbot.python.org/all/#/builders/223/builds/9

    Please fix the test or revert the change. The CI must remain green ;-)

    running: test_multiprocessing_spawn (31 min 51 sec), test_multiprocessing_forkserver (3 min 7 ms)
    3:20:42 load avg: 6.89 [415/423/1] test_multiprocessing_spawn failed (31 min 30 sec) -- running: test_multiprocessing_forkserver (3 min 7 sec)
    beginning 6 repetitions
    123456
    ....Process Process-1588:1:
    Traceback (most recent call last):
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 313, in _bootstrap
        self.run()
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 326, in _test_report_parent_status
        wconn.send("alive" if parent_process().is_alive() else "not alive")
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 206, in send
        self._send_bytes(_ForkingPickler.dumps(obj))
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
        self._send(header + buf)
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe
    Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
    Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
    Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
      Before: set()
      After:  {<weakref at 0xb6d7e840; to 'Process' at 0xb4573878>} 
    test test_multiprocessing_spawn failed -- Traceback (most recent call last):
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
        raise AssertionError("Could not communicate with child process")
    AssertionError: Could not communicate with child process

    /buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '
    /buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_b32d1a2f': [Errno 2] No such file or directory: '//psm_b32d1a2f'
    warnings.warn('resource_tracker: %r: %s' % (name, e))

    (...)

    Re-running test_multiprocessing_spawn in verbose mode

    ======================================================================
    FAIL: test_parent_process (test.test_multiprocessing_spawn.WithProcessesTestProcess)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
        raise AssertionError("Could not communicate with child process")
    AssertionError: Could not communicate with child process

    @vstinner vstinner reopened this Jun 12, 2019
    @vstinner
    Copy link
    Member

    New changeset 594d9b9 by Victor Stinner (Pierre Glaser) in branch 'master':
    bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
    594d9b9

    @miss-islington
    Copy link
    Contributor

    New changeset 4adc38e by Miss Islington (bot) in branch '3.8':
    bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
    4adc38e

    @vstinner
    Copy link
    Member

    Thanks Pierre Glaser. Let's see if it's enough to make the test more reliable on buildbots ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants