[Windows] multiprocessing: DupHandle.detach() race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE)

[vstinner]

BPO	38263
Nosy	@pitrou, @eryksun, @s0undt3ch, @pablogsal, @yan Ren

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-09-24.10:36:39.140>
labels = ['build', 'library', '3.9']
title = '[Windows] multiprocessing: DupHandle.detach() race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE)'
updated_at = <Date 2021-04-10.01:07:14.748>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-10.01:07:14.748>
actor = 'leonardo2'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-09-24.10:36:39.140>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38263
keywords = []
message_count = 8.0
messages = ['353064', '362460', '362466', '362471', '377296', '389832', '390229', '390683']
nosy_count = 7.0
nosy_names = ['pitrou', 'eryksun', 's0undt3ch', 'pablogsal', 'pemryan', 'jesvi22j', 'leonardo2']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue38263'
versions = ['Python 3.9']

[vstinner]

On Windows, the multiprocessing DupHandle.detach() method has race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE).

Error on duplicate():

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access is denied

Example: bpo-34714

Error on detach():

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 117, in spawn_main
    exitcode = _main(fd)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 127, in _main
    self = reduction.pickle.load(from_parent)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\connection.py", line 951, in rebuild_pipe_connection
    handle = dh.detach()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\reduction.py", line 133, in detach
    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

Example: bpo-34513

[s0undt3ch]

Any possible workaround for this issue?
I seem to be consistingly hitting this issue.

[eryksun]

I seem to be consistingly hitting this issue.

It will help with test development if you can provide a minimal example that reliably reproduces the problem.

In msg353064 I see DuplicateHandle calls failing with ERROR_ACCESS_DENIED (5). Assuming the process handles have the required PROCESS_DUP_HANDLE access, it's most likely the case that the underlying NtDuplicateObject system call is failing with STATUS_PROCESS_IS_TERMINATING (0xC000010A). For example:

    import ctypes
    ntdll = ctypes.WinDLL('ntdll')
    from subprocess import Popen, PIPE
    from _winapi import GetCurrentProcess, TerminateProcess
    from _winapi import DuplicateHandle, DUPLICATE_SAME_ACCESS

    p = Popen('cmd.exe', stdin=PIPE, stdout=PIPE, stderr=PIPE)
    TerminateProcess(p._handle, 0)

Try to duplicate the process handle into the terminated process:

    >>> source = GetCurrentProcess()
    >>> target = handle = p._handle
    >>> try:
    ...     DuplicateHandle(source, handle, target,
    ...         0, False, DUPLICATE_SAME_ACCESS)
    ... except:
    ...     status = ntdll.RtlGetLastNtStatus() 
    ...     print(f'NTSTATUS: {status & (2**32-1):#010x}')
    ...     raise
    ...
    NTSTATUS: 0xc000010a
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    PermissionError: [WinError 5] Access is denied

[s0undt3ch]

What I'm able to copy from the console(though I doubt I'm getting all of
the traceback):

Traceback (most recent call last):

File "<string>", line 1, in <module>

File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\spawn.py",
line 106, in spawn_main

    exitcode = _main(fd)

File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\spawn.py",
line 116, in _main

    self = pickle.load(from_parent)

File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\connection.py",
line 942, in rebuild_pipe_connection

    handle = dh.detach()

File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\reduction.py",
line 128, in detach

self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)

PermissionError: [WinError 5] Access is denied

Pedro Algarvio @ Phone

A sábado, 22/02/2020, 13:44, Eryk Sun <report@bugs.python.org> escreveu:

>
> Eryk Sun <eryksun@gmail.com> added the comment:
>
> > I seem to be consistingly hitting this issue.
>
> It will help with test development if you can provide a minimal example
> that reliably reproduces the problem.
>
> In msg353064 I see DuplicateHandle calls failing with ERROR_ACCESS_DENIED
> (5). Assuming the process handles have the required PROCESS_DUP_HANDLE
> access, it's most likely the case that the underlying NtDuplicateObject
> system call is failing with STATUS_PROCESS_IS_TERMINATING (0xC000010A). For
> example:
>
>     import ctypes
>     ntdll = ctypes.WinDLL('ntdll')
>     from subprocess import Popen, PIPE
>     from _winapi import GetCurrentProcess, TerminateProcess
>     from _winapi import DuplicateHandle, DUPLICATE_SAME_ACCESS
>
>     p = Popen('cmd.exe', stdin=PIPE, stdout=PIPE, stderr=PIPE)
>     TerminateProcess(p._handle, 0)
>
> Try to duplicate the process handle into the terminated process:
>
>     >>> source = GetCurrentProcess()
>     >>> target = handle = p._handle
>     >>> try:
>     ...     DuplicateHandle(source, handle, target,
>     ...         0, False, DUPLICATE_SAME_ACCESS)
>     ... except:
>     ...     status = ntdll.RtlGetLastNtStatus()
>     ...     print(f'NTSTATUS: {status & (2**32-1):#010x}')
>     ...     raise
>     ...
>     NTSTATUS: 0xc000010a
>     Traceback (most recent call last):
>       File "<stdin>", line 2, in <module>
>     PermissionError: [WinError 5] Access is denied
>
> 

---

nosy: +eryksun

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38263\>

---

[YanRen]

append Nosy List.

[eryksun]

spawn_main() could handle a PermissionError by checking the exit code. If the parent has terminated already, then simply exit quietly. _winapi.PROCESS_QUERY_LIMITED_INFORMATION would need to be defined. For example:

    def spawn_main(pipe_handle, parent_pid=None, tracker_fd=None):
        """Run code specified by data received over a pipe."""
        assert is_forking(sys.argv), "Not forking"

        if sys.platform == 'win32':
            import msvcrt
            import _winapi

            if parent_pid is not None:
                source_process = _winapi.OpenProcess(
                    _winapi.SYNCHRONIZE |
                    _winapi.PROCESS_DUP_HANDLE |
                    _winapi.PROCESS_QUERY_LIMITED_INFORMATION,
                    False, parent_pid)
            else:
                source_process = None
            try:
                new_handle = reduction.duplicate(
                    pipe_handle, source_process=source_process)
                fd = msvcrt.open_osfhandle(new_handle, os.O_RDONLY)
                exitcode = _main(fd, source_process)
            except PermissionError:
                if (source_process is None or
                      _winapi.GetExitCodeProcess(source_process) ==
                      _winapi.STILL_ACTIVE):
                    raise
                exitcode = 1
        else:
            from . import resource_tracker
            resource_tracker._resource_tracker._fd = tracker_fd
            exitcode = _main(pipe_handle, os.dup(pipe_handle))

        sys.exit(exitcode)

[jesvi22j]

File "c:/Users/jesvi/Documents/GitHub/Jesvi-Bot-Telegram/scripts/main.py", line 144, in thread_test
    p.start()
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 107, in spawn_main
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 79, in duplicate
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid

[leonardo2]

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Program Files\Python39\lib\multiprocessing\connection.py", line 967, in rebuild_pipe_connection
    handle = dh.detach()
  File "C:\Program Files\Python39\lib\multiprocessing\reduction.py", line 131, in detach
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Acesso negado

[vstinner]

While it's a real issue, sadly it seems like nobody is available to propose a real fix (PR). I don't have the bandwidth to work on this issue, so sorry I just close it.