OSError in os.waitpid() on Windows [3.5.0 regression]

[RoccoMatano]

BPO	25118
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @eryksun, @zooba, @RoccoMatano

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-09-15.08:34:47.048>
created_at = <Date 2015-09-15.05:52:11.104>
labels = ['type-bug', 'library', 'OS-windows']
title = 'OSError in os.waitpid() on Windows [3.5.0 regression]'
updated_at = <Date 2015-09-21.07:18:47.663>
user = 'https://github.com/RoccoMatano'

bugs.python.org fields:

activity = <Date 2015-09-21.07:18:47.663>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-15.08:34:47.048>
closer = 'vstinner'
components = ['Library (Lib)', 'Windows']
creation = <Date 2015-09-15.05:52:11.104>
creator = 'rocco.matano'
dependencies = []
files = []
hgrepos = []
issue_num = 25118
keywords = ['3.5regression']
message_count = 6.0
messages = ['250723', '250728', '250735', '250738', '250760', '250762']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower', 'rocco.matano']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25118'
versions = ['Python 3.5', 'Python 3.6']

[RoccoMatano]

On windows and python up to 3.4 using the combination of os.spawnX(os.P_NOWAIT, ...) and os.waitpid() worked as expected, but with python 3.5 this no longer works. In fact os.waitpid() now raises an OSError when the child process terminates.
Running this simple script demonstrates that:

import sys
import os
import time

print(sys.version)

file = r'c:\windows\system32\cmd.exe'
args = [file, '/C', 'ping', '1.1.1.1', '-n', '1', '>NUL']
env = os.environ

t0 = time.time()
ret = os.spawnve(os.P_NOWAIT, file, args, env)
pid, status = os.waitpid(ret, 0)
t1 = time.time()
print("process took %.1f seconds" % (t1 - t0))

I get the following outputs:

C:\Users\rmatano\test_py35>py -3.4 test_waitpid.py
3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)]
process took 3.6 seconds

C:\Users\rmatano\test_py35>py -3.5 test_waitpid.py
3.5.0 (v3.5.0:374f501f4567, Sep 13 2015, 02:27:37) [MSC v.1900 64 bit (AMD64)]
Traceback (most recent call last):
  File "twp.py", line 13, in <module>
    pid, status = os.waitpid(ret, 0)
OSError: [Errno 0] Error

I guess this is a bug rather than a intended change in behavior, isn't it?

[eryksun]

This bug is due to bpo-23285, which improved support for EINTR handling. os_waitpid_impl was changed to use the following do-while loop:

do {
    Py_BEGIN_ALLOW_THREADS
    res = _cwait(&status, pid, options);
    Py_END_ALLOW_THREADS
} while (res < 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()));
if (res != 0)
    return (!async_err) ? posix_error() : NULL;

The last test should be (res < 0) instead of (res != 0). That's why you're getting a no-error exception.

It seems to me this entire loop should be removed. The Windows C runtime doesn't set errno to EINTR. In the case of _cwait it doesn't even use an alertable wait (i.e. it can't be interrupted by a regular asynchronous procedure call).

[python-dev]

New changeset 9a80c687c28d by Victor Stinner in branch '3.5':
Issue bpo-25118: Fix a regression of Python 3.5.0 in os.waitpid() on Windows.
https://hg.python.org/cpython/rev/9a80c687c28d

[vstinner]

Oops :-/ Yet another Python 3.5.0 regression, it's now fixed.

os.waitpid() was not tested at all on Windows. os.waitpid() is not the best option to wait for a subprocess completion. By the way, you should use the subprocess module which is more portable, is widely used, etc. IMHO os.spawn*() is more kept for backward compatibility.

It seems to me this entire loop should be removed. The Windows C runtime doesn't set errno to EINTR. In the case of _cwait it doesn't even use an alertable wait (i.e. it can't be interrupted by a regular asynchronous procedure call).

Well, it looks like you are right: _cwait() cannot be interrupted on Windows. But I chose to only fix the if() after the loop... just in case.

[RoccoMatano]

I know that using os.spawn and and os.waitpid this way is not the best option, but a 3rd party tool i am using (scons) is doing it that way. So no scons with Python 3.5.0. (I am also aware that scons does not yet support Python 3.x officially.)

[vstinner]

I know that using os.spawn and and os.waitpid this way is not the best option, but a 3rd party tool i am using (scons) is doing it that way.

No worry, it was just a comment. Anyway, the bug is now fixed and will be part of the next Python 3.5.1 release.