Cleaning up a subprocess with a broken pipe #65818
Comments
|
The documentation for the “subprocess” module says that a “with” statement will “wait for” the process, implying that it does not leave a zombie. However this is not the case if there is buffered input data: $ python3 -Wall -bt -q
>>> import subprocess
>>> with subprocess.Popen(("true",), stdin=subprocess.PIPE, bufsize=-1) as p:
... from time import sleep; sleep(1) # Wait for pipe to be broken
... p.stdin.write(b"buffered data")
...
13
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "/usr/lib/python3.4/subprocess.py", line 899, in __exit__
self.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe
>>> # (Hit Ctrl-Z here)
[1]+ Stopped python3 -Wall -bt -q
[Exit 148]
$ ps
PID TTY TIME CMD
15867 pts/5 00:00:00 python3
15869 pts/5 00:00:00 true <defunct>
15873 pts/5 00:00:00 ps
32227 pts/5 00:00:10 bashSimilarly, calling Popen.communicate() does not clean the process up either if there is buffered input data and the process has already exited. The documentation does not spell out how a broken pipe is handled in communicate(), but after reading bpo-10963 I see that in many other cases it is meant to be ignored. The best way to clean up a subprocess that I have come up with to close the pipe(s) and call wait() in two separate steps, such as: try: |
|
Here is a patch to fix this by calling wait() even if stdin.close() fails, including a test case. With my patch, the subprocess context manager __exit__() will still raise a BrokenPipeError, but no zombie will be left. |
|
New changeset b5e9ddbdd4a7 by Serhiy Storchaka in branch '3.4': New changeset cdac249808a8 by Serhiy Storchaka in branch 'default': |
|
Thank you for your contribution Martin. |
|
Why not ignoring BrokenPipeError like communicate()? |
|
New changeset 1b4d916329e7 by Serhiy Storchaka in branch '3.4': New changeset eae459e35cb9 by Serhiy Storchaka in branch 'default': |
|
The test still sporadically fails on Windows: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/9323/steps/test/logs/stdio Traceback (most recent call last):
File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_subprocess.py", line 2515, in test_broken_pipe_cleanup
self.assertRaises(OSError, proc.__exit__, None, None, None)
AssertionError: OSError not raised by __exit__ |
Why communicate() ignores BrokenPipeError? |
|
It was added in bpo-10963. I don't know if this way is applicable to this issue. |
|
Le dimanche 1 mars 2015, Serhiy Storchaka <report@bugs.python.org> a écrit :
It's more convinient. There is nothing useful you can do on pipe error in For __exit__, what do you want to on broken pipe error? I did't check yet: |
|
When you write to the file you don't want the error was silently ignored. with open(filename, 'w') as f:
f.write(content)And also I don't want the error was silently ignored when write to the subprocess. with subprocess.Popen(cmd, universal_newlines=True, stdin=subprocess.PIPE) as p:
p.stdin.write(content) |
|
It seems two different issues have popped up: ## 1. Windows behaviour ## Windows apparently doesn’t handle broken pipes consistently, sometimes raising an EINVAL error, and sometimes not raising any error. I don’t know if this is a problem with Python, or a quirk with Windows. (Just tried with Wine, and it always raises ENOSPC instead, but I guess that’s a Wine-specific quirk.) Perhaps we can change the offending test case to the following, at least until someone can investigate and explain what is going on: ... ## 2. Suppressing BrokenPipeError ## I don’t have any strong opinions. I think in the past when writing to a process’s input raises BrokenPipeError, I have tended to skip the writing and go on to analyse the process’s output and/or exit status (the same as if writing succeeded). Some arguments for raising BrokenPipeError (current behaviour):
Arguments for suppressing BrokenPipeError:
Replying to Victor: “the process always exit after __exit__?”: __exit__ is meant to call wait(), so it only returns when the subprocess exits. It could potentially still be running despite closing its input pipe. This issue was originally about making sure __exit__ called wait() in all cases. |
|
Thanks for that link; the answer by Eryksun <https://stackoverflow.com/questions/23688492/oserror-errno-22-invalid-argument-in-subprocess#28020096\> is particularly enlightening. Apparently EINVAL actually represents an underlying broken pipe condition in Windows. Anyway, as far as the test is concerned, it doesn’t matter what particular exception is raised. This still doesn’t explain Serhiy’s observation that sometimes no exception is raised at all, <https://bugs.python.org/issue21619#msg236949\>. Does Windows sometimes not report broken pipe errors, or is my signalling logic not working and there is a race condition? The whole point of the test is to trigger an exception when stdin is closed, so it would be nice to be able to reliably do that on Windows. |
|
A few months ago, I modified Popen.communicate() to handle EINVAL on |
|
I opened the issue bpo-23570: Change "with subprocess.Popen():" (context manager) to ignore broken pipe error.
Serhiy: see existing test_communicate_epipe() and test_communicate_epipe_only_stdin() tests which are now reliable and portable. You should write more data (2**20 bytes) and set the buffer size to a value larger than the input data, to buffer all data, so the write occurs at stdin.close() in Popen.__exit__(). By the way, instead of an hardcoded value (2**20), support.PIPE_MAX_SIZE may be more appropriate. |
|
Aha! So perhaps Windows can accept a small amount of data into its pipe buffer even if we know the pipe has been broken. That kind of makes sense. Test case could be modified to: proc = subprocess.Popen([...], bufsize=support.PIPE_MAX_SIZE, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.stdout.read() # Make sure subprocess has closed its input
proc.stdin.write(bytes(support.PIPE_MAX_SIZE))
self.assertIsNone(proc.returncode)
# Expect EPIPE under POSIX and EINVAL under Windows
self.assertRaises(OSError, proc.__exit__, None, None, None) |
|
Could you provide a patch Martin? |
|
I would be safer to use a bufsize a little bit larger :-) proc = subprocess.Popen([...], bufsize=support.PIPE_MAX_SIZE * 2,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
...
proc.stdin.write(b'x' * support.PIPE_MAX_SIZE) |
|
New changeset 77a978716517 by Victor Stinner in branch '3.4': |
|
Looks as tests are fixed. Thank you Victor. |
|
Thanks for getting the test working. Just to tidy things up here I would like to get rid of my stdout signalling in the test, which is no longer needed and could be misleading. See overflow-pipe-test.patch. |
|
overflow-pipe-test.patch looks good to me. |
|
I think we should add __enter__ for consistency. |
|
Sure, new version is fine by me |
|
New changeset 4ea40dc3d26d by Serhiy Storchaka in branch '3.4': New changeset 41ce95a5b2d8 by Serhiy Storchaka in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: