New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
subprocess.Popen can hang in threaded applications in Python 2 #64517
Comments
Running python python_thread_bug.py -j4 often results in one of the threads failing to start until another thread finishes. The bug appears to be that subprocess's pipe_cloexec function is racy: if another thread forks between os.pipe() and _set_cloexec_flag, then the resulting process could hang on to the write end of the pipe. That will cause the Popen call that got rudely interrupted to wait until the whole resulting process tree dies. |
Hmm. I thought someone had already reported this, but I can't find an issue. I suspect it is fixed in 3.4 and it may not be practical to fix it in earlier versions. |
2.7 subprocess is fundamentally flawed and cannot be fixed. Install subprocess32 from PyPI. It's fixed in 3.2 and later. |
Would it be worth adding something to the Python 2.7 subprocess docs indicating that subprocess is known to be broken? |
FWIW, sticking a mutex in Popen.__init__ (wrapping the whole thing) seems to work around this issue (for programs that aren't using multiprocessing or fork, for example). This might be a good-enough fix and be safe enough to stick in the standard library. |
A pointer in the 2.7 subprocess docs to subprocess32 does seem like a good idea, its what i tell everyone to do anyways. :) If you've got a patch for this in 2.7 feel free to add it here and I can take a look. leaving this open as a reminder to me to update the docs at the very least. |
I added a pointer to subprocess32 in the 2.7 subprocess docs in dd52365c8721. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: