Refactor subprocess.Popen to let a subclass handle IO asynchronously

[Martiusweb]

BPO	28287
Nosy	@gpshead, @pitrou, @giampaolo, @1st1, @Martiusweb
PRs	bpo-28287: Refactor subprocess.Popen to let a subclass handle IO asynchronously #6878
Files	popen_execute_child_refactoring.patch: popen_execut_child_refactoring.patch, should work with python 3.4+

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2016-09-27.14:26:43.785>
labels = ['type-feature', '3.8']
title = 'Refactor subprocess.Popen to let a subclass handle IO asynchronously'
updated_at = <Date 2018-05-15.21:06:57.317>
user = 'https://github.com/Martiusweb'

bugs.python.org fields:

activity = <Date 2018-05-15.21:06:57.317>
actor = 'python-dev'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2016-09-27.14:26:43.785>
creator = 'martius'
dependencies = []
files = ['44844']
hgrepos = []
issue_num = 28287
keywords = ['patch']
message_count = 4.0
messages = ['277517', '298358', '300839', '301024']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'pitrou', 'giampaolo.rodola', 'yselivanov', 'martius']
pr_nums = ['6878']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28287'
versions = ['Python 3.8']

[Martiusweb]

Hi,

Currently, subprocess.Popen performs blocking IO in its constructor (at least on Unix): it reads on a pipe in order to detect outcome of the pre-exec and exec phase in the new child. There is no way yet to modify this behavior as this blocking call is part of a long Popen._execute_child() method.

This is a problem in asyncio (asyncio.subprocess_exec and asyncio.subprocess_shell).

I would like to submit a patch which breaks Popen.__init__() and Popen._execute_child() in several methods so it becomes possible to avoid blocking calls (read on pipe and waitpid) by overriding a few private methods without duplicating too much code. The goal is to use it in asyncio, as described in this pull request (which currently monkey-patches Popen):
python/asyncio#428

This patch only targets the unix implementation.

Thanks for your feedback.

[giampaolo]

Is the main goal to just make Popen.__init__ non-blocking? If not, bpo-1191964 aims at providing non-blocking reads/writes (including on Windows).

[Martiusweb]

Yes, the goal is to isolate the blocking IO in __init__ into other methods so Popen can be subclassed in asyncio.

The end goal is to ensure that when asyncio calls Popen(), it doesn't block the process. In the context of asyncio, there's no need to make Popen() IOs non-blocking as they will be performed with the asyncio API (rather than the IO methods provided by the Popen object).

[pitrou]

By the way, we're using GitHub PRs now for development, though you can still submit patch files if you prefer that.