subprocess: more general (non-buffering) communication

[ianbicking]

BPO	1260171
Nosy	@birkenfeld, @gpshead, @giampaolo, @vadmium
Files	subprocess_communicate_alternative.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2020-10-20.01:58:46.586>
created_at = <Date 2005-08-15.19:15:12.000>
labels = ['type-feature', 'library']
title = 'subprocess: more general (non-buffering) communication'
updated_at = <Date 2020-10-20.01:58:46.585>
user = 'https://bugs.python.org/ianbicking'

bugs.python.org fields:

activity = <Date 2020-10-20.01:58:46.585>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2020-10-20.01:58:46.586>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2005-08-15.19:15:12.000>
creator = 'ianbicking'
dependencies = []
files = ['8255']
hgrepos = []
issue_num = 1260171
keywords = ['patch']
message_count = 8.0
messages = ['54590', '54591', '54592', '54593', '161338', '222926', '245159', '379076']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'gregory.p.smith', 'ianbicking', 'astrand', 'giampaolo.rodola', 'cvrebert', 'rosslagerwall', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1260171'
versions = ['Python 3.5']

[ianbicking]

Right now you can use subprocess.Popen.communicate() to
make communication with subprocesses much easier and
less likely to block than communicating directly with
.stdin, .stdout, etc. However, that requires
completely buffering the input and output.

The functionality of communicate() (which is somewhat
complex because of platform issues) could be made more
general fairly easily. The current functionality of
communicate could then be implemented in terms of that
new method.

I attached a function I'm using which does that for the
posix systems (basically turning Popen's posix
communicate into a function with some modifications).
Replace "proc" with "self" (and give the function a
better name) and you'd have a method.

If patch 1175984 was accepted, then this wouldn't be
that much of an issue:
http://sourceforge.net/tracker/index.php?func=detail&aid=1175984&group_id=5470&atid=305470

[josiahcarlson]

Logged In: YES
user_id=341410

Would an asynchronous subprocess (which you would poll
manually) be better/sufficient?

http://python.org/sf/1191964

[josiahcarlson]

Logged In: YES
user_id=341410

Also, what you post is not a 'bug', it is a 'feature request'.

[birkenfeld]

Logged In: YES
user_id=1188172

Reclassifying as RFE. Assigning to Peter.

[rosslagerwall]

Closed bpo-14872 as a duplicate of this.

[BreamoreBoy]

Please note the work being performed on bpo-1191964 which was first referenced in msg54591.

[vadmium]

The non-blocking read and write features proposed in bpo-1191964 wouldn’t be sufficient. Perhaps they might be useful for implementing this feature on Windows though; I’m not sure. And a non-blocking write on Posix may be slightly more efficient as well, removing the limitation of copying in chunks of PIPE_BUF bytes.

Ian’s original patch posted here proposes a version of communicate() which accepts a file reader for the subprocess’s input, and file writers for the subprocess’s outputs. Another option could be something like my SubprocessWriter class <https://github.com/vadmium/pacman-tools/blob/9ffdd88/makeaur#L179\>. It provides a file writer interface for the caller to write to the subprocess’s input, while copying data from the subprocess’s output behind the scenes. I used it to stream a tar file, as it is written by the “tarfile” module, into a GPG subprocess to create a digital signature, while writing the output of GPG into a BytesIO buffer. Using Unix command pipeline pseudocode, it is used a bit like this:

tarfile.open(mode="w|") | Popen("gpg") | BytesIO()