subprocess line-buffering only works in universal newlines mode

[pitrou]

BPO	21471
Nosy	@gpshead, @pitrou, @benjaminp, @4kir4, @hynek

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-05-11.11:25:06.568>
labels = ['type-bug', 'library']
title = 'subprocess line-buffering only works in universal newlines mode'
updated_at = <Date 2014-05-11.12:20:06.750>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2014-05-11.12:20:06.750>
actor = 'akira'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2014-05-11.11:25:06.568>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 21471
keywords = []
message_count = 4.0
messages = ['218257', '218259', '218260', '218263']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'pitrou', 'benjamin.peterson', 'stutzbach', 'Arfrever', 'akira', 'hynek']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21471'
versions = ['Python 3.5']

[pitrou]

The docs for subprocess.Popen seem to imply that line-buffering is always available. However, bufsize=1 is a special value only when open the pipes in text mode, i.e. when "universal newlines" are enabled.

In the short term, we should probably fix the subprocess docs.
In the long term, perhaps we can add a line buffering capability to BufferedWriter?

[pitrou]

Ha, it seems actually worse than that, since no buffering argument is ever passed to the TextIOWrapper constructor. "bufsize=1" will simply get ignored, and line buffering doesn't work at all.

Example under 2.7:

$ python -c 'import subprocess; p = subprocess.Popen(["/bin/cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=1, universal_newlines=True); p.stdin.write("foo\n"); print(p.stdout.readline()); p.stdin.close()'
foo

Under 3.4, the same line hangs in the p.stdout.readline() call.

[pitrou]

Ok, the "not working at all" part is issue bpo-21332. Let's make this issue specific to binary mode, again.

[4kir4]

Until the current patch for issue bpo-21332 is committed; bufsize=1 is
equivalent to bufsize=-1 in both binary and text mode.

You are correct the patch in bpo-21332 fixes only the text mode
(universal_newlines=True) -- it also updates the documentation
to mention that bufsize=1 works only in text mode.