Make stdout and stderr truly unbuffered when using -u option

[serhiy-storchaka]

BPO	30404
Nosy	@pitrou, @vstinner, @benjaminp, @berkerpeksag, @serhiy-storchaka
PRs	bpo-30404: The -u option now makes the stdout and stderr streams unbuffered. #1667 bpo-28647: Update -u documentation #1655 bpo-28647: Update -u documentation after bpo-30404 #3961

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-04.17:26:10.442>
created_at = <Date 2017-05-19.15:52:05.216>
labels = ['interpreter-core', 'type-feature', '3.7', 'expert-IO']
title = 'Make stdout and stderr truly unbuffered when using -u option'
updated_at = <Date 2017-10-13.12:16:36.996>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-10-13.12:16:36.996>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-04.17:26:10.442>
closer = 'serhiy.storchaka'
components = ['Interpreter Core', 'IO']
creation = <Date 2017-05-19.15:52:05.216>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 30404
keywords = []
message_count = 9.0
messages = ['293960', '296315', '296316', '296344', '296345', '296352', '303716', '303717', '304332']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'benjamin.peterson', 'stutzbach', 'berker.peksag', 'serhiy.storchaka']
pr_nums = ['1667', '1655', '3961']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30404'
versions = ['Python 3.7']

[serhiy-storchaka]

In Python 2 when run the interpreter with the -u option the stdout and stderr streams are unbuffered. In Python 3 they become just line-buffered. This is because initially there was no way to create unbuffered text streams. But since Python 3.3 TextIOWrapper supports unbuffered output binary stream and accepts the write_through argument which switch off its own buffering.

Proposed patch makes the stdout and stderr streams truly unbuffered when run with the -u option.

[vstinner]

Oh, I like #1667

"If write_through is True, calls to write() are guaranteed not to be buffered: any data written on the TextIOWrapper object is immediately handled to its underlying binary buffer."

I didn't know write_through. It seems like it was introduced in Python 3.7: bpo-30526, commit 3c2817b.

[vstinner]

While I hope that users of the -u options expect the slow-down, would it be possible to benchmark it?

For example, try to write setup.py content character by character to stdout using -u or not, into a TTY, into a pipe and/or a file. I'm curious if the line buffering vs really unbuffered has a significant impact on performances.

[serhiy-storchaka]

Writing separate lines:

$ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
200 loops, best of 5: 1.07 msec per loop

$ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
Unpatched:  50 loops, best of 5: 5.89 msec per loop
Patched:    100 loops, best of 5: 3.32 msec per loop

Writing separate characters:

$ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
10 loops, best of 5: 30 msec per loop

$ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
Unpatched:  5 loops, best of 5: 49.2 msec per loop
Patched:    2 loops, best of 5: 137 msec per loop

[vstinner]

Hum. It has an huge impact on performances. Would it make sense to have two command line options to choose between unbuffered and line buffered?

The C setvbuf() function uses these constants:

          _IONBF unbuffered
          _IOLBF line buffered
          _IOFBF fully buffered

[serhiy-storchaka]

Note that if output by lines, the patch speeds up the output! Actually the output is fast enough with buffering and without. The only slowdown is exposed when output by characters, but this is uncommon case. And if you output by characters (in case of drawing a progressbar or like), then perhaps you want the characters been displayed immediately, without buffering.

See also bpo-13601.

[serhiy-storchaka]

If there is a need in making redirected stdout line-buffered, a new option can be added in separate issue.

[serhiy-storchaka]

New changeset 77732be by Serhiy Storchaka in branch 'master':
bpo-30404: The -u option now makes the stdout and stderr streams totally unbuffered. (bpo-1667)
77732be

[berkerpeksag]

New changeset 7f58097 by Berker Peksag in branch 'master':
bpo-28647: Update -u documentation after bpo-30404 (GH-3961)
7f58097