sys.stderr should be line-buffered when stderr is not a TTY

[pitrou]

BPO	13601
Nosy	@gvanrossum, @ncoghlan, @pitrou, @vstinner, @pjenvey, @benjaminp, @Bluehorn, @serhiy-storchaka, @miss-islington, @jendrikseipp, @hauntsaninja
PRs	bpo-13601: always use line-buffering for sys.stderr #17646 bpo-13601: Mention stderr's line buffering in What's New #20168

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-01.22:22:26.851>
created_at = <Date 2011-12-14.13:11:31.801>
labels = ['interpreter-core', 'type-bug', 'library', '3.9', 'expert-IO']
title = 'sys.stderr should be line-buffered when stderr is not a TTY'
updated_at = <Date 2020-05-18.05:08:00.186>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2020-05-18.05:08:00.186>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-01.22:22:26.851>
closer = 'pitrou'
components = ['Interpreter Core', 'Library (Lib)', 'IO']
creation = <Date 2011-12-14.13:11:31.801>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 13601
keywords = ['patch']
message_count = 21.0
messages = ['149447', '149454', '149560', '149561', '149887', '149889', '254027', '254028', '261852', '261853', '261916', '296303', '296655', '304064', '358593', '359171', '359172', '359527', '369004', '369159', '369177']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'gjb1002', 'ncoghlan', 'pitrou', 'vstinner', 'pjenvey', 'benjamin.peterson', 'stutzbach', 'torsten', 'serhiy.storchaka', 'miss-islington', 'jendrik', 'hauntsaninja']
pr_nums = ['17646', '20168']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13601'
versions = ['Python 3.9']

[pitrou]

In bpo-13597, Philip Jenvey points out:

“I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately”

Recent changes to the IO stack should allow stderr to be opened in fully unbuffered mode (and open(..., 'w', buffering=0) can be allowed too). Or at least it could be always line-buffered, even when redirected to a file.

[gvanrossum]

I *thought* I mimicked what C stdio did ~20 years ago... I'd be happy to follow what it does today if it changed or if I made a mistake.

That said, IMO:

Line-buffering should be good enough since in practice errors messages are always terminated by a newline.

I'm hesitant to make it line-buffered by default when directed to a file, since this could significantly slow down a program that for some reason produces super-voluminous output (e.g. when running a program with heavy debug logging turned on).

Maybe we need better command-line control to override the defaults? Are there precedents e.g. in Bash flags?

[pitrou]

Line-buffering should be good enough since in practice errors messages
are always terminated by a newline.

What I think too.

I'm hesitant to make it line-buffered by default when directed to a
file, since this could significantly slow down a program that for some
reason produces super-voluminous output (e.g. when running a program
with heavy debug logging turned on).

The slow-down is impressive in relative terms (6x) but the timings are
still small in absolute value:

$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=4096)" "f.write('log message\n')"
10000000 loops, best of 3: 0.156 usec per loop
$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=1)" "f.write('log message\n')"
1000000 loops, best of 3: 0.961 usec per loop

[pitrou]

Oops, I forgot the last two questions:

Maybe we need better command-line control to override the defaults?

We already have -u to switch all stdio to unbuffered. This issue proposes to make stderr line-buffered/unbuffered by default, since it's less surprising than fully buffered.

Are there precedents e.g. in Bash flags?

Well, man bash doesn't appear to say anything about stdio buffering.

[gjb1002]

I'm hesitant to make it line-buffered by default when directed to a
file, since this could significantly slow down a program that for some
reason produces super-voluminous output (e.g. when running a program
with heavy debug logging turned on).

Is that really the purpose of standard error though? Heavy debug output, in my experience, is usually sent to standard output or to another file.

Also, did anyone ever complain about this as a problem, given it is the default behaviour of Python 2?

In my view the requirements of seeing errors when they happen, and guaranteeing that they will always be seen no matter what happens afterwards, should weigh more heavily than this.

[gjb1002]

I think we all agree line-buffering is sufficient, so I change the title.

[terryjreedy]

As I read this, there was agreement that the status quo is sufficient. That would imply that this should be closed. Correct?

[terryjreedy]

A month after this discussion, the flush keyword was added to print, which cover partial lines sent to either stdout or stderr via print. https://bugs.python.org/issue13761

[ncoghlan]

This question came up today in the context of the final line of a traceback output potentially being missing from stderr if the closing flush of the standard streams is missed for any reason.

That's not going to be a common scenario (as far as I know it was an entirely hypothetical discussion), but the last line of a traceback is the one with the actual error message, so it's likely to be annoyingly cryptic if it does happen.

[ncoghlan]

Changing the target version and summarising my understanding of the status quo:

"python3": sys.stderr is line buffered at both the TextIOWrapper layer and may be fully buffered at the binary BufferedWriter layer if the output is redirected to a file
"python3 -u": the BufferedWriter layer is omitted entirely, leaving only the line buffering at the TextIOWrapper layer

Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

That means either line buffering or no buffering is considered acceptable, but full buffering is not.

So, at the very least, it seems to me that the way we configure stderr should be the same regardless of whether or not "-u" is used: omit the BufferedWriter layer.

Given that POSIX allows compliant implementations to use line-buffering on stderr, the "missing trailing newline and no implicit flush() is ever triggered" scenario is probably obscure enough not to worry about.