sys.stderr should be line-buffered when stderr is not a TTY

[pitrou]

BPO	13601
Nosy	@gvanrossum, @ncoghlan, @pitrou, @vstinner, @pjenvey, @benjaminp, @Bluehorn, @serhiy-storchaka, @miss-islington, @jendrikseipp, @hauntsaninja
PRs	bpo-13601: always use line-buffering for sys.stderr #17646 bpo-13601: Mention stderr's line buffering in What's New #20168

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-01.22:22:26.851>
created_at = <Date 2011-12-14.13:11:31.801>
labels = ['interpreter-core', 'type-bug', 'library', '3.9', 'expert-IO']
title = 'sys.stderr should be line-buffered when stderr is not a TTY'
updated_at = <Date 2020-05-18.05:08:00.186>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2020-05-18.05:08:00.186>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-01.22:22:26.851>
closer = 'pitrou'
components = ['Interpreter Core', 'Library (Lib)', 'IO']
creation = <Date 2011-12-14.13:11:31.801>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 13601
keywords = ['patch']
message_count = 21.0
messages = ['149447', '149454', '149560', '149561', '149887', '149889', '254027', '254028', '261852', '261853', '261916', '296303', '296655', '304064', '358593', '359171', '359172', '359527', '369004', '369159', '369177']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'gjb1002', 'ncoghlan', 'pitrou', 'vstinner', 'pjenvey', 'benjamin.peterson', 'stutzbach', 'torsten', 'serhiy.storchaka', 'miss-islington', 'jendrik', 'hauntsaninja']
pr_nums = ['17646', '20168']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13601'
versions = ['Python 3.9']

[pitrou]

In bpo-13597, Philip Jenvey points out:

“I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately”

Recent changes to the IO stack should allow stderr to be opened in fully unbuffered mode (and open(..., 'w', buffering=0) can be allowed too). Or at least it could be always line-buffered, even when redirected to a file.

[gvanrossum]

I *thought* I mimicked what C stdio did ~20 years ago... I'd be happy to follow what it does today if it changed or if I made a mistake.

That said, IMO:

Line-buffering should be good enough since in practice errors messages are always terminated by a newline.

I'm hesitant to make it line-buffered by default when directed to a file, since this could significantly slow down a program that for some reason produces super-voluminous output (e.g. when running a program with heavy debug logging turned on).

Maybe we need better command-line control to override the defaults? Are there precedents e.g. in Bash flags?

[pitrou]

Line-buffering should be good enough since in practice errors messages
are always terminated by a newline.

What I think too.

I'm hesitant to make it line-buffered by default when directed to a
file, since this could significantly slow down a program that for some
reason produces super-voluminous output (e.g. when running a program
with heavy debug logging turned on).

The slow-down is impressive in relative terms (6x) but the timings are
still small in absolute value:

$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=4096)" "f.write('log message\n')"
10000000 loops, best of 3: 0.156 usec per loop
$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=1)" "f.write('log message\n')"
1000000 loops, best of 3: 0.961 usec per loop

[pitrou]

Oops, I forgot the last two questions:

Maybe we need better command-line control to override the defaults?

We already have -u to switch all stdio to unbuffered. This issue proposes to make stderr line-buffered/unbuffered by default, since it's less surprising than fully buffered.

Are there precedents e.g. in Bash flags?

Well, man bash doesn't appear to say anything about stdio buffering.

[gjb1002]

I'm hesitant to make it line-buffered by default when directed to a
file, since this could significantly slow down a program that for some
reason produces super-voluminous output (e.g. when running a program
with heavy debug logging turned on).

Is that really the purpose of standard error though? Heavy debug output, in my experience, is usually sent to standard output or to another file.

Also, did anyone ever complain about this as a problem, given it is the default behaviour of Python 2?

In my view the requirements of seeing errors when they happen, and guaranteeing that they will always be seen no matter what happens afterwards, should weigh more heavily than this.

[gjb1002]

I think we all agree line-buffering is sufficient, so I change the title.

[terryjreedy]

As I read this, there was agreement that the status quo is sufficient. That would imply that this should be closed. Correct?

[terryjreedy]

A month after this discussion, the flush keyword was added to print, which cover partial lines sent to either stdout or stderr via print. https://bugs.python.org/issue13761

[ncoghlan]

This question came up today in the context of the final line of a traceback output potentially being missing from stderr if the closing flush of the standard streams is missed for any reason.

That's not going to be a common scenario (as far as I know it was an entirely hypothetical discussion), but the last line of a traceback is the one with the actual error message, so it's likely to be annoyingly cryptic if it does happen.

[ncoghlan]

Changing the target version and summarising my understanding of the status quo:

"python3": sys.stderr is line buffered at both the TextIOWrapper layer and may be fully buffered at the binary BufferedWriter layer if the output is redirected to a file
"python3 -u": the BufferedWriter layer is omitted entirely, leaving only the line buffering at the TextIOWrapper layer

Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

That means either line buffering or no buffering is considered acceptable, but full buffering is not.

So, at the very least, it seems to me that the way we configure stderr should be the same regardless of whether or not "-u" is used: omit the BufferedWriter layer.

Given that POSIX allows compliant implementations to use line-buffering on stderr, the "missing trailing newline and no implicit flush() is ever triggered" scenario is probably obscure enough not to worry about.

[vstinner]

I changed the title, since sys.stderr is already line-buffered when stderr is a TTY.

[serhiy-storchaka]

See also bpo-28647 and bpo-30404.

[pitrou]

Amusingly, I didn't realize I had already opened this issue when one of our users hit it recently which led me to add TextIOWrapper.reconfigure(): https://bugs.python.org/issue30526

Still, I think it would be a good idea to do this as well (switch sys.stderr to line-buffered unconditionally), especially now that Nick found a POSIX reference that states C stderr should never be fully buffered.

[Bluehorn]

Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

I was of the impression that this is defined in ISO C already. Unfortunately, I only have ISO C 99 at hand, but this clearly states in section 7.19.3 (Files), enumeration item 7:

As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.

I am quite sure this is just as as it was in the original ANSI C standard.

[jendrikseipp]

I took the liberty of increasing the target version. It would be great if someone could review my patch for this issue at #17646 .

[pitrou]

New changeset 5b90771 by Antoine Pitrou (Jendrik Seipp) in branch 'master':
bpo-13601: always use line-buffering for sys.stderr (GH-17646)
5b90771

[pitrou]

Jendrik, thank you for fixing this!

[vstinner]

So it just took 9 years to fix this old bug :-)

[hauntsaninja]

I'm wondering if this should be mentioned in Python 3.9's What's New, potentially at https://docs.python.org/3.9/whatsnew/3.9.html#sys ?

This change broke one of mypy's tests on 3.9 and it was a little tricky to find what had changed.

[gvanrossum]

Can you submit a PR and CC me?

[miss-islington]

New changeset d17f3d8 by Shantanu in branch 'master':
bpo-13601: Mention stderr's line buffering in What's New (GH-20168)
d17f3d8