An article on Python 3 stdout and stderr output buffering

[abitrolly]

BPO	41449
Nosy	@gvanrossum, @abitrolly, @tedder

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-08-03.04:33:06.195>
created_at = <Date 2020-07-31.10:10:32.510>
labels = ['docs']
title = 'An article on Python 3 stdout and stderr output buffering'
updated_at = <Date 2020-08-03.18:04:39.743>
user = 'https://github.com/abitrolly'

bugs.python.org fields:

activity = <Date 2020-08-03.18:04:39.743>
actor = 'gvanrossum'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-08-03.04:33:06.195>
closer = 'gvanrossum'
components = ['Documentation']
creation = <Date 2020-07-31.10:10:32.510>
creator = 'abitrolly'
dependencies = []
files = []
hgrepos = []
issue_num = 41449
keywords = []
message_count = 9.0
messages = ['374633', '374706', '374714', '374716', '374719', '374720', '374721', '374722', '374749']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'docs@python', 'abitrolly', 'tedder']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41449'
versions = []

[abitrolly]

It is hard to find info why Python 3 buffers stdout/stderr. The buffering causes problems when debugging Python apps in Docker and Kubernetes, and it is unclear if it is Python 3 who starts to buffer stdout if no tty is attached, it is Docker, or it is Kubernetes.

The only bit of info that could be searched is the description of -u option https://docs.python.org/3.8/using/cmdline.html?#cmdoption-u which is not linked to any article.

The -u description also says.

Changed in version 3.7: The text layer of the stdout and stderr streams now is unbuffered.

However, I don't understand what is the text layers of stdout. And there is no description of behaviour when the output is not attached, and when the output is redirected.

[gvanrossum]

Maybe you’ll get some explanation or pointers on python-list. Or were you volunteering to write something?

[abitrolly]

Not sure I can volunteer - need to find a sustenance ASAP. But given that nothing moved in this direction over the past 6 months, another week spent not writing this stuff won't change anything.

In any case it would start with the following problem - "Logs don't show up if running in Kubernetes" - bottlepy/bottle#1130

I've encountered it a 3 days session on adding CI to my favourite cheat sheet server chubin/cheat.sh@10bfaab and I am still not sure I am right. That's why I want to read something about it.

[gvanrossum]

I don't think that waiting another six months is going to make this happen either. Maybe you can petition realpython.com to publish something about this.

Honestly it sounds like this ties too many different topics together to easily fit in the library docs: in order to make a large number of different use cases easy, how I/O works under the hood in Python is truly complex.

So I'm going to close this rather than have it add to our issue count.

[abitrolly]

To avoid too much outside reference, the commit message I am unsure of is this.

Run Python 3 in unbuffered mode to see lines in docker logs as soon as they appear. When Python 3 is not attached to a terminal, it turns on the buffering, like when docker runs in a background.

I am not sure.

1. How Python detects it is attached to terminal or not.
2. That it is Python that turns on stdout buffering.
3. What happens to stdout with child processes when they run with docker in different modes - foreground, background (-d), interactive (-i), with tty (-t) and without tty.

[gvanrossum]

I don't know anything about Docker, so I can't help you there. Please don't reopen the issue again.

[abitrolly]

I didn't reopen the issue explicitly. Just pressed the submit button again when the notification about edited bug appeared.

[abitrolly]

The point is that without a clear description of buffering in Python 3, it is impossible to understand what goes when Python is wrapped by Docker, supervisord, redirected to file etc.

[gvanrossum]

If you don't even know whether it's Python, Docker or Kubernetes you really need to ask somewhere else. Plenty of user groups around.

Python detects tty using the standard UNIX isatty() function.

To a tty we always get line buffering.

To a file we use a larger buffer.

For more details see the source at

cpython/Python/pylifecycle.c

Line 1942 in 4660597

init_sys_streams(PyThreadState *tstate)