Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys.stderr should be line-buffered when stderr is not a TTY #57810

Closed
pitrou opened this issue Dec 14, 2011 · 21 comments
Closed

sys.stderr should be line-buffered when stderr is not a TTY #57810

pitrou opened this issue Dec 14, 2011 · 21 comments
Labels
3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir topic-IO type-bug An unexpected behavior, bug, or error

Comments

@pitrou
Copy link
Member

pitrou commented Dec 14, 2011

BPO 13601
Nosy @gvanrossum, @ncoghlan, @pitrou, @vstinner, @pjenvey, @benjaminp, @Bluehorn, @serhiy-storchaka, @miss-islington, @jendrikseipp, @hauntsaninja
PRs
  • bpo-13601: always use line-buffering for sys.stderr #17646
  • bpo-13601: Mention stderr's line buffering in What's New #20168
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-01-01.22:22:26.851>
    created_at = <Date 2011-12-14.13:11:31.801>
    labels = ['interpreter-core', 'type-bug', 'library', '3.9', 'expert-IO']
    title = 'sys.stderr should be line-buffered when stderr is not a TTY'
    updated_at = <Date 2020-05-18.05:08:00.186>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2020-05-18.05:08:00.186>
    actor = 'miss-islington'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-01-01.22:22:26.851>
    closer = 'pitrou'
    components = ['Interpreter Core', 'Library (Lib)', 'IO']
    creation = <Date 2011-12-14.13:11:31.801>
    creator = 'pitrou'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 13601
    keywords = ['patch']
    message_count = 21.0
    messages = ['149447', '149454', '149560', '149561', '149887', '149889', '254027', '254028', '261852', '261853', '261916', '296303', '296655', '304064', '358593', '359171', '359172', '359527', '369004', '369159', '369177']
    nosy_count = 13.0
    nosy_names = ['gvanrossum', 'gjb1002', 'ncoghlan', 'pitrou', 'vstinner', 'pjenvey', 'benjamin.peterson', 'stutzbach', 'torsten', 'serhiy.storchaka', 'miss-islington', 'jendrik', 'hauntsaninja']
    pr_nums = ['17646', '20168']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue13601'
    versions = ['Python 3.9']

    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 14, 2011

    In bpo-13597, Philip Jenvey points out:

    “I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately”

    Recent changes to the IO stack should allow stderr to be opened in fully unbuffered mode (and open(..., 'w', buffering=0) can be allowed too). Or at least it could be always line-buffered, even when redirected to a file.

    @pitrou pitrou added interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir topic-IO type-bug An unexpected behavior, bug, or error labels Dec 14, 2011
    @gvanrossum
    Copy link
    Member

    I *thought* I mimicked what C stdio did ~20 years ago... I'd be happy to follow what it does today if it changed or if I made a mistake.

    That said, IMO:

    Line-buffering should be good enough since in practice errors messages are always terminated by a newline.

    I'm hesitant to make it line-buffered by default when directed to a file, since this could significantly slow down a program that for some reason produces super-voluminous output (e.g. when running a program with heavy debug logging turned on).

    Maybe we need better command-line control to override the defaults? Are there precedents e.g. in Bash flags?

    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 15, 2011

    Line-buffering should be good enough since in practice errors messages
    are always terminated by a newline.

    What I think too.

    I'm hesitant to make it line-buffered by default when directed to a
    file, since this could significantly slow down a program that for some
    reason produces super-voluminous output (e.g. when running a program
    with heavy debug logging turned on).

    The slow-down is impressive in relative terms (6x) but the timings are
    still small in absolute value:

    $ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=4096)" "f.write('log message\n')"
    10000000 loops, best of 3: 0.156 usec per loop
    $ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=1)" "f.write('log message\n')"
    1000000 loops, best of 3: 0.961 usec per loop

    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 15, 2011

    Oops, I forgot the last two questions:

    Maybe we need better command-line control to override the defaults?

    We already have -u to switch all stdio to unbuffered. This issue proposes to make stderr line-buffered/unbuffered by default, since it's less surprising than fully buffered.

    Are there precedents e.g. in Bash flags?

    Well, man bash doesn't appear to say anything about stdio buffering.

    @gjb1002
    Copy link
    Mannequin

    gjb1002 mannequin commented Dec 19, 2011

    I'm hesitant to make it line-buffered by default when directed to a
    file, since this could significantly slow down a program that for some
    reason produces super-voluminous output (e.g. when running a program
    with heavy debug logging turned on).

    Is that really the purpose of standard error though? Heavy debug output, in my experience, is usually sent to standard output or to another file.

    Also, did anyone ever complain about this as a problem, given it is the default behaviour of Python 2?

    In my view the requirements of seeing errors when they happen, and guaranteeing that they will always be seen no matter what happens afterwards, should weigh more heavily than this.

    @gjb1002
    Copy link
    Mannequin

    gjb1002 mannequin commented Dec 19, 2011

    I think we all agree line-buffering is sufficient, so I change the title.

    @gjb1002 gjb1002 mannequin changed the title sys.stderr should be unbuffered (or always line-buffered) sys.stderr should always be line-buffered Dec 19, 2011
    @terryjreedy
    Copy link
    Member

    As I read this, there was agreement that the status quo is sufficient. That would imply that this should be closed. Correct?

    @terryjreedy
    Copy link
    Member

    A month after this discussion, the flush keyword was added to print, which cover partial lines sent to either stdout or stderr via print. https://bugs.python.org/issue13761

    @ncoghlan
    Copy link
    Contributor

    This question came up today in the context of the final line of a traceback output potentially being missing from stderr if the closing flush of the standard streams is missed for any reason.

    That's not going to be a common scenario (as far as I know it was an entirely hypothetical discussion), but the last line of a traceback is the one with the actual error message, so it's likely to be annoyingly cryptic if it does happen.

    @ncoghlan
    Copy link
    Contributor

    Changing the target version and summarising my understanding of the status quo:

    "python3": sys.stderr is line buffered at both the TextIOWrapper layer and may be fully buffered at the binary BufferedWriter layer if the output is redirected to a file
    "python3 -u": the BufferedWriter layer is omitted entirely, leaving only the line buffering at the TextIOWrapper layer

    Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

    That means either line buffering or no buffering is considered acceptable, but full buffering is not.

    So, at the very least, it seems to me that the way we configure stderr should be the same regardless of whether or not "-u" is used: omit the BufferedWriter layer.

    Given that POSIX allows compliant implementations to use line-buffering on stderr, the "missing trailing newline and no implicit flush() is ever triggered" scenario is probably obscure enough not to worry about.

    @vstinner vstinner changed the title sys.stderr should always be line-buffered sys.stderr should be line-buffered when stderr is not a TTY Mar 17, 2016
    @vstinner
    Copy link
    Member

    I changed the title, since sys.stderr is already line-buffered when stderr is a TTY.

    @serhiy-storchaka
    Copy link
    Member

    See also bpo-28647 and bpo-30404.

    @pitrou
    Copy link
    Member Author

    pitrou commented Jun 22, 2017

    Amusingly, I didn't realize I had already opened this issue when one of our users hit it recently which led me to add TextIOWrapper.reconfigure(): https://bugs.python.org/issue30526

    Still, I think it would be a good idea to do this as well (switch sys.stderr to line-buffered unconditionally), especially now that Nick found a POSIX reference that states C stderr should never be fully buffered.

    @pitrou pitrou added the 3.7 (EOL) end of life label Jun 22, 2017
    @Bluehorn
    Copy link
    Mannequin

    Bluehorn mannequin commented Oct 10, 2017

    Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

    I was of the impression that this is defined in ISO C already. Unfortunately, I only have ISO C 99 at hand, but this clearly states in section 7.19.3 (Files), enumeration item 7:

    As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer
    to an interactive device.

    I am quite sure this is just as as it was in the original ANSI C standard.

    @jendrikseipp
    Copy link
    Mannequin

    jendrikseipp mannequin commented Dec 17, 2019

    I took the liberty of increasing the target version. It would be great if someone could review my patch for this issue at #17646 .

    @jendrikseipp jendrikseipp mannequin added 3.9 only security fixes and removed 3.7 (EOL) end of life labels Dec 17, 2019
    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 1, 2020

    New changeset 5b90771 by Antoine Pitrou (Jendrik Seipp) in branch 'master':
    bpo-13601: always use line-buffering for sys.stderr (GH-17646)
    5b90771

    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 1, 2020

    Jendrik, thank you for fixing this!

    @pitrou pitrou closed this as completed Jan 1, 2020
    @vstinner
    Copy link
    Member

    vstinner commented Jan 7, 2020

    So it just took 9 years to fix this old bug :-)

    @hauntsaninja
    Copy link
    Contributor

    I'm wondering if this should be mentioned in Python 3.9's What's New, potentially at https://docs.python.org/3.9/whatsnew/3.9.html#sys ?

    This change broke one of mypy's tests on 3.9 and it was a little tricky to find what had changed.

    @gvanrossum
    Copy link
    Member

    Can you submit a PR and CC me?

    @miss-islington
    Copy link
    Contributor

    New changeset d17f3d8 by Shantanu in branch 'master':
    bpo-13601: Mention stderr's line buffering in What's New (GH-20168)
    d17f3d8

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir topic-IO type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    8 participants