Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

abort when stderr is closed #51360

Closed
petere mannequin opened this issue Oct 12, 2009 · 26 comments
Closed

abort when stderr is closed #51360

petere mannequin opened this issue Oct 12, 2009 · 26 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@petere
Copy link
Mannequin

petere mannequin commented Oct 12, 2009

BPO 7111
Nosy @loewis, @doko42, @amauryfa, @abalkin, @pitrou, @benjaminp, @naufraghi, @petere
Files
  • nostdio.patch
  • nostdio2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2011-11-28.18:22:26.989>
    created_at = <Date 2009-10-12.08:23:19.214>
    labels = ['interpreter-core', 'type-crash']
    title = 'abort when stderr is closed'
    updated_at = <Date 2011-11-28.18:22:26.987>
    user = 'https://github.com/petere'

    bugs.python.org fields:

    activity = <Date 2011-11-28.18:22:26.987>
    actor = 'pitrou'
    assignee = 'none'
    closed = True
    closed_date = <Date 2011-11-28.18:22:26.989>
    closer = 'pitrou'
    components = ['Interpreter Core']
    creation = <Date 2009-10-12.08:23:19.214>
    creator = 'petere'
    dependencies = []
    files = ['23788', '23794']
    hgrepos = []
    issue_num = 7111
    keywords = ['patch']
    message_count = 26.0
    messages = ['93891', '93917', '93919', '93946', '93950', '93969', '93971', '94425', '127773', '127775', '127803', '127809', '127811', '127812', '127813', '127814', '127815', '127817', '127821', '148425', '148449', '148460', '148461', '148505', '148507', '148508']
    nosy_count = 13.0
    nosy_names = ['loewis', 'doko', 'exarkun', 'amaury.forgeotdarc', 'belopolsky', 'pitrou', 'benjamin.peterson', 'stutzbach', 'naufraghi', 'Arfrever', 'petere', 'neologix', 'python-dev']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue7111'
    versions = ['Python 3.2', 'Python 3.3']

    @petere
    Copy link
    Mannequin Author

    petere mannequin commented Oct 12, 2009

    bash$ python3.1 -c 'pass' 2>&-
    Aborted (core dumped)

    (I verified, the core dump belongs to python.)

    If you remove the redirection thingy at the end, it works.

    Not sure why I ever wrote that code, but it has been working since
    forever up to python3.0.

    @petere petere mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump labels Oct 12, 2009
    @benjaminp
    Copy link
    Contributor

    The problem is the check_fd in _fileio.c checks fstat of 2, which
    returns EBADFD. I'm not sure what about this redirection makes it a bad
    file descriptor, though..

    @stutzbach
    Copy link
    Mannequin

    stutzbach mannequin commented Oct 13, 2009

    After some searching around with Google, "2>&-" means "close file
    descriptor 2" (i.e., standard error).

    @pitrou
    Copy link
    Member

    pitrou commented Oct 13, 2009

    Please note that normally an error message is output, but of course it
    doesn't display since stderr is invalid :-)

    It's clearer if you close stdout instead:

    $ ./python -c 'pass' >&-
    Fatal Python error: Py_Initialize: can't initialize sys standard streams
    OSError: [Errno 9] Bad file descriptor
    Abandon

    If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
    what the semantics should be. Should sys.std{in, out, err} be None? Or a
    file object which always throws an error?

    Under Python 2.x, you don't get a crash but the behaviour is quite
    unhelpful anyway:

    $ python -c 'print 1' >&-
    close failed in file object destructor:
    Error in sys.excepthook:

    Original exception was:

    @stutzbach
    Copy link
    Mannequin

    stutzbach mannequin commented Oct 14, 2009

    Is it even possible to portably test the validity of a file descriptor
    without trying to write/read it?

    When I first saw this bug, my gut feeling was "well, don't do that
    then!" However, I then recalled that Windows GUI applications have no
    stdin, stdout, or stderr.

    Python 2 will raise IOError: Bad File Descriptor when the user tries to
    write to stdout or stderr (more accurately, it raises the exception when
    trying to flush data to the file descriptor).

    I just tested pythonw.exe. If I set sys.stderr by hand to a file, then
    write to sys.stdout, 2.6 will correctly write the exception to the file.
    3.1 exits silently.

    @amauryfa
    Copy link
    Member

    3.1 exits silently.
    Did you use "print"? pythonw.exe 3.1 sets sys.stdout to None.
    if you use sys.stdout.write, you get an exception. But print() silently
    does nothing if the file is None.

    @petere
    Copy link
    Mannequin Author

    petere mannequin commented Oct 14, 2009

    For what it's worth, the code in question is used here (using "import
    distutils" instead of "pass"):

    http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/config/python.m4?rev=1.15;content-type=text%2Fx-cvsweb-markup

    This is obviously a completely gratuitous variant on 2>/dev/null, but it
    has apparently been working since forever. I'll probably go make the
    change anyway.

    Nevertheless, I think Python shouldn't core dump. It may choose to exit
    doing nothing (useful) if it doesn't want to deal with this case.

    Check this for possible alternative behaviors:

    $ ls 1>&-
    ls: write error: Bad file descriptor
    ($? = 2)
    $ ls 1>&- 2>&-
    ($? = 2, no output)

    @naufraghi naufraghi mannequin changed the title core dump when stderr is moved stdout closed Oct 24, 2009
    @naufraghi
    Copy link
    Mannequin

    naufraghi mannequin commented Oct 24, 2009

    sorry, title restored!

    @naufraghi naufraghi mannequin changed the title stdout closed core dump when stderr is moved Oct 24, 2009
    @benjaminp benjaminp changed the title core dump when stderr is moved abort when stderr is moved Oct 24, 2009
    @abalkin
    Copy link
    Member

    abalkin commented Feb 2, 2011

    If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
    what the semantics should be. Should sys.std{in, out, err} be None? Or a
    file object which always throws an error?

    I would say it should be a *pseudo*-file object which always throws a *descriptive* error. Note that setting sys.stdout to None makes print() do nothing rather than report an error:

    >> sys.stdout = None
    >> print('abc')

    See also bpo-6501.

    @abalkin
    Copy link
    Member

    abalkin commented Feb 2, 2011

    On the second thought, as long as python used fd 2 as the "message stream of last resort", we should probably not allow it to run with fd 2 closed. The problem is that in this case fd 2 may become associated with a very important file contents of which you don't want to see replaced with a python error message.

    @abalkin abalkin changed the title abort when stderr is moved abort when stderr is closed Feb 2, 2011
    @stutzbach
    Copy link
    Mannequin

    stutzbach mannequin commented Feb 3, 2011

    That's an interesting point.

    Do you know of places where we use fd 2 instead of sys.stderr?

    @pitrou
    Copy link
    Member

    pitrou commented Feb 3, 2011

    That's an interesting point.

    Do you know of places where we use fd 2 instead of sys.stderr?

    We normally don't. One reason is that buffering inside sys.stderr can
    make ordering of output incorrect. There are some places in C code where
    we do "fprintf(stderr, ...)" but that's for specialized debugging
    (disabled in normal builds) or fatal error messages.

    @abalkin
    Copy link
    Member

    abalkin commented Feb 3, 2011

    On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
    ..

    > Do you know of places where we use fd 2 instead of sys.stderr?

    We normally don't. One reason is that buffering inside sys.stderr can
    make ordering of output incorrect. There are some places in C code where
    we do "fprintf(stderr, ...)" but that's for specialized debugging
    (disabled in normal builds) or fatal error messages.

    This is the case that I had in mind. What does non-debug build do on
    a fatal error? Also, can we be sure that Python does not call C
    library functions that write to stderr behind the scenes? If vanilla
    Python is safe to run with closed fd 2, that may not be the case for
    3rd party extensions. What is the use case for "python >&-"? Is
    it important enough to justify the risk of accidental data loss?

    @abalkin
    Copy link
    Member

    abalkin commented Feb 3, 2011

    On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
    ..
    > Do you know of places where we use fd 2 instead of sys.stderr?

    We normally don't.

    Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
    Are you sure these are all debug-build only?

    @pitrou
    Copy link
    Member

    pitrou commented Feb 3, 2011

    > We normally don't. One reason is that buffering inside sys.stderr can
    > make ordering of output incorrect. There are some places in C code where
    > we do "fprintf(stderr, ...)" but that's for specialized debugging
    > (disabled in normal builds) or fatal error messages.

    This is the case that I had in mind. What does non-debug build do on
    a fatal error?

    It uses fprintf(stderr, ...). That's the only thing it can do (there's
    no way sys.stderr is guaranteed to be usable at that point). If C stderr
    is invalid, then too bad.

    Also, can we be sure that Python does not call C
    library functions that write to stderr behind the scenes?

    I think you can guess the answer :)

    What is the use case for "python >&-"? Is
    it important enough to justify the risk of accidental data loss?

    I don't think so. One more important use case is when running a Unix
    daemon, which has (AFAIK) to close all std handles. I don't know how
    that interacts with using C stderr, especially if the handle closing is
    done in Python (and therefore only calls C close() and not fclose()!).

    Perhaps we should provide a sys function to fclose() C std{in,out,err}.

    @pitrou
    Copy link
    Member

    pitrou commented Feb 3, 2011

    Le jeudi 03 février 2011 à 19:59 +0000, Alexander Belopolsky a écrit :

    Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:

    > On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
    > ..
    >> Do you know of places where we use fd 2 instead of sys.stderr?
    >
    > We normally don't.

    Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
    Are you sure these are all debug-build only?

    "grep -C2" seems to say most of them are. I haven't done a survey.

    @stutzbach
    Copy link
    Mannequin

    stutzbach mannequin commented Feb 3, 2011

    On Thu, Feb 3, 2011 at 11:56 AM, Alexander Belopolsky
    <report@bugs.python.org> wrote:

    3rd party extensions.    What is the use case for "python >&-"?    Is
    it important enough to justify the risk of accidental data loss?

    I don't think closing stderr via the command line is an important use
    case, but pythonw.exe and Unix daemon processes are important use
    cases.

    @pitrou
    Copy link
    Member

    pitrou commented Feb 3, 2011

    I don't think so. One more important use case is when running a Unix
    daemon, which has (AFAIK) to close all std handles.

    I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
    uses dup2() to redirect standard streams, which is far nicer.

    @stutzbach
    Copy link
    Mannequin

    stutzbach mannequin commented Feb 3, 2011

    On Thu, Feb 3, 2011 at 12:18 PM, Antoine Pitrou <report@bugs.python.org> wrote:

    I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
    uses dup2() to redirect standard streams, which is far nicer.

    I'm more worried about the case where a daemon launches python.

    At startup, could we check that 2 and 3 are valid file descriptors,
    and, if not, open /dev/null? That way, they cannot later be
    inadvertently assigned to some other file?

    @pitrou
    Copy link
    Member

    pitrou commented Nov 26, 2011

    Attached patch allows Python to run even if no standard stream is available. I use dup() to detect whether a fd is valid.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Nov 27, 2011

    (No Rietveld link):

    +is_valid_fd(int fd)
    [...]
    + dummy_fd = dup(fd);
    + if (dummy_fd < 0)
    + return 0;
    + close(dummy_fd);

    Why not use fstat() instead (does Windows have fstat()? And dup()?).

    + @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
    + def test_no_stdin(self):

    It would maybe be more direct with skipUnless(os.name == 'posix').

    Finally, it's not that important, but it could maybe be possible to factorize the code, i.e. make a helper function that takes a list of streams and defines the preexec() function and code to test those streams, and then just call:

    def test_no_stdin(self):
        out, err = self._test_with_closed_streams(['stdin'])
        [...]
    
    def test_no_streams(self):
        out, err = self._test_with_closed_streams(['stdin', 'stdout', 'stderr'])
        [...]

    @pitrou
    Copy link
    Member

    pitrou commented Nov 27, 2011

    +is_valid_fd(int fd)
    [...]

    • dummy_fd = dup(fd);
    • if (dummy_fd < 0)
    •    return 0;
      
    • close(dummy_fd);

    Why not use fstat() instead (does Windows have fstat()? And dup()?).

    Windows has dup(), but no fstat().

    • @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
    • def test_no_stdin(self):

    It would maybe be more direct with skipUnless(os.name == 'posix').

    Hmm, indeed.

    Finally, it's not that important, but it could maybe be possible to
    factorize the code, i.e. make a helper function that takes a list of
    streams and defines the preexec() function and code to test those
    streams, and then just call:

    Ah, indeed perhaps.

    @pitrou
    Copy link
    Member

    pitrou commented Nov 27, 2011

    Updated patch.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Nov 28, 2011

    Updated patch.

    LGTM.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Nov 28, 2011

    New changeset f15943505db0 by Antoine Pitrou in branch '3.2':
    Issue bpo-7111: Python can now be run without a stdin, stdout or stderr stream.
    http://hg.python.org/cpython/rev/f15943505db0

    New changeset c86fb10eaf68 by Antoine Pitrou in branch 'default':
    Issue bpo-7111: Python can now be run without a stdin, stdout or stderr stream.
    http://hg.python.org/cpython/rev/c86fb10eaf68

    @pitrou
    Copy link
    Member

    pitrou commented Nov 28, 2011

    Thanks, committed.

    @pitrou pitrou closed this as completed Nov 28, 2011
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants