Interactive interpreter doesn't flush stderr prompty

[marczellm]

BPO	21425
Nosy	@ncoghlan, @pitrou, @tjguk, @skrah, @davidmalcolm, @zware, @eryksun
Files	flush_stderr.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-12.09:30:01.001>
created_at = <Date 2014-05-04.08:28:40.048>
labels = ['interpreter-core', 'type-bug']
title = "Interactive interpreter doesn't flush stderr prompty"
updated_at = <Date 2014-06-12.09:30:01.000>
user = 'https://bugs.python.org/marczellm'

bugs.python.org fields:

activity = <Date 2014-06-12.09:30:01.000>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-12.09:30:01.001>
closer = 'berker.peksag'
components = ['Interpreter Core']
creation = <Date 2014-05-04.08:28:40.048>
creator = 'marczellm'
dependencies = []
files = ['35206']
hgrepos = []
issue_num = 21425
keywords = ['patch']
message_count = 19.0
messages = ['217861', '218218', '218219', '218252', '218261', '218262', '218264', '218265', '218266', '218267', '218269', '218271', '218272', '218274', '218276', '218277', '218280', '218281', '218282']
nosy_count = 11.0
nosy_names = ['ncoghlan', 'pitrou', 'tim.golden', 'Arfrever', 'skrah', 'dmalcolm', 'python-dev', 'zach.ware', 'eryksun', 'David.Edelsohn', 'marczellm']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21425'
versions = ['Python 3.4', 'Python 3.5']

[marczellm]

When I run a Python 3.3.4 prompt inside Emacs 24.3 on Windows 7, correct commands are evaluated immediately, but incorrect ones are delayed (I have to press Enter one more time), as seen below:

    >>> 1
    1
    >>> nonsense
    >>>
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    NameError: name 'nonsense' is not defined

Python 2 does not do this. I've filed an Emacs bug report (http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17304) and got the response to try this on the command line (where cat.exe is from an MSYS installation):

python 2>&1 | cat.exe

and it behaves the same way as in Emacs.

[pitrou]

This can be reproduced as easily under Linux:

$ ./python 2>&1 | cat
Python 3.5.0a0 (default:17689e43839a+39f2a78f4357+, May  9 2014, 00:30:19) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 5
5
>>> 1/0
>>> nonsense
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>> 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'nonsense' is not defined
>>>

[pitrou]

Attached patch fixes it under Linux, and adds tests.
I'd be glad if someone could give it a run under Windows.

[eryksun]

The patch fixes python 2>&1 | cat.exe for me on Windows, where cat.exe is from GnuWin32. The tests also pass. It looks like failing would entail hanging until a timeout.

[python-dev]

New changeset ab3e012c45d0 by Antoine Pitrou in branch '3.4':
Issue bpo-21425: Fix flushing of standard streams in the interactive interpreter.
http://hg.python.org/cpython/rev/ab3e012c45d0

New changeset d1c0cf44160c by Antoine Pitrou in branch 'default':
Issue bpo-21425: Fix flushing of standard streams in the interactive interpreter.
http://hg.python.org/cpython/rev/d1c0cf44160c

[pitrou]

Thanks for the report. This should now be fixed in 3.4 and 3.5.

[pitrou]

Apparently the new tests hang on the "Fedora without threads" buildbot.
Stefan, could you take a look?

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.4/builds/123

[pitrou]

Also on the PowerLinux buildbot:
http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/113/

[pitrou]

Judging by the faulthandler traceback in the PowerLinux build:

Timeout (1:00:00)!
Thread 0x0000008074e46670 (most recent call first):
File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 192 in interactive_python
File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/contextlib.py", line 59 in __enter__
File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 205 in check_repl_stderr_flush
File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 220 in test_repl_stderr_flush
[snip]

... the test is actually stuck trying to read the interactive prompt from stderr:

            # Drain stderr until prompt
            while True:
                data = stderr.read(4)
                if data == b">>> ":
                    break
                stderr.readline()            <-- HERE

[python-dev]

New changeset ffae7aad9dfa by Antoine Pitrou in branch 'default':
Add debugging output for bpo-21425
http://hg.python.org/cpython/rev/ffae7aad9dfa

[pitrou]

So, it seems like under Fedora (or PowerLinux), the Python interactive prompt produces different byte contents on stderr:

[ 21/389] test_cmd_line_script
repl stderr[:4]: b'Pyth'
repl stderr[:4]: b'[GCC'
repl stderr[:4]: b'Type'
repl stderr[:4]: b'\x1b[?1'

I wonder if it's because of GNU readline?

[pitrou]

This looks like a well-known issue on Fedora (and Red Hat?):
http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html

Apparently using XTERM=vt100 would workaround the issue.
(I can't reproduce under Ubuntu)

[python-dev]

New changeset e57718ac8ff8 by Antoine Pitrou in branch 'default':
Try workaround for test issues in bpo-21425
http://hg.python.org/cpython/rev/e57718ac8ff8

[pitrou]

According to https://bugzilla.redhat.com/show_bug.cgi?id=880393, this has actually been reported to us as bpo-19884.

[pitrou]

Workarounds seem to work.

[python-dev]

New changeset 58e2116576cf by Antoine Pitrou in branch '3.4':
Try workaround for test issues in bpo-21425
http://hg.python.org/cpython/rev/58e2116576cf

[pitrou]

Workaround actually broke tests under shared library builds, because LD_LIBRARY_PATH isn't forwarded anymore. Will try another fix :-(

[python-dev]

New changeset 5a1b2566d68e by Antoine Pitrou in branch 'default':
Try to fix issue bpo-21425 workaround for shared library builds
http://hg.python.org/cpython/rev/5a1b2566d68e

[python-dev]

New changeset 974c0718c7e0 by Antoine Pitrou in branch '3.4':
Try to fix issue bpo-21425 workaround for shared library builds
http://hg.python.org/cpython/rev/974c0718c7e0