Re-opening /dev/tty breaks readline

[silvioricardoc]

BPO	29396
Nosy	@vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-01-31.10:53:07.633>
labels = ['type-feature', 'library']
title = 'Re-opening /dev/tty breaks readline'
updated_at = <Date 2017-09-09.00:57:11.088>
user = 'https://bugs.python.org/silvioricardoc'

bugs.python.org fields:

activity = <Date 2017-09-09.00:57:11.088>
actor = 'martin.panter'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-01-31.10:53:07.633>
creator = 'silvioricardoc'
dependencies = []
files = []
hgrepos = []
issue_num = 29396
keywords = []
message_count = 4.0
messages = ['286521', '301519', '301574', '301757']
nosy_count = 2.0
nosy_names = ['silvioricardoc', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29396'
versions = ['Python 3.4']

[silvioricardoc]

The following code works on python2 (tested with 2.7.6), but breaks readline for python3 (tested with 3.4.3):

>> import sys, readline
>> sys.stdin = open('/dev/tty', 'r')
>> input("> ") # press "7<Enter>"
>> input("> ") # press "up" key

Re-opening /dev/tty as stdin is a very useful technique. For example, one could debug a script that takes a file as stdin by re-opening the stdin as /dev/tty and using pdb.set_trace().

See bug report for python2 in 2002 here: https://bugs.python.org/issue512981

[vadmium]

I think the difference between Python 2 and 3 here is that Python 2’s file objects, including sys.stdin, wrap C library FILE objects, which is supported by the Readline library. However Python 3 has its own kind of file objects, independent of standard C and Readline. Python 3 only uses Readline if sys.stdin corresponds to the original C stdin FILE object.

Perhaps Python 3 could support Readline with other file objects (or at least file descriptors), but I think that would be a new feature.

[silvioricardoc]

So, if I understood correctly, the readline module only works for an unmodified sys.stdin, which is implemented in terms of C FILE* structures. Anything created by open will not be implemented in terms of C FILE*, and so all history and completion features of readline will be (silently) disabled.

Is there any workaround to make open (or some other file-opening function) return a file object wrapping a C FILE*?

Maybe this behavior should be documented in the readline documentation? (https://docs.python.org/3/library/readline.html)

[vadmium]

I agree it would be good to document when the Readline library is invoked. Yes, the “readline‭” module is only designed to work with the original sys.stdin and sys.stdout. Python’s “open” function does not use <stdio.h> FILE objects, but Python does use <stdio.h> FILE objects internally for its Readline hook, and passes them to the Readline library. Python falls back to dumber implementations, which may not implement history nor completion, when it decides not to use Readline.

The “readline” module is implemented in Modules/readline.c and uses “rl_instream” <https://cnswww.cns.cwru.edu/php/chet/readline/readline.html#IDX228\> and “rl_outstream” to specify <stdio.h> FILE objects that the Readline library uses. These FILE objects are passed through the PyOS_ReadlineFunctionPointer hook <https://docs.python.org/3.4/c-api/veryhigh.html#c.PyOS_ReadlineFunctionPointer\> from the PyOS_Readline function in Parser/myreadline.c. They are required to be terminals (checked with the Posix “isatty” call).

The implementation of Python’s “input” function is in the “builtin_input” C function in Python/bltinmodule.c. Before calling PyOS_Readline, it requires that sys.stdin.fileno() and sys.stdout.fileno() match the <stdio.h> stdin and stdout FILE objects. It also does its own isatty checks.

You might have some luck calling “fopen” with the “ctypes” module, or writing your own Readline module, but it wouldn’t be straightforward. You might be able to fool the check by reopening file descriptors 0 and 1, but that seems rather hacky. Or you might rely on the OS to provide history and/or completion (I think the Windows console does this in a limited way), or an external Readline wrapper program (e.g. search for “rlwrap”).

[smemsh]

in bltinmodule.c function builtin_input_impl() it says:

we should only use (GNU) readline if Python's sys.stdin and sys.stdout are the same as C's stdin and stdout, because we need to pass it to those

and goes on to check fd == fileno(stdin) && isatty(fd), added in eba7696 to "make [input] behave properly with the new I/O library."

So it's really only checking that sys.stdin.fileno() matches C's fileno(stdin). You can get away with closing stdin fd and then reopening. sys.stdin.close() does not seem to release the descriptor to the system, so os.close() is needed.

This construct seems to work to debug cli filters (and retain readline):

# save stdin, pdb needs stdio fds itself
if select([sys.stdin], [], [], None)[0]:
    inbuf = sys.stdin.read() # todo: problematic with large inputs
    os.close(sys.stdin.fileno())
    try: sys.stdin = open('/dev/tty')
    except: pass # no ctty, but then pdb would not be in use
else:
    # non-filter use case, ie open files as "inbuf"

I think maybe Python should only check isatty() on the descriptor when deciding about readline, and not care if it happens to be fd 0. Supporting termios should be enough for readline to function properly.

As such, this issue should potentially be reclassified as a bug, not a feature request.