readline windows issues.

[melund]

It is has been a while since I used the readline shell on windows. I only switch to it when I need speed.

Now it is giving me some issues:

If I press tab when it launches, the 'Display all ## possibilities' question shows up with ansi escapes:

If I press n I get this error:

Readline internal error
Traceback (most recent call last):
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\console\console.py", line 768, in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 571, in readline
    self._readline_from_keyboard()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 536, in _readline_from_keyboard
    if self._readline_from_keyboard_poll():
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 556, in _readline_from_keyboard_poll
    result = self.mode.process_keyevent(event.keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 243, in process_keyevent
    r = self.process_keyevent_queue[-1](keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 286, in _process_keyevent
    r = dispatch_func(keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\basemode.py", line 257, in complete
    completions = self._get_completions()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\basemode.py", line 200, in _get_completions
    r = self.completer(ensure_unicode(text), i)
  File "C:\Users\mel\Anaconda3\lib\cmd.py", line 275, in complete
    self.completion_matches = compfunc(text, line, begidx, endidx)
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 18735, in completedefault
    show_completions = self._querycompletions(completions, endidx - begidx)
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 18695, in _querycompletions
    rl_on_new_line()
TypeError: 'NoneType' object is not callable

If I pressy i get an other error:

Readline internal error
Traceback (most recent call last):
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\console\console.py", line 768, in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 571, in readline
    self._readline_from_keyboard()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 536, in _readline_from_keyboard
    if self._readline_from_keyboard_poll():
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 556, in _readline_from_keyboard_poll
    result = self.mode.process_keyevent(event.keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 243, in process_keyevent
    r = self.process_keyevent_queue[-1](keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 286, in _process_keyevent
    r = dispatch_func(keyinfo)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\basemode.py", line 257, in complete
    completions = self._get_completions()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\modes\basemode.py", line 200, in _get_completions
    r = self.completer(ensure_unicode(text), i)
  File "C:\Users\mel\Anaconda3\lib\cmd.py", line 275, in complete
    self.completion_matches = compfunc(text, line, begidx, endidx)
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 18735, in completedefault
    show_completions = self._querycompletions(completions, endidx - begidx)
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 18698, in _querycompletions
    lines = columnize(completions, width=w)
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 7124, in columnize
    nrows = nelem // ncols
ZeroDivisionError: integer division or modulo by zero

[melund]

Just a follow-up. If I issue a clear command first, the prompt doesn't show ansi escapes. But the error messages are the same.

[melund]

Another issue with the readline shell. If I excute a command like ls it first prints the output, but then enters a infinite loop printing this:

Traceback (most recent call last):
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\console\console.py", line 768, in hook_wrapper_23
    res = ensure_str(readline_hook(prompt))
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 569, in readline
    self.readline_setup(prompt)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 565, in readline_setup
    self._print_prompt()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 466, in _print_prompt
    x, y = c.pos()
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\console\console.py", line 269, in pos
    self.GetConsoleScreenBufferInfo(self.hout, byref(info))
ctypes.ArgumentError: argument 2: <class 'TypeError'>: expected LP_CONSOLE_SCREEN_BUFFER_INFO instance instead of pointer to CONSOLE_SCREEN_BUFFER_INFO

[ghost]

Hey @melund! Does this happen on both cmd.exe and powershell?

[melund]

Hmm. I wouldn't have thought it made a difference since both powershell and cmd.exe use conhost.exe behind the scene. But it doesn't actually show the initial ansi escapes if launch from power shell.

I get the same trace-backs though when press y/n, as well as the readline error when running a command.

[melund]

Also, if I just start xonsh in readline mode and leave it for 30s I get the history error below:

Traceback (most recent call last):
  File "C:\Users\mel\Anaconda3\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Users\mel\Anaconda3\lib\site-packages\xonsh\__amalgam__.py", line 18968, in run
    if line == readline.get_history_item(i - 1):
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 154, in get_history_item
    return self.mode._history.get_history_item(index)
  File "C:\Users\mel\Anaconda3\lib\site-packages\pyreadline\lineeditor\history.py", line 52, in get_history_item
    item = self.history[index - 1]
IndexError: list index out of range

[ghost]

i think powershell added ansi escape codes(?)
i wonder what will bash use when it lands on windows10.

[melund]

@laerus It has nothing to do with PowerShell. The windows console (conhost.exe) which both PowerShell and cmd.exe use, has gotten full support for ANSI escapes in Windows 10.

This is also what Bash uses in the Windows Subsystem for Linux (WSL). I have suggested that PTK should just use ANSI escapes on windows :
prompt-toolkit/pymux#58

That would speed PTK a lot and provide true color support.

[ghost]

playing around on a windows10 machine with this script

import sys
sys.stdout.write('\x1b[2J')

it seems that PowerShell correctly clears the screen from the cursor up
but in cmd.exe it just prints the string.

[melund]

Yes. I think powershell does something to activate the features. Something that also kicks in when I for example call MSys tools from ls.exe or clear.exe tools from 'Git for Windows'

If you notice some of the pictures I posted. The prompt is rerendered in colors. That would work with the readline shell if conhost didn't interpret ansi escapes. I don't know exactly what is going on here.

[melund]

Maybe it is something with the codepage, and how output characters are interpreted.

[ghost]

adding this snippet from here :

import ctypes

kernel32 = ctypes.windll.kernel32
kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)

solves it!

[AstraLuma]

Has this been tested on Windows <10? They still exist.

[melund]

Has this been tested on Windows <10? They still exist.

The method @laerus suggest can be used to detect if Windows has the ansi escape features. I asked this question on microsoft/WSL#1356 last year. If the command fails we can assume it is an older (<10) version of Windows.

All the other issues aside we better ensure that @laerus's code snippet is always executed on Windows.

[scopatz]

In terms of integrating the fix, we do have a bit of a windows abstraction layer in winutils. https://github.com/xonsh/xonsh/blob/master/xonsh/winutils.py

[melund]

Where are they used? I guess there are options doing many of the same things (finding terminal sizes etc) with ansi escapes? These are some of the things that are now possible on Windows.

[scopatz]

@melund - what do you mean by where are they used?

In some ways this is a duplicate of #398 and so it would be nice if we could get the fix integrated and then close out that one too.

[scopatz]

Another issue with the readline shell. If I excute a command like ls it first prints the output, but then enters a infinite loop printing this:

@melund diving into this expected LP_CONSOLE_SCREEN_BUFFER_INFO instance instead of pointer to CONSOLE_SCREEN_BUFFER_INFO one a bit shows that it is actually a bug in pyreadline, see pyreadline/pyreadline#21. The good news is that it has been fixed (pyreadline/pyreadline#24). The bad news is that it hasn't made it into a release yet. I am not sure what we can do about this other than perhaps update the conda-forge package with a patch that fixes it.

[scopatz]

Also worth noting that the fix to pyreadline went in in 2014

[melund]

Well. They haven't even merged the fix. I also just realised that pyreadline is no longer bundled with Anaconda by default. It really seems everyone is abandoning ship.

[scopatz]

Yeah, it does seem that way. I believe that pyreadline's flakiness was one of the reasons for jupyter/ipython abandoning readline. I think this is probably not xonsh issue, but we can maybe patch it in conda-forge.

[melund]

Closed by #2208