Codes printed to terminal when using typeahead with FinalTerm sequences

[asmeurer]

Consider the finalterm-shell-integration.py example. Make the following change to it

--- a/examples/finalterm-shell-integration.py
+++ b/examples/finalterm-shell-integration.py
@@ -8,7 +8,7 @@ from __future__ import unicode_literals
 from prompt_toolkit import prompt
 from prompt_toolkit.token import Token
 import sys
-
+import time

 BEFORE_PROMPT = '\033]133;A\a'
 AFTER_PROMPT = '\033]133;B\a'
@@ -29,8 +29,9 @@ def get_prompt_tokens(cli):


 if __name__ == '__main__':
-    answer = prompt(get_prompt_tokens=get_prompt_tokens)
-
-    sys.stdout.write(BEFORE_OUTPUT)
-    print('You said: %s' % answer)
-    sys.stdout.write(AFTER_OUTPUT.format(command_status=0))
+    while True:
+        answer = prompt(get_prompt_tokens=get_prompt_tokens)
+        time.sleep(2)
+        sys.stdout.write(BEFORE_OUTPUT)
+        print('You said: %s' % answer)
+        sys.stdout.write(AFTER_OUTPUT.format(command_status=0))

Then run the script and type

hello
hello
hello
hello

several times without waiting for the prompt. Eventually, you'll see something like this:

Say something: # hello
^[[45;1Rello
You said: hello

The ^[[45;1R is printed before the output. I don't understand what that escape code is exactly. It doesn't look like one of the FinalTerm codes. I believe the final term codes themselves are being printed correctly, as the "select output" feature in iTerm2 works correctly (it selects everything, including the ^[[45;1R).

[asmeurer]

(Unrelated note, but as it's an example, you should have sys.stdout.flush() after the sys.stdout.write(BEFORE_OUTPUT) call. Without this, in a regular shell situation, the output timing feature will not work correctly.)

[jonathanslenders]

The escape sequence, ^[[45;1R is a response to a CPR (cursor position request). The only way for prompt_toolkit to know the cursor position is by writing a CPR to stdout. The terminal emulator will parse that sequence and send the x/y coordinates of the cursor back to the stdin of the application.

The situation that you have right here is that the CPR reply is probably sent with a small delay, actually after prompt_toolkit finished reading for input. The result is that this CPR reply is sent to stdin at the point where the terminal is back into ECHO mode. This means that whatever is send as input to stdin, will be echoed back to stdout.

I'm not entirely sure what's going on. It can have to do with timing, but it can also be iterm2 doing something slightly different. In any case, I think we can handle this better. (But it's tricky).

[asmeurer]

This seems to happen almost every time if instead of spamming some command you send an input in programmatically with a PipeInput().

[asmeurer]

As far as I can tell this is fixed in 2.0. I think somewhere between the new typeahead code and the better CPR handling it got fixed (I can bisect if you are interested).

I think I figured out the issue with PipeInput specifically (see #610 (comment)). Basically the stdout Output class assumes the input can respond to a CPR request, and has no way to know if it is PipeInput. Perhaps PipeInput should get a faked CPR request responder to deal with this? The real problem seems to be the disconnect between Input and Output classes in the code, even though CPR requests and responses inherently link the two (and require they both be a real TTY). At any rate, for now, I've come the conclusion that PipeInput should never be used for interactive purposes (only for things like unit tests). I'm still working on translating my code to 2.0 so I can't say yet if my issues fully exist there.

One thing I couldn't figure out though is what exactly the CPR responses even do. I commented out the CPR request code, and everything seemed to still act normal in my application. I also noticed that 2.0 has some code to automatically stop sending CPR codes if it notices they are not being responded to. So I'm curious exactly what I should expect to stop working if CPR codes are disabled. And if it's nothing I care about if I could just disable them myself.

[jonathanslenders]

The disconnect between input and output is indeed annoying. I wish there was a nicer solution, but it's the way it is. Output and input following different flows. For instance, if we consider Windows, then the output can be either VT100 escape sequences or Windows console API calls, while the input is coming through a separate path in the event loop. Then there is also this weirdness that CPR is something which has to do with output rendering, but the CPR response comes through stdin, completely asynchronous. Terminals are complex.

[max-sixty]

FYI I'm getting this issues, starting a couple of days ago. That could be upgrading IPython, iTerm, or one of their dependencies. It's intermittent, though i can trigger it by holding down the return key.

Other escape codes I see include ^[[68;1R^[[68;1R. I sometimes get them repeated, sometimes not.

[max-sixty]

For more info, my ipython_config.py consists of:

import sys
from prompt_toolkit.key_binding.vi_state import InputMode, ViState


def get_input_mode(self):
    return self._input_mode


def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 2, InputMode.REPLACE: 4}.get(mode, 6)
    raw = u'\x1b[{} q'.format(shape)
    if hasattr(sys.stdout, '_cli'):
        out = sys.stdout._cli.output.write_raw
    else:
        out = sys.stdout.write
    out(raw)
    sys.stdout.flush()
    self._input_mode = mode


ViState._input_mode = InputMode.INSERT
ViState.input_mode = property(get_input_mode, set_input_mode)
c.TerminalInteractiveShell.editing_mode = 'vi'

Partly from #192 (comment)