cursor shape (e.g. for vim mode)

[davidszotten]

hey, thanks for an awesome library!

would it be possible to support cursor shapes, like https://hamberg.no/erlend/posts/2014-03-09-change-vim-cursor-in-iterm.html (or http://vim.wikia.com/wiki/Change_cursor_shape_in_different_modes)

[jonathanslenders]

Hey @davidszotten,

That is probably possible, yes. Although I need to have a look how to implement this cleanly [1] and making sure that most terminals understand it. It's not a priority for me right now, but I'll put it on the to-do list.

Jonathan

[1] The tricky part is that the Vi state (insert/navigation) is currently not available from the CLI/renderer and the state is not able to access the renderer. So, I'm not sure yet how to make it fit cleanly in the architecture.

[sassanh]

@jonathanslenders What about dirty temporary workarounds for endusers? It's not healthy for vim users to work with wrong cursor shape 😉
Is there anything like a signal fired when vim mode is changed so that we can print the appropriate character sequence to terminal that it changes the cursor shape? I know the character sequences that change cursor shape for my terminal, so if we have those signals and a way to print to terminal when those signals are fired we can have temporary workarounds for our setups.

[jonathanslenders]

Hi @sassanh, I understand that this is useful. But so far I did not had time yet (actually it wasn't a priority). If you have a patch, then I'm more than happy to review it. Otherwise, I'll have a look at this later on myself. Probably it needs to go somewhere in the Renderer class, but I'm not sure what's the best approach.

[jonathanslenders]

Other question: is there somewhere some official documentation about escape sequences for cursor shapes? Which terminals support it? Do all terminals use the same escape sequences? And if not, how do I know which escape sequences to use?

[sassanh]

@jonathanslenders I understand, no rush. Just wanted to know if I can use a temporary hacky solution to acheive this till you have time to implement it.
When I was using vim (not neovim) I had some vimscript in my vimrc file to change the cursor shape for iTerm in vim as vim didn't do it by itself on iTerm. After upgrading to neovim I removed those lines as neovim handles it by itself. But neovim is the only application I know that handles it by itself. I had to customize readline configuration to achieve this for bash/ipython<5/psql/etc. I had to customize fish for this purpose too. So I guess it's totally alright for prompt-toolkit to let the user do it by himself and just gives the user the ability to print required sequences when special events occur in prompt-toolkit. I don't expect it to handle it by itself as even bash didn't handle it. Giving the user the ability to print some text to terminal when special events occur in prompt-line may be helpful for other situations too. User will be able to change color of things, print bell sequence, etc.

[sassanh]

It may be helpful: http://vim.wikia.com/wiki/Change_cursor_shape_in_different_modes

[joelostblom]

This would be really useful! Especially since the python-prompt-toolkit has such great support for multiline commands (thanks so much for that!), there is now more incentive to use vi-mode for command editing.

In terms of escape sequences, the snippet below is what I am using for cursor shape changes in zsh for gnome-terminal and terminator (there are more approaches for zsh listed over at the Unix & Linux SE.).

bindkey -v
# Remove delay when entering normal mode (vi)
KEYTIMEOUT=5
# Change cursor shape for different vi modes.
function zle-keymap-select {
  if [[ $KEYMAP == vicmd ]] || [[ $1 = 'block' ]]; then
    echo -ne '\e[1 q'
  elif [[ $KEYMAP == main ]] || [[ $KEYMAP == viins ]] || [[ $KEYMAP = '' ]] || [[ $1 = 'beam' ]]; then
    echo -ne '\e[5 q'
  fi
}
zle -N zle-keymap-select
# Start with beam shape cursor on zsh startup and after every command.
zle-line-init() { zle-keymap-select 'beam'}

The escape sequences I am using here (\e[1 q and \e[5 q) are not universal for all terminal emulator (this blog suggest that it is \E]50;CursorShape=0\C-G and \E]50;CursorShape=1\C-G for iTerm2). So the most flexible approach might be if there is a signal emitted when the vi-mode changes, similar to what @sassanh suggested, and then leave it up to the user to specify in their configuration file the appropriate escape sequence to be inserted for their terminal.

Edit: I asked this on SO just before finding this issue

[amzyang]

I've made a quick and dirty change. It works well for me (on iterm2).

diff --git i/prompt_toolkit/eventloop/base.py w/prompt_toolkit/eventloop/base.py
index db86fac..c9e2173 100644
--- i/prompt_toolkit/eventloop/base.py
+++ w/prompt_toolkit/eventloop/base.py
@@ -9,7 +9,7 @@ __all__ = (
 
 
 #: When to trigger the `onInputTimeout` event.
-INPUT_TIMEOUT = .5
+INPUT_TIMEOUT = .05
 
 
 class EventLoop(with_metaclass(ABCMeta, object)):
diff --git i/prompt_toolkit/key_binding/vi_state.py w/prompt_toolkit/key_binding/vi_state.py
index 92ce3cb..81dd6d0 100644
--- i/prompt_toolkit/key_binding/vi_state.py
+++ w/prompt_toolkit/key_binding/vi_state.py
@@ -1,3 +1,4 @@
+from __future__ import print_function
 from __future__ import unicode_literals
 
 __all__ = (
@@ -59,3 +60,18 @@ class ViState(object):
         self.waiting_for_digraph = False
         self.operator_func = None
         self.operator_arg = None
+
+    @property
+    def input_mode(self):
+        return self._input_mode
+
+    @input_mode.setter
+    def input_mode(self, mode):
+        cursor_shape = '\x1b[5 q'
+        if mode == InputMode.NAVIGATION:
+            cursor_shape = '\x1b[1 q'
+        elif mode == InputMode.REPLACE:
+            cursor_shape = '\x1b[3 q'
+
+        print(cursor_shape, end='')
+        self._input_mode = mode

[sassanh]

@grassofhust Thanks. Now I'm happy too.

[kovidgoyal]

And here is code you can add to your ipython_config.py to avoid needing to patch prompt_toolkit

# Set vi keybindings
import sys
from prompt_toolkit.key_binding.vi_state import InputMode, ViState


def get_input_mode(self):
    return self._input_mode


def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 1, InputMode.REPLACE: 3}.get(mode, 5)
    out = getattr(sys.stdout, 'buffer', sys.stdout)
    out.write(u'\x1b[{} q'.format(shape).encode('ascii'))
    sys.stdout.flush()
    self._input_mode = mode


ViState._input_mode = InputMode.INSERT
ViState.input_mode = property(get_input_mode, set_input_mode)
c.TerminalInteractiveShell.editing_mode = 'vi'

[kovidgoyal]

@jonathanslenders The vast majority of terminals support the escape code of the form <ESC>[<number><space>q for changing cursor shapes (the most glaring exception I know of on linux is konsole). The ones that dont should really be fixed to support it. It is documented here: http://invisible-island.net/xterm/ctlseqs/ctlseqs.html (see the q CSI escape code)

[kovidgoyal]

And here is modified code for ipython_config.py that also allows mapping of jj to exit insert mode:

# Set vi keybindings
import sys
from operator import attrgetter
from prompt_toolkit.key_binding.vi_state import InputMode, ViState
import prompt_toolkit.key_binding.defaults as pt_defaults
from prompt_toolkit.filters.cli import ViInsertMode
from prompt_toolkit.keys import Keys


def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 1, InputMode.REPLACE: 3}.get(mode, 5)
    out = getattr(sys.stdout, 'buffer', sys.stdout)
    out.write(u'\x1b[{} q'.format(shape).encode('ascii'))
    sys.stdout.flush()
    self._input_mode = mode


ViState._input_mode = InputMode.INSERT
ViState.input_mode = property(attrgetter('_input_mode'), set_input_mode)
orig_bindings_func = pt_defaults.load_vi_bindings


def load_vi_bindings(*a, **kw):
    registry = orig_bindings_func(*a, **kw)
    esc_handler = registry.get_bindings_for_keys((Keys.Escape,))[-1].handler
    # Use jj to exit insert mode
    registry.add_binding('j', 'j', filter=ViInsertMode())(esc_handler)
    return registry

pt_defaults.load_vi_bindings = load_vi_bindings
c.TerminalInteractiveShell.editing_mode = 'vi'

[sassanh]

The only thing that remains for me is preserving the input mode, currently new prompt line always comes in insert mode, but I want it to preserve the input mode from last line. Do you have any idea for that?

[kovidgoyal]

No idea, as I dont need this, I have not spent the time to look, but probably this belongs in ipython not prompt_toolkit as presumably state between consecutive prompts is not stored in prompt toolkit but in whatever program is calling it.

[kovidgoyal]

And here is an updated set_input_mode() function for recent changes to prompt toolkit:

def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 1, InputMode.REPLACE: 3}.get(mode, 5)
    raw = u'\x1b[{} q'.format(shape)
    if hasattr(sys.stdout, '_cli'):
        out = sys.stdout._cli.output.write_raw
    else:
        out = sys.stdout.write
    out(raw)
    sys.stdout.flush()
    self._input_mode = mode

[max-sixty]

Thanks @kovidgoyal !

If you want non-blinking cursors:

def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 2, InputMode.REPLACE: 4}.get(mode, 6)  # <-- changes here
    raw = u'\x1b[{} q'.format(shape)
    if hasattr(sys.stdout, '_cli'):
        out = sys.stdout._cli.output.write_raw
    else:
        out = sys.stdout.write
    out(raw)
    sys.stdout.flush()
    self._input_mode = mode

[max-sixty]

A couple of issues:

1. There's some delay between hitting the esc key and the cursor changing - do other people see this? I don't have the same delay in my normal terminal (fish).
   Is it anything to do with this variable? https://github.com/jonathanslenders/python-prompt-toolkit/blob/ed79b83d5d6a35ce6af412aacabe1f3f34409c30/prompt_toolkit/application/application.py#L197

2. If I quit ipython in insert mode, the cursor in the terminal stays as the insert cursor. Is there any way to reset the cursor back to its original state on exiting ipython?

[kovidgoyal]

I dont notice a delay, but I use jj not esc. Processing solitary esc in terminal is not easy, because there is no way to distinguish between a single esc and the start of an escape sequence, until the next key is pressed.

And I use an insert mode cursor in my shell as well, so I dont care that the cursor is not restored on quit, but it should be trivial to fix that. Use atexit to wriite the shape reset escape code to stdout when ipython quits. ipython/prompt-toolkit probably have dedicated exit callbacks that you can use, but I dont have th etime to go spelunking in the code to find them right now.

[jonathanslenders]

@maxim-lian, keep in mind that IPython still uses prompt_toolkit 1.0. They will upgrade in one of the following releases. The snippet that you have there is from prompt_toolkit 2.0. But as @kovidgoyal say, processing the escape key without delay is tricky.

I hope to come back to this issues soon so that we can provide a clean way in prompt_toolkit for defining cursor shapes.

[petobens]

But as @kovidgoyal say, processing the escape key without delay is tricky.

Hi @jonathanslenders! Before upgrading to 2.0 I used the following binding to exit insert mode without delay:

 def vi_movement_mode(event):
    buffer = event.current_buffer
    vi_state = event.cli.vi_state
    vi_state.reset(InputMode.NAVIGATION)
    if bool(buffer.selection_state):
        buffer.exit_selection()
 r.add_binding(
     'j', 'j', filter=vi_insert_mode, eager=True
 )(lambda ev: vi_movement_mode(ev))

On 2.0 I'm now using

def vi_movement_mode(event):
    event.cli.key_processor.feed(KeyPress(Keys.Escape))

But I do notice a delay. Is there an equivalent of my first vi_movement_mode function that works (without delay) on 2.0? Thanks in advance

[denis-savran]

@petobens I reduced the vi mode change delay by reducing input flush timeout to 10ms.

ipython_config.py

import sys

from prompt_toolkit.key_binding.vi_state import InputMode, ViState


def get_input_mode(self):
    if sys.version_info[0] == 3:
        from prompt_toolkit.application.current import get_app

        app = get_app()
        # Decrease input flush timeout from 500ms to 10ms.
        app.ttimeoutlen = 0.01
        # Decrease handler call timeout from 1s to 250ms.
        app.timeoutlen = 0.25

    return self._input_mode


def set_input_mode(self, mode):
    shape = {InputMode.NAVIGATION: 2, InputMode.REPLACE: 4}.get(mode, 6)
    cursor = "\x1b[{} q".format(shape)

    if hasattr(sys.stdout, "_cli"):
        write = sys.stdout._cli.output.write_raw
    else:
        write = sys.stdout.write

    write(cursor)
    sys.stdout.flush()

    self._input_mode = mode


ViState._input_mode = InputMode.INSERT
ViState.input_mode = property(get_input_mode, set_input_mode)

# Shortcut style to use at the prompt. 'vi' or 'emacs'.
c.TerminalInteractiveShell.editing_mode = "vi"
c.TerminalInteractiveShell.prompt_includes_vi_mode = False

I call get_app from get_input_mode getter function to get ipython's prompt_toolkit.application.Application instance and then change its ttimeoutlen. Doing this in global scope has no effect.

My ipython_config.py.

[NilsIrl]

@kovidgoyal I tried your method but there is a problem: nav and ins still appear at the left of the prompt

[marskar]

@blaxpy, your ipython_config.py is great, but I needed to set both ttimeoutlen and timeoutlen to 0.01 to get rid of the cursor shape change delay from ins to nav:

        app = get_app()
        app.ttimeoutlen = 0.01
        app.timeoutlen = 0.01

It may just be me, perhaps because I have caps lock mapped to Control when held and Esc when tapped. IPython vi mode works really great for me now, both in the terminal and in vs code.

[max-sixty]

@marskar thanks for adding to the thread — that does improve the cursor delay.

But with those changes (I think it's app.timeoutlen = 0.01), cc and dd don't seem to work. Do you see the same effect?

[denis-savran]

@max-sixty Yes, it breaks those shortcuts. Set timeoutlen to 0.25.

[marskar]

Thanks @max-sixty and @blaxpy, indeed setting timeoutlen too low breaks cc and dd (and also f, F, and yy), but interestingly not other shortcuts that start with c or d, e.g. cw, caw, ciw, dw, daw, diw.

In order to use the aforementioned shortcuts, I set timeoutlen to 0.2, but I think 0.25 will be better for most people.

I want timeoutlen to be as low as possible, because I have shortcuts that use alt (escape).
If timeoutlen is set too high, there is a lag when switching from ins to nav mode and sometimes pressing escape then another key is captured as an alt shortcut.

Update: I found that it was hard to use combos like daW, ciB, ct}, etc. with a timeoutlen of 0.2 or 0.25, so I set it to 0.4. It is hard to balance exiting insert mode quickly and providing enough time for combos.

[micimize]

With ipython v8 it seems this happens automatically – not sure if that is due to a change in prompt toolkit itself.

For esc flush speed, I found ipython sets it to 0.1, but emacs bindings can mess things up considerably.
Relevant config:

# ~/.ipython/profile_default/ipython_config.py
from prompt_toolkit.key_binding.vi_state import InputMode, ViState

c.TerminalInteractiveShell.editing_mode = 'vi'
c.TerminalInteractiveShell.modal_cursor = False
c.TerminalInteractiveShell.emacs_bindings_in_vi_insert_mode = False

[ltrujello]

I've traced the following issue I and others are experiencing in IPython v8 to this thread, as it looks like this commit in IPython here might be causing it, and that code came from this thread.

Basically, running IPython changes the cursor from a block to a beam, even outside of IPython itself. This replaces the cursor with a beam in the terminal prompt and even in other programs like Vim. I'll keep looking for a fix but I mention this since someone here is probably more equipped to know the cause and fix for this.

[kovidgoyal]

There are various bugs in that patch, does no one review code in the Ipython project?

1. They are using 6 instead of 5 which gives a steady bar instead of a blinking bar
2. They need to restore the cursor to 1 (blinking block, the default cursor shape) when:
   a) running an external command
   b) quitting ipython

[ltrujello]

Thanks Kovid, I guess they don't review code. That commit I linked is a pretty bad case of someone ctrl+c ctrl+v-ing code they clearly didn't understand into a widely used project and it not getting filtered. I'm trying to implement your suggestions to fix this bug.

Does anyone know what this part of the buggy code is doing? I'll paste it here below too.

    """
    I do not understand the following if branch.
    """
    if hasattr(sys.stdout, "_cli"):  # What's this for?
        write = sys.stdout._cli.output.write_raw
    else:
        write = sys.stdout.write

    ...

Clearly it's for displaying the cusor but in what case does sys.stdout possibly have an attribute _cli?

That is definitely not part of the standard library, so I did various grep calls on the prompt-toolkit and IPython repositories but nothing came up.

I do however notice that there's a method write_raw which comes from this abstract base class that some other classes inherit. So I assume there is a real reason for that if-branch but I cannot find the connection. Maybe it was once relevant to an older commit of prompt-toolkit?

[kovidgoyal]

I'm not familiar enough with prompt_tookit to tell you for sure, but if I had to
guess it would likely be that prompt_toolkit overrides sys.stdout,
perhaps to filter escape codes, so this code tries to detect that and
bypass it.

[kovidgoyal]

Oh and I was slightly off about (2) above, it needs to be restored to 0 not 1. Some terminals allow users to override the default cursor shape and 0 is supposed to set to that while 1 sets to blinking block always.

[ltrujello]

Great, thanks again Kovid, I seem to have resolved it with echoing \x1b[0 q in sys.stdout and I made a pull request to IPython.

[jonathanslenders]

I've been adding native support for cursor shapes in prompt_toolkit: #1558
Ideally, IPython should switch over to that. (If I have time, I can look into creating a PR, but no promises yet.)

[Systemhlp]

hi

[Freed-Wu]

#1558 has been merged. So this issue can be closed?