Issues with Vim and comprehensive keyboard handling

[comex]

In short: When running Vim on Kitty on macOS, if you input Cmd-<anything> (well, anything that isn't a Kitty shortcut), then for the rest of the Vim session, Ctrl-<anything> keystrokes will have their Vim-default behavior, ignoring any mappings the user has set up for them in Vim.

Sounds weird, right?

This results from the combination of two different issues related to comprehensive keyboard handling. Vim supports, and enables by default, xterm's variant of it, aka "modifyOtherKeys" (relevant vim documentation).

The first issue is that Vim's default sequence for enabling modifyOtherKeys is "CSI > 4 ; 2 m". Kitty tries to be compatible with this xterm feature, but it only recognizes "CSI > 4 ; 1 m". According to xterm documentation (this part plus this part), both of those sequences enable reporting of modifier keys, but they differ in which modifier keys are reported.

Since Kitty doesn't recognize "CSI > 4 ; 2 m", it ignores it and leaves comprehensive keyboard handling disabled. This leads to the second issue, which could arguably be considered either a Vim bug or a Kitty bug:

Kitty documents that "Key events that could not be represented in legacy mode are encoded using a CSI u escape code, that most terminal programs should just ignore." However, Vim doesn't ignore them. Whenever it sees any such escape code, it sets a global boolean seenModifyOtherKeys (regardless of whether it tried to enable modifyOtherKeys in the first place). The effect of this variable is to make the mapping lookup code ignore legacy representations for Ctrl (e.g. 0x01 => Ctrl-A) and Alt (e.g. 0xc0 => Alt-A), permanently until the user quits Vim. (This functionality was added in 2019 in this commit.)

To Reproduce
I reproduced this with the latest master of Vim (be01090ef) and Kitty (b1375e5).

1. Run kitty -c /dev/null
2. In the terminal, run vim --noplugin -u /tmp/my-vimrc, where /tmp/my-vimrc should contain:

set nocompatible
map <C-a> aGot C-a<ESC>

3. Type Ctrl-A; the text "Got C-a" is added to the buffer.
4. Type Cmd-F (or any other keystroke that Kitty will represent using a CSI u code).
5. Type Ctrl-A again; no text is added to the buffer.

[kovidgoyal]

Just add CSI > 4 ; 1 m to t_ti in your vimrc and you should be fine. This really needs to be fixed in vim, however since the chances of vim doing the sensible thing arent very high, I suppose I can have 4;2m also work the same as 4;1m in kitty.

[page-down]

I recently discovered an anomaly with vim. On vim 8.2 without any configuration, pressing ESC to exit Insert mode undoes all input.

# press ESC
1 change; before #1  0 seconds ago
# ESC
Already at oldest change

Then I found this recent changes.

After troubleshooting and rolling back this change, ESC worked fine in vim again.

Before this, at least the ESC worked, after the change, the user has to configure t_ti for each environment before the ESC key can be used properly in vim, which seems a bit difficult.

[kovidgoyal]

Hmm that sucks, I didnt notice because I dont use ESC in vim. This is
why we cant have nice things. I guess all that's left to do is disable
the xterm compatibility escapes totally. If software wants to use
extended keys in kitty they will need to ask for them using kitty's
escape codes. I believe kakoune currently uses the xterm escape codes to
turn on this mode in kitty as well. So heads up for @mawww you should
update kakoune if that is indeed the case.

[comex]

Hello,

d6a43a7 brought back the original issue, where Vim does not successfully enable comprehensive keyboard handling, but after receiving a 'CSI u' sequence thinks it has, causing Ctrl to stop working properly.

I also found the cause of the ESC issue (maybe @kovidgoyal already realized this). The only 'CSI u' sequences that Vim recognizes are, to quote the source code:

    // Key with modifier:
    //  {lead}27;{modifier};{key}~
    //  {lead}{key};{modifier}u

If it's just (to use Vim's terminology) {lead}{key}u without any modifier, Vim won't recognize it and will pass the whole CSI sequence through unchanged. In the case of pressing the escape key, the CSI sequence is " [ 2 7 u" (without spaces). Vim interprets this as pressing the escape key (exit insert mode), then the command [2 (cursor to 2nd previous unmatched #if, #else, or #ifdef), then the command 7u (undo 7 times)!

This could easily be fixed on Vim's end and probably should be. On the other hand, Vim's current behavior is not completely bogus. xterm itself never sends 'CSI u' sequences without modifiers, and even the name of the option that Vim enables ("modifyOtherKeys") suggests that it's about modifiers. So it's reasonable to be unprepared for sequences without modifiers.

If it's worth supporting existing versions of Vim, I think there are two options:

1. Support the xterm sequence, but instead of just making it a synonym for comprehensive keyboard handling, make it enable a different mode where only keystrokes with modifiers are sent using CSI sequences.
2. Don't support the xterm sequence, and don't send 'CSI u' sequences if comprehensive keyboard handling isn't enabled, even for keystrokes like Cmd-<key> which have no other representation.

[kovidgoyal]

Yeah but that issue is a lot better than ESC not working. And IMO this needs to be fixed in vim. I dont particularly care to implement a whole new, and completely inferior, keyboard mode just for vim, that I will then have to carry around in kitty's code forever. There is absolutely no reason it should be enabling modifyOtherKeys on seeing CSI u. Nowhere is xterm's (admittedly completely inadequate) docs does it suggest applications do that. I am guessing the original motivation for it was that xterm stupidly makes their modifyOtherKeyboard mode user controllable rather than application controllable. So sending the XTMODKEYS escape code to turn it on is completely meaningless with no defined effect.

As for not sending CSI u for keys that have no representation, that just means those keys become unusable everywhere without adding support for the full mode. Which punishes the entire ecosystem because of one badly behaving program.