Some mappings in insert mode fail

[axbender]

Steps to reproduce

1. gvim.exe -i NONE -N -u NONE --noplugin -U NONE
2. :inoremap °1 #abc
3. :inoremap !1 #abc
4. In insert mode "°1" will not be expanded, "!1" will.

Expected behaviour

Both imappings should work.

Version of Vim

gvim 8.2.4847, 64-bit

Environment

Windows 7 64-bit,

Logs and stack traces

No response

[LemonBoy]

8.2.4807 made some changes to how the keys are handled in gVim and one of the most important ones is that we now behave similarly to other GUI (namely the GTK one) wrt the modifier keys.
Your mapping inoremap °1 #abc works just fine when changed to inoremap <S-°>1 #abc as the shift key is pressed.

One could argue that for ° the shift modifier is implicit, but Vim doesn't really know that.

[brammool]

Making mappings work the same way on different systems is good, you can take your setup to another computer and it will still work. That this may break some old mappings can be annoying, but perhaps we can argue that it was a bug that they worked?

[axbender]

For me, this is ok as there's a workaround. Thx.

[chrisbra]

@LemonBoy can we somehow put this into the documentation, that one needs to be explicit about modifiers?

[LemonBoy]

The problem with writing documentation is that one should have a clear picture of what works, what's supposed to be working and what's not. And at the moment I'm quite confused.

Changing the key-handling code is quite some work because there's no automated testing and the probability of introducing regressions and angering users is quite high.
I tried to keep the W32 patches as self-contained as possible because of this reason, but there's still some work to do.

Beside merging _OnSysChar and _OnChar the main question is, what's the correct and desired simplification sequence for the keypresses?

The GTK GUI does:

• Special processing for keys in the special_key table.
• simplify_key
• may_adjust_key_for_ctrl
• may_remove_shift_modifier
• check_for_interrupt

While the W32 GUI does:

• Special processing for keys in the special_key table.
• simplify_key
• may_remove_shift_modifier
• extract_modifiers
• Inlined check_for_interrupt (does not call merge_modifyOtherKeys)

At the moment there are still some small differences between the two GUIs (eg. for W32 CTRL-2 is not treated as CTRL-@) that I'm planning to eliminate in a future PR, but I need some input from your side.

Which sequence is the "correct" one? The GTK one is pretty similar but not identical to handle_key_with_modifier so I think that's more likely to be the one to follow, but I'm not 100% sure about that.

[jottkaerr]

I'm sorry, but I think putting this into the documentation is wrong, because the behavior is wrong. For example, on a German keyboard, ° is a shifted character -- as are all of !"§$%&/()=?*. Why demand a <S- for ° and not for any/all of the other ones?

[LemonBoy]

Why demand a <S- for ° and not for any/all of the other ones?

There are a few levels of simplifications behind the scenes, may_remove_shift_modifier is quite dumb and only applies to several ASCII characters. ° is out of the ASCII range and de-shifting it requires some help from the windowing toolkit, that's obviously something that you can't do when running in a terminal.

#7016 (comment) a similar inconsistency was noted and fixed by removing the extra S- modifier.

We're back to the original question: what's the expected behaviour here? Do we want to add some more logic in the GUI layer to perform this translation? Do all the GUIs support this (looking at you, Motif)? What about the terminal?

Coming up with an answer is the hardest part, writing the code is a piece of cake.

[jottkaerr]

I find this problematic, one reason being the variety of national keyboard layouts. For example pound (£), degree (°), and section (§) are all characters outside the ASCII range. On an English (UK) Extended keyboard both are available next to each other as shifted keys on the digits row (3 and 4, respectively). Should a user of this layout really be expected to map them as <S+£> and $, respectively? They are equally accessible, fall into the same category, and are only internally differentiated because of a 1960's standard which today isn't relevant or even known to most users anymore.

On a Canadian (bilingual) keyboard the dollar sign keeps it place, but the pound sign is moved to AltGr+3. No shift needed. How to exchange configuration between users of these two layouts?

Same problem for German and Swiss (both German and Swiss mode) layouts -- German has § as Shift+3, Swiss has it unshifted on the first key of the digits row.

Regarding the ° key, the Dutch layout has it as an unshifted key to the left of the backspace key, and on an Icelandic keyboard it seems to be a dead-letter key at the start of the digits row, again unshifted.

There also superscript 2 and 3, which are sometimes entered as a combination of AltGr plus the corresponding base digit (e.g., on a German keyboard), but are entered as the unshifted and shifted first key on the digits row on a French (Belgium) keyboard.

I know it's hard to find a rule for when an additional <C- or even more so an additional <S- is needed, but personally I would not think of ° needing an additional <S-. I have already pressed <S-^> to enter it; I can't press Shift twice or twice as strong. ;-)

[brammool]

Right, so we cannot know by looking at the character we get whether it would have been different without the shift key. Thus we don't know if the shift key is extra information or is already included in the character. For ASCII uppercase letters we know, but for others it depends on the keyboard layout. Thus there should be a system API to tell us whether the shift modifier was used or not. This requires someone familiar with the API to figure this out, write the code and verify it actually works.

[jottkaerr]

I can't think of any character which would not be different if the same key is pressed with or without the shift key. Either they are letter-like symbols which change case, or shifted keys generate completely different symbols from unshifted keys. Making map <S-A> ... equivalent to map A ... might make sense, but making map <S-1> ... equivalent to map ! ... (if one is working with a German or similar keyboard layout) does not. And even less so does demanding a <S-°> just because ° is not an ASCII character.

There is probably only one printable character on every keyboard where the notation of an additional shifting makes sense, and that's <space>.

A combination of different modifiers is another story. There clearly can be a need to distinguish between <C-A> and <C-S-A>. That's fine, but how would a combination of ctrl and a printable symbol be interpreted? On most English (US and British) keyboards ; and : share the same physical key. If we press <Ctrl>+<:> does this get interpreted as <C+S+;>? Has a mapping to use this notation or <C+:>? And what if you switch to a German layout? There <;> is <Shift>+<,> and <:> is <Shift>+<.>. The need for <...+S+...> in the notation would make a mapping defined with a German layout in mind incompatible with an English layout.

In regards to combinations with ctrl and shift, I guess it comes down to how someone thinks about the base character. When I write mappings, for letter-like symbols (mostly Latin letters and German umlauts, in my case) I would distinguish between shifted and unshifted keys by including -S- in the mapping, especially because <c-a> and <C-A> have always been equivalent in Vim script. But for <Ctrl>+<(> I would not think "Hey, that's <C-S-8>." That's <C-(> in my mind. And for <Ctrl>+<°> it's <C-°>, not <C-S-^> and not <C-S-°>.

(And don't get me started on <C-[> on a German keyboard. ;-) )

P.S.: It seems some of the comments in this issue did not get through to the mailing list. (Something I have also noticed for other Github issues lately.) So excuse me, if I don't always react in a timely manner.

[brammool]

I can't think of any *character* which would not be different if the

I think you mean "printable character" here.

same key is pressed with or without the shift key. Either they are letter-like symbols which change case, or shifted keys generate completely different symbols from unshifted keys. Making `map <S-A> ...` equivalent to `map A ...` might make sense, but making `map <S-1> ...` equivalent to `map ! ...` (if one is working with a German or similar keyboard layout) does not. And even less so does demanding a `<S-°>` just because `°` is not an ASCII character. There is probably only one printable character on every keyboard where the notation of an additional shifting makes sense, and that's `<space>`.

There are many more, considering there are dozens of different keyboard layouts. Most printable characters have a key which will produce a different character with shift, but not all of them. Space is just one example. Tab is another one.

A combination of different modifiers is another story. There clearly can be a need to distinguish between `<C-A>` and `<C-S-A>`. That's fine, but how would a combination of ctrl and a printable symbol be interpreted? On most English (US and British) keyboards `;` and `:` share the same physical key. If we press `<Ctrl>+<:>` does this get interpreted as `<C+S+;>`? Has a mapping to use this notation or `<C+:>`? And what if you switch to a German layout? There `<;>` is `<Shift>+<,>` and `<:>` is `<Shift>+<.>`. The need for `<...+S+...>` in the notation would make a mapping defined with a German layout in mind incompatible with an English layout. In regards to combinations with ctrl and shift, I guess it comes down to how someone thinks about the base character. When I write mappings, for letter-like symbols (mostly Latin letters and German umlauts, in my case) I would distinguish between shifted and unshifted keys by including `-S-` in the mapping, especially because `<c-a>` and `<C-A>` have always been equivalent in Vim script. But for `<Ctrl>+<(>` I would not think "Hey, that's `<C-S-8>`." That's `<C-(>` in my mind. And for `<Ctrl>+<°>` it's `<C-°>`, not `<C-S-^>` and not `<C-S-°>`. (And don't get me started on `<C-[>` on a German keyboard. ;-) )

The processing of actually pressed keys to some character plus modifiers is very much dependent on the system. Therefore we need the system to tell us whether the shift key was used to change the character or not. Anything else is guessing. We might be good at guessing...

…

-- # echo reboot >universe # chmod +x universe # ./universe /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[jottkaerr]

I can't think of any character which would not be different if the

I think you mean "printable character" here.

Yes, "printable" makes it clearer about what characters I was talking.

[...]
There is probably only one printable character on every keyboard
where the notation of an additional shifting makes sense, and that's
<space>.

There are many more, considering there are dozens of different keyboard
layouts. Most printable characters have a key which will produce a
different character with shift, but not all of them. Space is just one
example. Tab is another one.

I actually think of Tab more as a control code than as an actual printable character. And of course it makes sense (to be able) to distinguish between a combination of modifiers with <Tab> and plain <Tab>.

[...]

The processing of actually pressed keys to some character plus modifiers
is very much dependent on the system. Therefore we need the system to
tell us whether the shift key was used to change the character or not.
Anything else is guessing. We might be good at guessing...

Yes, and the variety of different keyboard layouts is the reason why I think it's a bad idea to require the presence of <S- in a mapping just because the character happens to be a non-ASCII character. On some keyboards it might be printed on the shifted position of a key, on some keyboards not.