[RDY] Apply :lmap in macros

[hardenedapple]

I mentioned this patch in the bug #5652, but realised it should be a PR.

Vim ignores :lmap when executing a mapped command as documented in :help language-mapping, but this messes up macro replay.

Example:

:set iminsert=1<CR>
:lnoremap a l<CR>
qaiaaa<CR><Esc>q@a

which inserts two lines: lll and aaa.

This extra line adds an exception for macro replay.

[justinmk]

If I recall, this has come up on vim_dev recently (couldn't find the recent discussion, but here is an older one). Bram always rejects it. Should we break Vim-compat here? Why? What are the alternatives? Can you comment on that discussion (if it's relevant, I didn't look closely).

Maybe @chrisbra can comment. Also, did this discussion ever get fully resolved? (Is it relevant? I didn't look closely.)

cc @ZyX-I @blueyed

[chrisbra]

I believe the discussion you are refering to has been merged as the 'langnoremap' option. However there have been a couple of more discussions, e.g. vim/vim#543. The problem is, this part of the code seems extremly fragile, changing something usually breaks in subtile ways, so one needs to have propper tests for those kind of changes.

[hardenedapple]

As if to prove @chrisbra 's point of it being fragile, I noticed that my change added in some bad behaviour when a macro recording includes an :nmap that inserts some text including some :lmapped characters.

:nmap l ix
:lmap x q
qalll<CR><Esc>q@a

You see one line of xll and another of qll with my patch, and two lines of xll beforehand, so what I have suggested here is broken.

Reading the discussion of vim/vim#543 (thanks for pointing to that) I think the changes that are discussed there sound much better than the patch I gave here. Bram says

I look at keymap/langmap as a low level operation, which is applied when
typing on the keyboard, before it goes to the rest of Vim, which is
unaware of this filtering.

which would work with recording a macro and trying to replay it (unlike the current behaviour), and make things easier to understand than now.
Also, the discussion there of applying 'langmap' to cmdline mode seems sensible too.

As a side note: it seems that starting nvim with nvim -u NONE means that while :set langnoremap? says the setting is on, the effect isn't seen.

I'll try and comprehensibly write down the current behaviour and where it differs from the mental model of a conversion between the user typed characters and Vim itself, so we can see what other changes that simpler model would add. I'll also have a look at the patch in vim/vim#543 to see if that patch makes this change.

[hardenedapple]

My opinion (though I'm biased because the current behaviour annoys me) is that
the current behaviour is too difficult to reason about, and in this specific
instance (macros and keymap) essentially blocks macros from being used.
At the moment, the only way to run a macro with characters you don't have on
the keyboard but have translated in the keymap is to manually put the text
into that register instead of recording it interactively, which is enough of a
pain that I don't do it.

After thinking more on it I would actually propose a different solution to my
problem: instead of applying :lmap mappings to the register when executing
it, I would apply them when interactively recording into the register (I
haven't yet checked whether this would be an easy change or not).
Assuming other users of keymap also don't record macros interactively because
of this difficulty, then this wouldn't break anyones workflow, and would make
macros with keymap much more usable.

Assuming the settings langnoremap and using :lmap then there are only
really two places that the mental model of a pre-translation of characters is
broken. One is what happens during macro recording that is the original problem
above, and the other is in the priority that mappings between :lmap and
:imap are applied.
Instead of :lmap getting priority (as it would be if the mental model would
hold, it depends on the order the mappings are defined).

Example:

:set iminsert=1
:lmap a l
:imap a i
oaaa

Inserts iii -- i.e. the imap mappings are applied first.

Although this is be dependant on the order the mappings were defined
Example:

:set iminsert=1
:imap a i
:lmap a l
oaaa

Inserts lll -- i.e. the lmap mappings are applied first.

I think this should be different, but frankly it bothers me much less, so I
don't mind if not.

I agree there should be thorough testing here -- it's been pretty difficult
for me to understand what's going on sometimes. I'll be happy to start on that.

In the meantime I'll have a look at what changes the vim/vim#543 PR introduces to see if there's anything interesting there.

[justinmk]

instead of applying :lmap mappings to the register when executing it, I would apply them when interactively recording into the register

But then the input from the macro (during replay) is not respecting keymap or langmap? Which is correct, based on Bram's comments in vim/vim#543.

The 'keymap' option is meant for when typing text in Insert mode. It
does not apply to when typing commands, in Normal mode. The 'langmap'
option is the other way around.

A register contains commands, which may start Insert mode and then
insert some text. Thus it's a mix. Translating all characters with
'keymap' (or 'langmap') will cause problems.

I wonder how you managed to get characters into a register that you want
to apply 'keymap' or 'langmap' to. This translation should have already
happened when typing the characters.

And later:

Perhaps the problem is that when recording into
a register the keymap/langmap is not applied, resulting in the wrong
characters in the register?

I look at keymap/langmap as a low level operation, which is applied when
typing on the keyboard, before it goes to the rest of Vim, which is
unaware of this filtering.

@hardenedapple so I think you're right, the keys should be translated before being stored in the macro register. Based on Bram's comments, it is already expected to do so.

And using the Neovim test infrastructure it should be easy to correctly test this. (helpers.feed() is identical to raw user input)

[hardenedapple]

I believe this fixes most of the problems with :lmap, 'keymap', and 'langmap' while still keeping the implementation essentially the same.

I intend to try and write a completely new implementation, but seeing as that will probably have backwards compatibility problems I'll keep that for a different PR.
It turned out my alternate implementation isn't any better at all.

[hardenedapple]

Making 'keymap' respect the mental model of a translation before getting to vim proper was a reasonably clean and simple task.
Making 'langmap' respect the mental model is much more awkward, and I would like some input on what we should aim for.

Hence I've made this PR just fix the 'keymap' / :lmap discrepancies, and will be opening another PR to show and start working on the 'langmap' behaviour.

[justinmk]

in general use command() instead of feed_command() unless the asynchrony is intended. Also feed_command() doesn't bubble errors up (though it does set v:errmsg, but that needs to be checked explicitly)

[justinmk]

Let's put this in test/functional/options/keymap_spec.lua.

(test/functional/ui/... is for testing the UI)

[justinmk]

Leave :lnoremap alone, letting users still have the functionality but possibly introducing confusing situations if they use :lnoremap to influence the behaviour of 'keymap' (my preferred option as this seems like an unlikely case).

Let's try that, and revisit if needed.