allow KEYMAP with @a

[rpuntaie]

I've debugged into vim and found the location, in which normal keymap application diverges when playing a register macro (@A).

It is in getchar.c: 2189

            if (mp->m_keys[0] == c1
                && (mp->m_mode & local_State)
                && ((mp->m_mode & LANGMAP) == 0
                || typebuf.tb_maplen == 0))

Both (no@a) and (@A) have

mp->m_mode == 32 == LANGMAP
local_State == 48 == 32+16

but only (@A) has: typebuf.tb_maplen > 0

Cited from getchar.c regarding the tb_maplen:

• The head of the buffer may contain the result of mappings, abbreviations
• and @A commands. The length of this part is typebuf.tb_maplen.

THE LAST TWO LINES COMMENTED AWAY MADE KEYMAP WORK WITH @A COMMANDS.

So before: no LANGMAP for @A. That's what ((mp->m_mode & LANGMAP) == 0 || typebuf.tb_maplen == 0) says, anyway.
With patch: LANGMAP for @A, i.e. without the condition.

tb_maplen is only set in ins_typebuf(), if the nottyped parameter is TRUE.
There are 12 such calls in check_abbr, do_execreg, exec_normal_cmd, server_to_input_buf, put_reedit_in_typebuf, put_in_typebuf.
As for abbreviation I found no difference in behavior before and after the patch.

Personally I would like to use KEYMAP with @A.
I also regard it as a general deficiency, when e.g. a russian guy cannot use @A to insert russian text.
But maybe there are unwanted side effects, I'm not aware of.

Bram, maybe you can think of side effects or know why this condition was put there in the first place.

[brammool]

Roland Puntaier wrote:

I've debugged into vim and found the location, in which normal keymap application diverges when playing a register macro (@A).

It is in getchar.c: 2189

              if (mp->m_keys[0] == c1
                  && (mp->m_mode & local_State)
                  && ((mp->m_mode & LANGMAP) == 0
                  || typebuf.tb_maplen == 0))

Both (no@a) and (@A) have

mp->m_mode == 32 == LANGMAP
local_State == 48 == 32+16

but only (@A) has: typebuf.tb_maplen > 0

Cited from getchar.c regarding the tb_maplen:

• The head of the buffer may contain the result of mappings, abbreviations
• and @A commands. The length of this part is typebuf.tb_maplen.

THE LAST TWO LINES COMMENTED AWAY MADE KEYMAP WORK WITH @A COMMANDS.

So before: no LANGMAP for @A. That's what ((mp->m_mode & LANGMAP) == 0 || typebuf.tb_maplen == 0) says, anyway.
With patch: LANGMAP for @A, i.e. without the condition.

The 'keymap' option is meant for when typing text in Insert mode. It
does not apply to when typing commands, in Normal mode. The 'langmap'
option is the other way around.

A register contains commands, which may start Insert mode and then
insert some text. Thus it's a mix. Translating all characters with
'keymap' (or 'langmap') will cause problems.

I wonder how you managed to get characters into a register that you want
to apply 'keymap' or 'langmap' to. This translation should have already
happened when typing the characters.

tb_maplen is only set in ins_typebuf(), if the nottyped parameter is TRUE.
There are 12 such calls in check_abbr, do_execreg, exec_normal_cmd,
server_to_input_buf, put_reedit_in_typebuf, put_in_typebuf.
As for abbreviation I found no difference in behavior before and after
the patch.

Personally I would like to use KEYMAP with @A.
I also regard it as a general deficiency, when e.g. a russian guy
cannot use @A to insert russian text.
But maybe there are unwanted side effects, I'm not aware of.

Bram, maybe you can think of side effects or know why this condition
was put there in the first place.

Yes, there are undesired side effects.

ARTHUR: Shut up! Will you shut up!
DENNIS: Ah, now we see the violence inherent in the system.
ARTHUR: Shut up!
DENNIS: Oh! Come and see the violence inherent in the system!
HELP! HELP! I'm being repressed!
The Quest for the Holy Grail (Monty Python)

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///

[vim-ml]

On Wed, Dec 30, 2015 at 9:10 PM, Bram Moolenaar
vim-dev-github@256bit.org wrote:

Roland Puntaier wrote:

I've debugged into vim and found the location, in which normal keymap application diverges when playing a register macro (@A).

It is in getchar.c: 2189

if (mp->m_keys[0] == c1
&& (mp->m_mode & local_State)
&& ((mp->m_mode & LANGMAP) == 0
|| typebuf.tb_maplen == 0))

Both (no@a) and (@A) have

mp->m_mode == 32 == LANGMAP
local_State == 48 == 32+16

but only (@A) has: typebuf.tb_maplen > 0

Cited from getchar.c regarding the tb_maplen:

• The head of the buffer may contain the result of mappings, abbreviations
• and @A commands. The length of this part is typebuf.tb_maplen.

THE LAST TWO LINES COMMENTED AWAY MADE KEYMAP WORK WITH @A COMMANDS.

So before: no LANGMAP for @A. That's what ((mp->m_mode & LANGMAP) == 0 || typebuf.tb_maplen == 0) says, anyway.
With patch: LANGMAP for @A, i.e. without the condition.

The 'keymap' option is meant for when typing text in Insert mode. It
does not apply to when typing commands, in Normal mode. The 'langmap'
option is the other way around.

A register contains commands, which may start Insert mode and then
insert some text. Thus it's a mix. Translating all characters with
'keymap' (or 'langmap') will cause problems.

I wonder how you managed to get characters into a register that you want
to apply 'keymap' or 'langmap' to. This translation should have already
happened when typing the characters.

tb_maplen is only set in ins_typebuf(), if the nottyped parameter is TRUE.
There are 12 such calls in check_abbr, do_execreg, exec_normal_cmd,
server_to_input_buf, put_reedit_in_typebuf, put_in_typebuf.
As for abbreviation I found no difference in behavior before and after
the patch.

Personally I would like to use KEYMAP with @A.
I also regard it as a general deficiency, when e.g. a russian guy
cannot use @A to insert russian text.

To insert text, I would use "a (or, in Insert mode, ^Ra), not @A. But
then, I very seldom use recording (about the only case is when I wan
something to go on repeating until it errors out). I have set apart
register q for those rare cases; but most of my other registers
contain text strings that I want to be ready to paste anytime. Some
are rather long text which I don't use very often but wouldn't
remember by heart, others are short but frequent. And of course if I
want to be able to repeatedly insert some Russian text I'll put it in
the register in Cyrillic; for English (or French or German or …) I
would put it there in Latin. The help for i_CTRL-R says "The text is
inserted as if you typed it, but mappings and abbreviations are not
used."; language-mappings (keymaps) are not used either, so that e.g.
in an HTML file in Russian, when I hit ^Ri what gets inserted is
(the contents of the register) and not <и> which wouldn't be a proper
HTML tag, even if at that point my "cyrillic-phonetic" keymap is
enabled ('iminsert' = 1).

I'm not very familiar with the 'langmap' option but I would expect
registers containing Cyrillic text to be pasted in Cyrillic and
registers containing Latin text to be pasted in Latin in that case
too. Does 'iminsert' apply to this option too? I couldn't find the
answer in the help for any of 'langmap', 'iminsert' and i_CTRL-^
(maybe translate even text typed in Insert mode when 'iminsert' is 0
[to insert Latin text] but not when it is 1 [to insert Cyrillic
text])?

But maybe there are unwanted side effects, I'm not aware of.

Bram, maybe you can think of side effects or know why this condition
was put there in the first place.

Yes, there are undesired side effects.

ARTHUR: Shut up! Will you shut up!
DENNIS: Ah, now we see the violence inherent in the system.
ARTHUR: Shut up!
DENNIS: Oh! Come and see the violence inherent in the system!
HELP! HELP! I'm being repressed!
The Quest for the Holy Grail (Monty Python)

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///

[rpuntaie]

Dear Bram,

A register contains commands, which may start Insert mode and then insert some
text. Thus it's a mix. Translating all characters with 'keymap' (or 'langmap')
will cause problems.

Not ALL characters are translated, because the map hash includes the vim State:

local_State = get_real_state();
...
mp = curbuf->b_maphash[MAP_HASH(local_State, c1)]; //getchar.c:2158 

The hash is generated with LANG_MAP:

(void)do_map(2, buf, LANGMAP, FALSE);

For vim.h:

#define LANGMAP     0x20    /* Language mapping, can be combined with
                   INSERT and CMDLINE */

This means keymap only applies to INSERT and CMDLINE, as it should be.

Note: With LANGMAP I meant and mean the define in the C code not :set langmap.
The latter is handled via LANGMAP_ADJUST using the global static garray_T langmap_mapga;
defined in options.c.
keymap on the other hand is done via the global or local maphash.
langmap is handled before keymap.
The C LANGMAP define only applies to keymap.

Note: Probably russians will have a russian keyboard layout and use :set
langmap instead of :set keymap. Russian is only an example, though.
Since keymap applies to INSERT and CMDLINE for direct typing,
it should do so also for @A macros. I think everybody will expect
@A to behave the same way as direct typing.

I wonder how you managed to get characters into a register that you want to
apply 'keymap' or 'langmap' to. This translation should have already happened
when typing the characters.

• do ":help russian"
• use the keymap as described there
• insert some russian text
• record a macro @A that inserts some russian text
• play the macor @A

Yes, there are undesired side effects.

Please name the undesired side effects, else I can't decide, whether you think, there are,
or you know them.

[vim-ml]

On 12/30/2015 9:59 PM, Tony Mechelynck wrote:

The help for i_CTRL-R says "The text is inserted as if you typed it,
but mappings and abbreviations are not used."; language-mappings
(keymaps) are not used either, so that e.g. in an HTML file in
Russian, when I hit ^Ri what gets inserted is (the contents of the
register) and not <и> which wouldn't be a proper HTML tag, even if at
that point my "cyrillic-phonetic" keymap is enabled ('iminsert' = 1).

Does 'iminsert' apply to this option too? I couldn't find the answer
in the help for any of 'langmap', 'iminsert' and i_CTRL-^ (maybe
translate even text typed in Insert mode when 'iminsert' is 0 [to
insert Latin text] but not when it is 1 [to insert Cyrillic text])?

:set keymap=russian-jcukenwin
Will set iminsert=1. So this means keymap set.
^Ri is completely independent of keymap and it remains so with my patch.

I'm not very familiar with the 'langmap' option but I would expect
registers containing Cyrillic text to be pasted in Cyrillic and
registers containing Latin text to be pasted in Latin in that case too.

If you had e.g. a russian keyboard layout, you wouldn't be able to use
vim, e.g. to move around in normal mode, like with the english
keyboard's hjkl. langmap will translate ролд to hjkl.
While langmap applies for normal mode, keymap works in INSERT and
CMDLINE mode.
Langmap will not apply for CMDLINE, so you will not be able to enter vim
commands like :g, because the g is п on the russian keyboard. So langmap
is probably not very useful.
Keymap on the other hand will keep the english keyboard in normal mode
and for : commands. So you can still enter :g. It applies for /, so you
can search for russian text. (Behavior not changed with my patch.)

[vim-ml]

On Thu, Dec 31, 2015 at 12:30 AM, Roland Puntaier
roland.puntaier@chello.at wrote:

On 12/30/2015 9:59 PM, Tony Mechelynck wrote:

The help for i_CTRL-R says "The text is inserted as if you typed it, but
mappings and abbreviations are not used."; language-mappings (keymaps) are
not used either, so that e.g. in an HTML file in Russian, when I hit ^Ri
what gets inserted is (the contents of the register) and not <и> which
wouldn't be a proper HTML tag, even if at that point my "cyrillic-phonetic"
keymap is enabled ('iminsert' = 1).

Does 'iminsert' apply to this option too? I couldn't find the answer in
the help for any of 'langmap', 'iminsert' and i_CTRL-^ (maybe translate even
text typed in Insert mode when 'iminsert' is 0 [to insert Latin text] but
not when it is 1 [to insert Cyrillic text])?

:set keymap=russian-jcukenwin
Will set iminsert=1. So this means keymap set.
^Ri is completely independent of keymap and it remains so with my patch.

I'm not very familiar with the 'langmap' option but I would expect
registers containing Cyrillic text to be pasted in Cyrillic and registers
containing Latin text to be pasted in Latin in that case too.

If you had e.g. a russian keyboard layout, you wouldn't be able to use vim,
e.g. to move around in normal mode, like with the english keyboard's hjkl.
langmap will translate ролд to hjkl.
While langmap applies for normal mode, keymap works in INSERT and CMDLINE
mode.
Langmap will not apply for CMDLINE, so you will not be able to enter vim
commands like :g, because the g is п on the russian keyboard. So langmap is
probably not very useful.
Keymap on the other hand will keep the english keyboard in normal mode and
for : commands. So you can still enter :g. It applies for /, so you can
search for russian text. (Behavior not changed with my patch.)

I see. Sounds like we need to:
(a) make langmaps apply to command-line mode by default, so Russian,
Greek, etc. users can type ex-commands by using them;
(b) implement some way of toggling them on/off "on the fly" so we could e.g. do
:let list = [{name: "Иван", patronym: "Петрович", family:
"Михайлов", city: "Москва"}, {name: "Ирина", patronym: "Андреевна",
family: "Свердлова", city: "Санкт-Петербург"}, {name: "Фёдор",
patronym: "Афанасьевич", family: "Воронин", city: "Нижний Новгород"}]
etc., even with a Cyrillic layout used with langmap.

Best regards,
Tony.

[brammool]

Roland Puntaier wrote:

Dear Bram,

A register contains commands, which may start Insert mode and then
insert some text. Thus it's a mix. Translating all characters with
'keymap' (or 'langmap') will cause problems.

Not ALL characters are translated, because the map hash includes the vim State:

local_State = get_real_state();
...
mp = curbuf->b_maphash[MAP_HASH(local_State, c1)]; //getchar.c:2158 

The hash is generated with LANG_MAP:

(void)do_map(2, buf, LANGMAP, FALSE);

For vim.h:

#define LANGMAP       0x20    /* Language mapping, can be combined with
                   INSERT and CMDLINE */

This means keymap only applies to INSERT and CMDLINE, as it should be.

Note: With LANGMAP I meant and mean the define in the C code not :set langmap.
The latter is handled via LANGMAP_ADJUST using the global static garray_T langmap_mapga;
defined in options.c.
keymap on the other hand is done via the global or local maphash.
langmap is handled before keymap.
The C LANGMAP define only applies to keymap.

Note: Probably russians will have a russian keyboard layout and use :set
langmap instead of :set keymap. Russian is only an example, though.
Since keymap applies to INSERT and CMDLINE for direct typing,
it should do so also for @A macros. I think everybody will expect
@A to behave the same way as direct typing.

No, the langmap and keymap options relate to the keyboard. Replaying a
register does not involve the keyboard. When replaying a register one
doesn't know where the characters came from, they might be typed or the
result of something else, e.g. a yank operation.

I wonder how you managed to get characters into a register that you want to
apply 'keymap' or 'langmap' to. This translation should have already happened
when typing the characters.

• do ":help russian"
• use the keymap as described there
• insert some russian text
• record a macro @A that inserts some russian text
• play the macor @A

This is a bit vague... Perhaps the problem is that when recording into
a register the keymap/langmap is not applied, resulting in the wrong
characters in the register?

I look at keymap/langmap as a low level operation, which is applied when
typing on the keyboard, before it goes to the rest of Vim, which is
unaware of this filtering.

It's possible that this interferes with other things, such as terminal
codes and other mappings. It's quite complicated. Many changes in this
area are not backwards compatible and break someone's carefully tuned
setup. I have thought of re-implementing this, but haven't found the
time to dive into it.

No letters of the alphabet were harmed in the creation of this message.

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///

[vim-ml]

On 12/31/2015 6:22 PM, Bram Moolenaar (Vim Github Repository) wrote:

Note: Probably russians will have a russian keyboard layout and use :set
langmap instead of :set keymap. Russian is only an example, though.
Since keymap applies to INSERT and CMDLINE for direct typing,
it should do so also for @A macros. I think everybody will expect
@A to behave the same way as direct typing.

No, the langmap and keymap options relate to the keyboard. Replaying a
register does not involve the keyboard. When replaying a register one
doesn't know where the characters came from, they might be typed or the
result of something else, e.g. a yank operation.

I wonder how you managed to get characters into a register that
you want to
apply 'keymap' or 'langmap' to. This translation should have
already happened
when typing the characters.

• do ":help russian"
• use the keymap as described there
• insert some russian text
• record a macro @A that inserts some russian text
• play the macor @A

This is a bit vague... Perhaps the problem is that when recording into
a register the keymap/langmap is not applied, resulting in the wrong
characters in the register?

I look at keymap/langmap as a low level operation, which is applied when
typing on the keyboard, before it goes to the rest of Vim, which is
unaware of this filtering.

It's possible that this interferes with other things, such as terminal
codes and other mappings. It's quite complicated. Many changes in this
area are not backwards compatible and break someone's carefully tuned
setup. I have thought of re-implementing this, but haven't found the
time to dive into it.

As of now
//getchar.c:2515
if (keylen > typebuf.tb_maplen)
gotchars(typebuf.tb_buf + typebuf.tb_off
+ typebuf.tb_maplen,
keylen - typebuf.tb_maplen);

is before
//getchar.c:2610
i = ins_typebuf(s, noremap,

i.e. the current code does not put the result of the keymap mapping into
the register.
So my patch fits to the current implementation by applying the keymap to
chars in the register.

Since my last commit I have

• left the original version as comment for reference in case of problems.
• added langmap application to ':'-CMDLINE, as proposed by Tony Mechelynck,
  in order to make langmap a better opposite to keymap.
  I did LANGMAP_ADJUST after gotchars(),
  because currently it is applied on chars from registers (@A).

I have also updated the documentation to mention:

• langmap applies to ':'-CMDLINE mode (new)
• keymap applies to non-':'-CMDLINE mode (current)

Best Regards, Roland

[rpuntaie]

Hello Bram,

I've seen the recently added (issues/572) test_langmap.vim's use of feedkeys().

That langmap should apply after feedkeys() does not fit
to my changes to the langmap implementation,
which apply langmap at a very early place (inchar())
and also apply langmap in ':'-command line mode.
The latter because langmap is intended to be complementary to
keymap and because langmap's intention likely was
to make vim usable from cyrillic or greek keyboards.

Personally I'm favoring my solution, but not passionately (As I've mentioned I'm more interested
in the keymap correction).

Anyway it's up to you to decide. If you reject this pull request,
allow me to open another pull request with only the originally intended change regarding keymap.

Thanks.
Best, Roland

[brammool]

Roland Puntaier wrote:

Hello Bram,

I've seen the recently added (issues/572) test_langmap.vim's use of
feedkeys().

That langmap should apply after feedkeys() does not fit
to my changes to the langmap implementation,

Note that depending on flags feedkeys() inserts a bit earlier or not.
Esp. whether the keys are mapped and/or typed. Well, it's the same
buffer but with different flags (we may split it into more than one
buffer some day to make this easier).

which apply langmap at a very early place (inchar())
and also apply langmap in ':'-command line mode.
The latter because langmap is intended to be complementary to
keymap and because langmap's intention likely was
to make vim usable from cyrillic or greek keyboards.

Personally I'm favoring my solution, but not passionately (As I've
mentioned I'm more interested in the keymap correction).

Anyway it's up to you to decide. If you reject this pull request,
allow me to open another pull request with only the originally
intended change regarding keymap.

I haven't looked at your latest patch yet, but if langmap/keymap is
applied before where feedkeys() inserts characters, how can you test it?
Perhaps we need another flag in feedkeys() to insert keys even earlier?

Seen it all, done it all, can't remember most of it.

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///

[rpuntaie]

call feedfields() is in :-Command-mode, where langmap according this pull request already applies.
Testing can still be done with old style tests. The current tests are in test_mapping.in.
They include #572.

[brammool]

No comments for a long time. At least low-level testing with feedkeys() should work now.
Any updates?

[Opisek]

Any progress?