Meta sends escape enabled while running filter from "autocmd"

[antofthy]

Describe the bug
After upgrading to Vim-8.2 the 'xterm modifyOtherKeys' patch has caused issues with me... I use "autocmd" to read in a binary encrypted file, which is then decrypted using a password read directly from the users TTY.

If I read from the xterm TTY from the command run from "autocmd" and I type 'A' I get the input '\e[27;2;65~' which is the ANSI code for a 'Shifted-A' and not an actual 'A'. As you can imagine the password is then wrong, the the command fails to decrypt the file I want to edit. Only a problem if the passwords contain shifted characters in it!

If I use a :%!command directly, there is no problem... So VIM is disabling the 'xterm modifyOtherKeys' it for that. It only becomes a problem when I do this from a "autocmd".

This sort of "autocmd" input is what would be used by commands like "gpg" and "openssl enc" to encrypt and decrypt files via vim "autocmd" and this is currently failing!

To Reproduce
run vim file.test in a xterm does not matter if the file does not exist

Inside vim...

:" check t_TI is set to t_TI=^[[>4;2m
:set t_TI
:" Set an autocommand to read from TTY
:autocmd  BufWritePre,FileWritePre   *.test !bash -c 'cat </dev/tty >/dev/tty'
:" now write the file to run the above autocmd
:w

The command will not be waiting from some input from the TTY...
type 'A' and you will see the resulting ANSI sequence.

You can not even type ctrl-D to terminate the above, and will need to kill the "cat" from another terminal!

If you did the command directly without using autocmd, EG:

" Direct shell call
:!bash -c 'cat </dev/tty >/dev/tty'

It works perfectly fine. The problem only happens from autocmd, for both call '!' or filter '|' shell commands.

Expected behavior
Input should be what the user typed, not ANSI escape sequences.

Environment (please complete the following information):
Fedora Packages:
vim-common-8.2.109-1.fc31
vim-enhanced-8.2.109-1.fc31
vim-filesystem-8.2.109-1.fc31

vim --version
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Jan 10 2020 00:00:00)
Included patches: 1-109
Modified by bugzilla@redhat.com
Compiled by bugzilla@redhat.com

Terminal xterm-351-1.fc31
TERM=xterm-256color

Additional context
This appears to be the reason the patch was added...
#4974 (comment)

A temporary fix was provided in
#5200 and #5612 (comment)

[brammool]

it is now possible to temporarily disable modifyOtherKeys with: call echoraw(&t_TE)
And enable it again with: call echoraw(&t_TI)

[antofthy]

@brammool Added to the functions my autocmd's call for encrypted files...
"call echoraw(&t_TE)

E117: Unknown function: echoraw

At the moment the only solution I have is to disable it entirely, or output the ANSI sequence myself in a password reader wrapper (non-vim solution).

ASIDE...
Vim Encrypted file handler...
https://antofthy.gitlab.io/software/#encrypt.vim

[k-takata]

Do you use 8.2.0258 or later?

[antofthy]

Update from fedora repo is only until 8.2.236
and that is only on my local machine. Not on others.

But I already have a temporary (non-vim) solution, Just looking to get this fixed for the future without needing any extras. EG: the filters called by the autocmd's do the right thing. That way after a period of time, after all machines have updated I can get rid of my temporary solution, and forget about this 'weird period'.

[antofthy]

Something is very strange about these ANSI control sequences as defined!
echoing the "t_TE" control sequence before getting user input was causing the first letter
to the prompt that follows to 'disappear'!

So I did some tests of the ANSI sequences vim uses...
The 'echo' command are being performed just on the command line bash, not from in vim itself.

For example this loses the first letter 'A': echo $'\e[>4mABCD'
And this prints none of the numbers!: echo $'\e[>4m1234'
Adding a RETURN does not help... echo $'\e[>4m\r1234'
Nor does adding a NEWLINE... echo $'\e[>4m\n1234'
The newline IS output, but NOT the numbers!

As it is missing one letter adding a 'junk' character to the ANSI control sequences seems to fix the problem... For example: echo $'\e[>4m~1234' or echo $'\e[>4m~ABCD'

I think I may be missing something about ANSI sequences! or XTerm implements it wrong!
But if I am right, then the vim termcap settings are not correct.

So is this a "XTerm" fault, or a "vim" fault (mis-read of ANSI sequences)?

[chrisbra]

I suppose you were using :!echo ...? I cannot reproduce this.

[antofthy]

This is was in normal bash shell (editing previous comment to that effect), just testing the control sequences VIM is using for t_TI and t_TE termcap entries. Run them on the command line to see the effects of the ANSI escape sequence.

Basically either XTerm is not terminating the sequence on the 'm', or VIM needs to output an extra 'junk' character.

[antofthy]

Update...

The extra character required weirdness in the implementation of the ANSI sequences in XTerm, as shown above appears to be fixed now (Fedora 31). That seems to be a non-vim problem.

But the original problem, as demonstrated in the original first post, where the 'xterm modifyOtherKeys' is still enabled while running an autocmd filter that reads from a terminal, remains!

Current solutions...

• And I can now use echoraw() to wrapper the filter function call within the autocommand.
  as used in autocmd vim function in https://antofthy.gitlab.io/software/#encrypt.vim

call echoraw(&t_TE)
"... do the terminal input filter
call echoraw(&t_TI)

• I can also disable the use of modifyOtherKeys completely using

if (v:version >= 820)
  let &t_TI = "<Csi>>2m"
  let &t_TE = ""
endif

• Or ensure input program disables the 'xterm modifyOtherKeys' mode.
  For example from a bash script using something like...

if [[ $RTYPE =~ FC3[12] ]] && tty >/dev/null; then
  #echo >/dev/tty "Fixing XTerm ANSI 'ModifyOtherKeys'..."
  case "$TERM" in
  xterm*) echo -n $'\e[>4n' >/dev/tty ;;
  esac
fi

In fact I am actually doing all three, to ensure this debug does not interfere with the terminal input (reading a password, with echoed stars to decrypt encrypted files for editing).

[antofthy]

Please note however that I disagree with the closure of this bug.

Adding the ability to use echoraw() does NOT solve the bug, it only provides a work around for the bug!

Current version of vim... 8.2.694-1.fc31

[brammool]

OK, I found the spot where the behavior is different when autocommands are executed. Hmm, we can't just change that without lots of side effects. One solution is to do what we do for bracketed paste. That seems like an acceptable solution.

[Shane-XB-Qian]

@brammool though not very sure about the issue desc here, but probably better revert the patch#8.2.0767, looks it made vim (ater this patch) a little mess ..
it probably fixed this issue in pure xterm terminal, but made a little mess in e.g xfce-term terminal ..
e.g: vim --clean , then just ':' , it output the 'seq' sth ..., and output a lot in somewhere if you had a little complicated vimrc ..
wish you can revert ... revert ... -_-# or looks lots compatible work need to do to fit this change ...