OpenGL GPU-accelerated terminal screen stutters with Vim, doesn't happen with neovim

[poetaman]

Describe the bug
Vim screen stutters a lot with modern OpenGL-based terminals that offload screen rendering to GPU. AFAIK the philosophy of these terminals is different, they don't redraw part of the screen, and rather update entire screen as its fast & cheap. The same stutter behavior is not seen in old terminals that use CPU to do the rendering, and draw only part of the screen that needs update. But users are & will keep moving to new OpenGL-based terminals in future as they are super fast & low power (even compared to GVim GUI on macOS).

To Reproduce

1. Open one of the OpenGL-based terminals like Alacritty, Kitty, or WezTerm (problem happens with iTerm2 too, but not as apparent in this very short example)
2. Run vim --clean
3. Open a file :e ~/.vimrc
4. Run command :redraw! multiple times.

Expected behavior
There should never be screen stutters on openGL terminals when executing commands (like :redraw!). In this example it seems like :redraw! is the culprit, though redraw also does it when called from inside functions (like in plugin Startify [minimal example at bottom]). Note: NeoVim doesn't have such stutters, so it already is doing something different for redraws compared to Vim.

MP4 Videos (new GitHub feature works only on laptop/desktop for me)

1. Stuttering in Vim:

startify_stutter.mp4

2. No stuttering in NeoVim:

startify_nostutter.mp4

The minimal .vimrc in case you want to try it with :Startify command in above videos:

set nocompatible 
filetype plugin on
set noswapfile
syntax enable

call plug#begin('~/.vim/plugged')
Plug 'mhinz/vim-startify'
call plug#end()

set background=dark

let g:startify_lists = [
    \ { 'type': 'files',     'header': ['   MRU']            },
    \ { 'type': 'dir',       'header': ['   MRU '. '$PWD'] },
    \ { 'type': 'sessions',  'header': ['   Sessions']       },
\ ]

nnoremap <silent> <leader>] :Startify<CR>

The above setup should also make stutters clearly visible in iTerm2.

Environment (please complete the following information):

• VIM - Vi IMproved 8.2 (Included patches: 1-2576) (macOS version - arm64)
• macOS 11.2.3 ARM64 (machine: Apple M1 based Mac mini)
• OpenGL-based Alacritty, Kitty, or WezTerm

Additional context
Options like lazyredraw, and ttyfast have no effect on this.

[brammool]

This is not reproducible. No idea what you are doing in those videos, I don't see a :redraw! command being typed.

Please provide a reproducible example without using a plugin. E.g. editing the Vim source code.

[poetaman]

@brammool Here we go... Below is a complete example with no plugins & me typing :redraw! visible... Sorry for the ambiguity in previous video (you couldn't see what I was doing because of this nnoremap <silent> <leader>] :Startify<CR>). I was pressing <leader>]

vim_stutter_minimal.mp4

Steps are same as in video:

1. Open Alacritty, Kitty or WezTerm. (the effect happens less often on iTerm2 for reason mentioned below)
2. vim --clean filename.vim (I am using a vim filetype, if its syntax highlighting matters)
3. Create a vertical & horizontal split (:vs<CR>, <C-w>w, :sp<CR> is what I did)
4. Enter :redraw! multiple times...

Please let me know if you are able to reproduce with these steps.

iTerm2 uses metal renderer, vs others that use OpenGL. Given performance difference, effect is visible on iTerm2 less often (given lower delays), but it does... Instead of every time, it would be visible probably once in 20 times for this simple example. When used with the plugin example above, it happens 80% of the time.

Also, with iTerm2 the GPU rendering is not enabled all the time! You need to explicitly enable it when not connected to power (its annoying...) I learnt it hard way on my older intel Mac laptop last night. (another reason to use other openGL terminals for such testing).

Some more setup information: I have connected my Apple M1 (ARM64) based Mac mini to a 27inch 5k display. Not sure if size matters, but it could.

[brammool]

What I see here is perfectly normal: screen is cleared and then the text is drawn.
Perhaps neovim flushes less often, so that it's all done in one go?
That may reduce flicker, but delay output in other situations.
Investigation would require intercepting what exactly is sent to the terminal, possibly with timing.
I'll leave that to someone who wants to spend time on it.

[poetaman]

@brammool: @bfredl from neovim team has some preliminary insight in his comment: neovim/neovim#14225 (comment).

@bfredl: Any chance to see what's going on in Vim? Thanks!

[brammool]

@brammool: @bfredl from neovim team has some preliminary insight in his comment: neovim/neovim#14225 (comment). @bfredl: Any chance to see what's going on in Vim? Thanks!

I believe what is said here that Neovim doesn't clear the screen, but draws over the text. In many cases that is slower, but the advantage is that the user sees less flicker. A risk is that when some text was drawn outside of Vim's control it might not be replaced. The Vim :redraw command only has two modes: without !: redraw what is outdated, with !: clear and redraw. There is no way to redraw with NOT_VALID, which would redraw everything but not clear. Perhaps ":redraw all" could be added.

…

-- hundred-and-one symptoms of being an internet addict: 74. Your most erotic dreams are about cybersex /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[monkoose]

Good day @brammool

Any updates on fixing this issue?
It's really eyehurting to scroll filetypes with heavy regex syntax (too much redraws equal too much flickering) or using such popular tool like fzf inside vim.

Sorry to be annoying (I understand that your focus is now on vim9 and have read that You don't want to do this work by yourself, but seems like noone else have knowledge or willing to fix this either) and understandable that you personally do not have such issues.

[brammool]

The priority on this issue is very low, since I don't see why a user would type ":redraw!", other then when the screen is messed up and then the behavior is totally justified.

If screen redrawing is so fast, then why would there be any flicker?

Is there any normal use that is affected?

[monkoose]

I guess OP missguide You or something. You don't need to invoke :redraw! to see flickering. My english isn't the best so it's hard to explain. It flickers the screen too much when you just scroll with <C-f> some filetypes which syntax has some heavy regexes as example new vim9 syntax file by lacygoill https://github.com/lacygoill/vim9-syntax ( Just add it to plugins open some big vim9 file) and do some <C-f> <C-b>with enabled options 'number' and laststatus=2. And for me screen redraws on like every such keypress that will flicker especially number column and statusline too. When at the same time neovim is smooth.
Or I'm not sure if you know, but fzf cli tool is very popular among linux and vim users, and when you use it in popup terminal window inside vim it flickers too. This are the two usecases that bothers me personally, but i do believe there are much more.
And it's not GPU accelerated terminals related issue, for me it behaves the same in any terminal emulator i have tested.
What else I can provide gifs movs (that compare both editors) to ensure you that this problem is present in just normal usage, not only in a nonsense :redraw! spam?

[lacygoill]

I don't know whether it will help, but clearing 't_ut', and setting 'lazyredraw' might be worth a try:

set t_ut=
set lazyredraw

[chrisbra]

Perhaps it's also worth setting writedelay option to some value. That may help to identify problems with redrawing, because redrawing is slow downed. You should see if the screen is always cleared

[brammool]

When you say "it flickers" when you use CTRL-F, then it's normal that the screen is redrawn, since the text is updated.
Flicker means that some text is cleared and then the same text is drawn again. Where do you see that? Only the number column and the status line?

I'm not sure a recording is much help showing what you see. You can use the "script" command to record the output, comparing Vim and Neovim to see what is sent and finding the relevant parts. Perhaps a "clear screen" code t_cl appears?

[monkoose]

@lacygoill already had lazyredraw and t_ut haven't change anything.
@chrisbra Thank You, I have played with with writedelay I'm not sure how it should look but even with writedelay=2 it is very very slow line redrawing, while in neovim it is still almost instant. For me vim with writedelay=2 == neovim with writedelay=200

Tested with vim -u NONE without any sufficient improvement, maybe slightly better because of disabled syntax etc.
Why is it so?

[lacygoill]

You can use the "script" command to record the output, comparing Vim and Neovim to see what is sent and finding the relevant parts. Perhaps a "clear screen" code t_cl appears?

To record the output:

$ script --quiet --timing=/tmp/script_timing.log --command='vim -Nu NONE' /tmp/script_record.log

To replay the recording:

$ scriptreplay --typescript=/tmp/script_record.log --timing=/tmp/script_timing.log

On Ubuntu, the script command is provided by the bsdutils package:

$ sudo apt install bsdutils

The control sequences are documented here. The xterm package might provide a local documentation:

$ vim $(locate ctlseqs.txt)

[monkoose]

@lacygoill not sure really what is going on

[lacygoill]

I guess that some escapes are missing from the log; they should be represented with their caret notation (^[).

For example, this sequence:

[2;2R

Might actually be:

^[[2;2R

If so, it's probably how the terminal answers to a request about the cursor position. Here, it means that the cursor is on row 2, column 2.

---

Anyway, about this question which was asked previously:

Perhaps a "clear screen" code t_cl appears?

What's the value of the 't_cl' option?

:new | put =&t_cl

For me, it's:

^[[H^[[J

Can you find this sequence somewhere in the log?

[monkoose]

I posted the logs ? symbols are escapes as ^[

[monkoose]

What's the value of the 't_cl' option?

^[[H^[[2J

And yes I can find them in vim and neovim too.

This logs were from just enering and leaving vim and neovim. Should I scroll for some time for relevent results?
Because in current logs this screen clear sequence appears 2 times in both.

[lacygoill]

Should I scroll for some time for relevent results?

Yes, I think you should.

---

I can't help much more, but here are 2 other suggestions.

You can attach the logfiles to a github issue by dragging and dropping the files. This could help preserving special characters, such as raw escapes (and avoid the weird replacement characters �).

Also, you might separate the escape sequences, one per line, with this substitution command:

:% substitute/\ze\e/\r/g

Just to make the logs a little easier to read.

[monkoose]

vim.log
10 screen clears

neovim.log
only 2

And this is how opening popup terminal window with fzf file fuzzy searching and set writedelay=2 looks like in both vim and neovim

vim
https://user-images.githubusercontent.com/6261276/161395203-6a461162-a114-4994-9add-c5f8c6d09965.mp4

neovim

neovim.mp4

Don't say to me you that you don't see the difference @brammool :)
and i guess this jumping curssor is what causing "flickering" to me if we skip slow rendering part.

[monkoose]

@lacygoill @chrisbra Thank You both. At least for learning some new stuff and to understand what is going on.

[brammool]

I looked at only part of the output, but it appears that Vim's strategy is to minimize the text sent to the terminal, while Neovim avoids using the 'screen clear' sequence and instead sends a "clear to end of line' after every line.
At some point the Vim output is "ESC[H" (cursor home) "ESC[2J" (clear screen) and then for each line position the cursor (if there is leading white space) and output the text.
While Neovim does "ESC[H" (cursor home) and prints spaces, then a line of text, then "ESC[K" (clear to end of line) and CR.

As an experiment, you could make a vertical split, which should Vim change strategy to avoid a clear screen.
I wonder what Neovim will do then.

[monkoose]

As an experiment, you could make a vertical split, which should Vim change strategy to avoid a clear screen.

Make a vertical split and do the same thing i did before?

[lacygoill]

Make a vertical split and do the same thing i did before?

I think so, yes. Start Vim with a vertical split (e.g. $ vim -O /tmp/file1 /tmp/file2), then scroll a little to reproduce the issue.

Don't forget to attach new logs to check how the vertical split influences the issue.

[hungpham3112]

vim.log 10 screen clears

neovim.log only 2

And this is how opening popup terminal window with fzf file fuzzy searching and set writedelay=2 looks like in both vim and neovim

vim https://user-images.githubusercontent.com/6261276/161395203-6a461162-a114-4994-9add-c5f8c6d09965.mp4

neovim

neovim.mp4
Don't say to me you that you don't see the difference @brammool :) and i guess this jumping curssor is what causing "flickering" to me if we skip slow rendering part.

I have the same problem. This happen almost daily when I use Fzf.

[monkoose]

script logs for one vertical split of python file with -u NONE vimrc

vim

vim.log

neovim

neovim.log

some visual preview of a terminal screen updates with one vertical split

It shows following step:

- opening the editor itself

- wait till it finishes filling the screen with new content

- and then one <C-f> keypress

vim with set writedelay=4

vim-writedelay-4.mp4

neovim with set writedelay=4

neovim-writedelay-4.mp4

neovim with set writedelay=150

neovim-writedelay-150.mp4

Which differences my eye can see:
Vim writes content of the first split and the border at the same time, then writes content of the second split
updates line character by character (or at least small chunks). Hides cursor when it start updating content, then it clears screen line by line, and then pasting new content line by line character by character, then it shows the cursor.
Neovim differ at how it draws border of the split, first it writes content of the first split, then the border, and then content of the second split. It doesn't hide cursor, but like places it at the end of the line it currently process, it doesn't clear lines of the whole part of the screen that should be updated it just pastes new content line by line (at least i don't see this teletyping that i see in vim (bigger chunks?)).

@brammool I'm not sure how else i can convince you that at leat 50x increase in speed of refreshing a screen is worth to implementing for such a greet editor as vim. My perspective is just from regular user, maybe neovim screen updates are broken on some old hardware or platforms, but maybe it can then be implemented as vim flag option or compiler flag?
I understand that its FOSS you don't own me anything, but human knowledge get outdated, how things used in the past changes too (and if like 10-20 years ago such screen rendering wasn't recognized as a problem, doesn't mean it isn't today). I just don't understand your reaction that you don't recognize such slow screen redraws as a problem at all. If your focus on other things - its understandable, if you currently do not feel to change this behavior it is understandble too, but not recongnizing this as a problem at all just popping questions in my head. That's all.

[brammool]

It appears neovim uses "writedelay" differently, causing most of the difference in speed.

I notice a few things. Vim clears part of the screen by outputting spaces, then draws text in that space, that looks wrong.
Neovim uses CSI nr X to clear the screen, Vim draws spaces. What matters here is that there is no highlighting involved. When there is it might work very differently. Especially when there is a background color. And that is much slower, that is what needs to be optimized for.

The original question was whether a vertical split would avoid the screen-clear. It looks like it does. So did this solve the flickering?

[monkoose]

It appears neovim uses "writedelay" differently, causing most of the difference in speed.

Oh sorry, then my screencasts a almost useless :(

So did this solve the flickering?

Not really, not sure screencasts without writedelay with fps limits could show the difference but let me do some and see myself if it does, so i will then let you know.

Can you also explain a little what is going on with the cursor on my first screencast of popup terminal? Why it jumping around so much?

[monkoose]

@brammool
EDITED: Oh i see some patch, description below for no patch cases

with -u NONE and set nu rnu nowrap ls=2 and syntax on

only one window with <C-f> <C-b> presses or holddowns (for much more frequent flickers)

1. I can't reproduce flickering of statusline (i guess my custom statusline requires more refreshes or something).
2. I can't reproduce flickering of LineNr column

vertical split

1. statusline doesn't flicker
2. Here is intresting thing LineNr column on <C-f>presses it sometimes flicker (better visible on holddowns). But it doesn't flicker with <C-b> even on holddowns.

with custom vimrc and set nu rnu nowrap ls=2

single window

Visible flickering of both LineNr column for both scroll directions and statusline

vertical split

1. No flickering of statusline
2. Same behaviour as with -u NONE it flickers on <C-f>, but doesn't on <C-b>

Can provide logs for all the cases if you are intrested.

[monkoose]

Another question. Do you need logs for opening external tool in terminal popup window that causes inadequate cursor jumps or you are not intrested to fix this issue, just once again using such tool in neovim doesn't cause such abnormal cursor jumps?

[brammool]

> It appears neovim uses "writedelay" differently, causing most of the difference in speed. Oh sorry, then my screencasts a almost useless :(

They are useful to see how the redrawing works. You can't use it to judge performance.

> So did this solve the flickering? Not really, not sure screencasts without writedelay with fps limits could show the difference but let me do some and see myself if it does, so i will then let you know.

The statusline should not be cleared and redrawn now. I'm not sure how serious this flickering is, I haven't really noticed it. Avoiding the screen clear may cause other redrawing to become slow. I know most terminal emulators can clear very fast, while drawing characters takes much more time.

Can you also explain a little what is going on with the cursor on my first screencast of popup terminal? Why it jumping around so much?

Probably the cursor is not disabled while redrawing. For a terminal Vim cannot know how the job in the terminal uses the cursor. The cursor is switched off when drawing, it's back on when it wasn't disabled in the terminal.

…

-- ROBIN: (warily) And if you get a question wrong? ARTHUR: You are cast into the Gorge of Eternal Peril. ROBIN: Oh ... wacho! "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[monkoose]

Didn't test this patch by myself yet, but i appreciate any work that you have done to at least reduce such flickering on some cases.

Probably the cursor is not disabled while redrawing. For a terminal Vim
cannot know how the job in the terminal uses the cursor.

1. I will log this situation with script for better understanding a little bit later today, so better understanding what is going on can be done. So your assumption that this external tool makes cursor jumps around?
2. The weird part that this cursor jumps happens when nothing is going on on the screen nor lines clears nor lines writings. It just happens for some reason before redraws.
3. So nothing can be done from vim side? Because neovim doesn't has this problem with the same tool.
4. I will collect more info and logs and then create new issue, maybe it would help you to trully decide if something can be done from vim's side.

[monkoose]

@hungpham3112 should be fixed with the last fzf commit junegunn/fzf@b3ab631
At least it fixes this flying cursors for me, there is still sublte difference with other a little bit flickering parts compare to neovim, but it is now not annoying anymore. Thanks to Bram that points me out that vim is nothing to do with the cursor in a terminal window.

[brammool]

Is this working well enough now that this issue can be closed?

[monkoose]

Hello, Bram. Not sure if the problem was addressed. Redraw is behaving as documented, so Vim is doing it job right. So I guess this issue is intended behavior of the redraw.

But the problem is that a lot of plugins overuse redraw and redraw! (maybe authors do not understand what exactly it is doing, or with redraws they want to fix something, but at the same time introduce such flickering). And it is hard to address from the user side, and when on Neovim such redraws doesn't do any harm, on Vim they produce a lot of flickering. As example pinned issue of coc.nvim above.
And as some text example a lot of the plugins I know use redraw to just clear command line, to remove press enter message, or just echo multiple consecutive messages. Maybe something could be implemented to clear only command-line.

[brammool]

A way to clear the command line is feedkeys(":\<BS>", 'xt')

Otherwise there is no specific issue remaining, let's close this.
If you can reproduce flicker with an example, feel free to create a new issue.

[brammool]

Another question. Do you need logs for opening external tool in terminal popup window that causes inadequate cursor jumps or you are not intrested to fix this issue, just once again using such tool in neovim doesn't cause such abnormal cursor jumps?

There is support for showing and hiding the cursor in a terminal window, is that external tool using it?

…

-- Sorry, no fortune today. /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[monkoose]

There is support for showing and hiding the cursor in a terminal window, is that external tool using it?

Yes. Currently it is after I have proposed to do so junegunn/fzf#2781