New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance with syntax file (look-behind patterns: limit to length/bytes) #243

Open
blueyed opened this Issue Dec 11, 2014 · 11 comments

Comments

Projects
None yet
8 participants
@blueyed

blueyed commented Dec 11, 2014

From this discussion, Ruby highlight was especially slow after the introduction of the new regex engine, due to frequent use of zero-width look-behind assertions @<! and @<= without using the character-limiting features added to address the problem of their slowness in the new engine (:help /@<= , :help /@<! , scroll down to the part about limiting by bytes).

Looking at https://github.com/vim-ruby/vim-ruby/blob/master/syntax/ruby.vim this issue was never fully addressed, only for one of the occurrences.

Via https://groups.google.com/d/msg/vim_dev/O7GuJkbH9tk/IEMJqWt3SN8J

@chrisbra

This comment has been minimized.

Show comment
Hide comment
@chrisbra

chrisbra Jan 27, 2015

Please also take a look at issue 282 from the vim bugtracker. There are some serious expansive patterns in the syntax (rubySymbol and rubyPredefinedConstant) file. You might want to explicitly use the old backtracing engine for those patterns or try to limit the look-around assertions if it is not possible to further tune those patterns.

chrisbra commented Jan 27, 2015

Please also take a look at issue 282 from the vim bugtracker. There are some serious expansive patterns in the syntax (rubySymbol and rubyPredefinedConstant) file. You might want to explicitly use the old backtracing engine for those patterns or try to limit the look-around assertions if it is not possible to further tune those patterns.

@gpakosz

This comment has been minimized.

Show comment
Hide comment
@gpakosz

gpakosz Jan 8, 2016

Please see vim/vim#282 (comment) where I mentioned rubyConditionalExpression causing lag.

gpakosz commented Jan 8, 2016

Please see vim/vim#282 (comment) where I mentioned rubyConditionalExpression causing lag.

@benoittgt

This comment has been minimized.

Show comment
Hide comment
@benoittgt

benoittgt May 27, 2016

It seems that people from vim/vim forward you here and people on vim-ruby forward you to vim...

But what can we do?

benoittgt commented May 27, 2016

It seems that people from vim/vim forward you here and people on vim-ruby forward you to vim...

But what can we do?

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed May 27, 2016

@benoittgt
I've created the issue for vim-ruby, because it can be fixed here - although I do not use it myself (I've just stumbled upon it back then).
(Unsubscribing from this issue)

blueyed commented May 27, 2016

@benoittgt
I've created the issue for vim-ruby, because it can be fixed here - although I do not use it myself (I've just stumbled upon it back then).
(Unsubscribing from this issue)

@bluz71

This comment has been minimized.

Show comment
Hide comment
@bluz71

bluz71 May 28, 2017

This is still a genuine issue. Same goes for Vim 282.

I have relativenumber set and scrolling RSpec files was borderline unusable on a CoreM powered Macbook (aka a slower Mac) using iTerm2 with both Vim 8 and Neovim.

Setting lazyredraw slightly mitigates this issue, but it has other negative effects so I do not set it.

Setting set regexpengine=1 at the global level fixed the issue for me. Note, setting regexpengine=1 in an autocmd FileType ruby resulted in no performance difference, only setting the regexpengine at the top level worked. Why is that? Weird.

Anyway, vim-ruby clearly has performance issues with relativenumber and the newer Vim regexpengine (which is now the default). Doing some Googling seems to indicate that many Ruby folks are experiencing this issue

Maintainers, please test vim-ruby with both the old and new regexpengine (on slow machines), there is a clear performance regression with some vim-ruby rules when run against the new engine.

bluz71 commented May 28, 2017

This is still a genuine issue. Same goes for Vim 282.

I have relativenumber set and scrolling RSpec files was borderline unusable on a CoreM powered Macbook (aka a slower Mac) using iTerm2 with both Vim 8 and Neovim.

Setting lazyredraw slightly mitigates this issue, but it has other negative effects so I do not set it.

Setting set regexpengine=1 at the global level fixed the issue for me. Note, setting regexpengine=1 in an autocmd FileType ruby resulted in no performance difference, only setting the regexpengine at the top level worked. Why is that? Weird.

Anyway, vim-ruby clearly has performance issues with relativenumber and the newer Vim regexpengine (which is now the default). Doing some Googling seems to indicate that many Ruby folks are experiencing this issue

Maintainers, please test vim-ruby with both the old and new regexpengine (on slow machines), there is a clear performance regression with some vim-ruby rules when run against the new engine.

bluz71 added a commit to bluz71/dotfiles that referenced this issue May 28, 2017

Ruby performance is terrible with regexpengine=2, force regexpengine=1.
See these two issues:
  vim/vim#282
  vim-ruby/vim-ruby#243

Ruby syntax highlighting performance is much MUCH worse the new Vim
regexpengine, hence force old engine until a fix is forthcoming (if
ever).
@tpope

This comment has been minimized.

Show comment
Hide comment
@tpope

tpope May 28, 2017

Member

Friendly reminder that maintainers are not your personal servants. If you wanna get this fixed, provide a patch or a paycheck, or at the very least do a fresh assessment of which patterns are the bottleneck.

Member

tpope commented May 28, 2017

Friendly reminder that maintainers are not your personal servants. If you wanna get this fixed, provide a patch or a paycheck, or at the very least do a fresh assessment of which patterns are the bottleneck.

@bluz71

This comment has been minimized.

Show comment
Hide comment
@bluz71

bluz71 May 29, 2017

Agreed, I do genuinely apologise if offence was taken, it was not the intention, though rereading my post I can see how that may have been the case.

I will try and make myself useful and I will provide as many details as possible soon: syntimes, CPU usage and possibly which patterns are the expensive ones. Details to follow.

My gut feeling is that vim-ruby syntax highlighter is triggering some expensive CPU processing with the newer NFA engine. Folks with fast CPUs are likely not seeing the issue, those with slower machines notice a huge performance issue. The old regexp engine is likely much lighter on CPU.

The first posts of vim issue 282 perfectly detail the issue.

bluz71 commented May 29, 2017

Agreed, I do genuinely apologise if offence was taken, it was not the intention, though rereading my post I can see how that may have been the case.

I will try and make myself useful and I will provide as many details as possible soon: syntimes, CPU usage and possibly which patterns are the expensive ones. Details to follow.

My gut feeling is that vim-ruby syntax highlighter is triggering some expensive CPU processing with the newer NFA engine. Folks with fast CPUs are likely not seeing the issue, those with slower machines notice a huge performance issue. The old regexp engine is likely much lighter on CPU.

The first posts of vim issue 282 perfectly detail the issue.

@bluz71

This comment has been minimized.

Show comment
Hide comment
@bluz71

bluz71 May 29, 2017

Results,

  • 2015 Macbook with slow CoreM processor (1.1GHz)
  • Vim 8 in iTerm2 (3.0.15) with tmux (2.4) (note, Neovim 0.2 has the same issue)
  • Latest vim-ruby
  • relativenumber is set (this is the main trigger)
  • foldmethod=indent (not syntax which I know is expensive)

syntime on / syntime report produced no meaningful culprit between my fast mode (regexpengine=1) or my slow mode (regexpengine=0 or 2).

CPU results:

  • regexpengine=1 resulted in MAX CPU usage of 85% (usually around 80%)
  • regexpenging=2 resulted in MAX CPU usage of 100% (usually around 96%)

There is no single bad rule in the vim-ruby syntax highlighter. Basically you have to comment out half the file to get reasonable performance. Even enabling let ruby_no_expensive = 1 had no meaningful performance benefit on my rig.

On my slow Macbook regexpenging=1 does not max the CPU hence why it has such a massive benefit for me. However if vim-ruby had a few more rules then even that old engine would be cooked I suspect.

I think the issue is the number of rules in vim-ruby in combination with Vim not-caching syntax results when relativenumber is enabled AND a slow CPU. Disabling relativenumber is a fix (but one I am not willing to do since relativenumber is so useful); setting lazyredraw somewhat addresses the issue as well (but it causes other worse problems).

I believe the best solution lies with a fix in Vim / Neovim itself for relativenumber or cursorline modes, that being to cache syntax highlighting and not recalculate when scrolling. The recalculation is the killer. Bram himself suggested it 18 months ago here.

I don't think vim-ruby maintainers can really do that much to fix this open issue except to encourage Bram and the Neovim teams to cache and cache some more. Luckily Bram himself suggested the actual fix.

bluz71 commented May 29, 2017

Results,

  • 2015 Macbook with slow CoreM processor (1.1GHz)
  • Vim 8 in iTerm2 (3.0.15) with tmux (2.4) (note, Neovim 0.2 has the same issue)
  • Latest vim-ruby
  • relativenumber is set (this is the main trigger)
  • foldmethod=indent (not syntax which I know is expensive)

syntime on / syntime report produced no meaningful culprit between my fast mode (regexpengine=1) or my slow mode (regexpengine=0 or 2).

CPU results:

  • regexpengine=1 resulted in MAX CPU usage of 85% (usually around 80%)
  • regexpenging=2 resulted in MAX CPU usage of 100% (usually around 96%)

There is no single bad rule in the vim-ruby syntax highlighter. Basically you have to comment out half the file to get reasonable performance. Even enabling let ruby_no_expensive = 1 had no meaningful performance benefit on my rig.

On my slow Macbook regexpenging=1 does not max the CPU hence why it has such a massive benefit for me. However if vim-ruby had a few more rules then even that old engine would be cooked I suspect.

I think the issue is the number of rules in vim-ruby in combination with Vim not-caching syntax results when relativenumber is enabled AND a slow CPU. Disabling relativenumber is a fix (but one I am not willing to do since relativenumber is so useful); setting lazyredraw somewhat addresses the issue as well (but it causes other worse problems).

I believe the best solution lies with a fix in Vim / Neovim itself for relativenumber or cursorline modes, that being to cache syntax highlighting and not recalculate when scrolling. The recalculation is the killer. Bram himself suggested it 18 months ago here.

I don't think vim-ruby maintainers can really do that much to fix this open issue except to encourage Bram and the Neovim teams to cache and cache some more. Luckily Bram himself suggested the actual fix.

@tpope

This comment has been minimized.

Show comment
Hide comment
@tpope

tpope May 29, 2017

Member

If it's fine without 'relativenumber' then I agree, it's not something that can be solved with changes to vim-ruby.

Member

tpope commented May 29, 2017

If it's fine without 'relativenumber' then I agree, it's not something that can be solved with changes to vim-ruby.

@bluz71

This comment has been minimized.

Show comment
Hide comment
@bluz71

bluz71 May 30, 2017

Tim,

relativenumber is key here, when enabled each scroll event (up or down) forces a full syntax highlight cycle for every visible line since each line subtly changes (in the number column).

vim-ruby syntax has about 240 regexps, multiply by the number of visible lines results in a lot of syntax work when relativenumber is enabled. When relativenumber is not enabled, no issue at all.

I have spawned a new feature request over in the Vim repository, that being syntax highlight caching, a suggestion Bram himself noted in #282. I suspect it is not an easy request to implement, so may not happen.

As for here, I doubt there is anything that can be done in vim-ruby repository to address this specific issue. The fix needs to occur upstream.

Lastly, I do appreciate all the work maintainers do, thank you.

bluz71 commented May 30, 2017

Tim,

relativenumber is key here, when enabled each scroll event (up or down) forces a full syntax highlight cycle for every visible line since each line subtly changes (in the number column).

vim-ruby syntax has about 240 regexps, multiply by the number of visible lines results in a lot of syntax work when relativenumber is enabled. When relativenumber is not enabled, no issue at all.

I have spawned a new feature request over in the Vim repository, that being syntax highlight caching, a suggestion Bram himself noted in #282. I suspect it is not an easy request to implement, so may not happen.

As for here, I doubt there is anything that can be done in vim-ruby repository to address this specific issue. The fix needs to occur upstream.

Lastly, I do appreciate all the work maintainers do, thank you.

mrhead added a commit to mrhead/dotfiles that referenced this issue Jan 30, 2018

Disable relativenumber
It is so slow in combination with vim-ruby that I want to kill myself
when I'm working with ruby files.

:(

References:

- vim-ruby/vim-ruby#243
- vim/vim#282

GabeIsman added a commit to GabeIsman/dotfiles that referenced this issue Feb 22, 2018

Revert to old regexp engine
Recently after updating vim and several plugins, my experience of
editing large ruby files became unacceptably slow. After some
frustrating debugging I found suggestions to turn off cursorline and
relativenumber since these cause frequent redrawing. This really just
masks the problem though, which turned out to be ruby syntax
highlighting being crazy slow. With syntax=off everything worked
perfectly. With syntax on even inserting a line was noticeably laggy as
vim redrew the rest of the file.

Setting regexpeengine=1 at the global level seems to largely resolve the
issue.

vim-ruby/vim-ruby#243 (comment)
@juanibiapina

This comment has been minimized.

Show comment
Hide comment
@juanibiapina

juanibiapina May 21, 2018

For me it's not fine without relativenumber. It's better, but still slow.

Using regexpengine=1 makes it as fast as neovim (which doesn't have that problem regardless of the choice of engine o.O).

juanibiapina commented May 21, 2018

For me it's not fine without relativenumber. It's better, but still slow.

Using regexpengine=1 makes it as fast as neovim (which doesn't have that problem regardless of the choice of engine o.O).

kejadlen added a commit to kejadlen/dotfiles that referenced this issue Jul 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment