Highlight definition for go lags nvim on files > 50K

[leoluz]

Describe the bug

I work on the Argo CD project which is mainly Go development. I noticed an increasing lag (moving the cursor around the file) depending on the file size for a long time and decided to narrow down the plugin causing the issue. It turns out that the issue is caused by nvim-treesitter-refactor if configured with highlight-definitions. I ran a small benchmark on the project files and came up with the following:

- 24K     server/repository/repository_test.go (works ok)
- 32K     cmd/argocd/commands/project.go (works ok)
- 52K     pkg/apiclient/project/project.pb.gw.go (noticeable lag starts)
- 100K    pkg/apiclient/repository/repository.pb.go (impractical)
- 124K    pkg/apiclient/application/application.pb.gw.go (freezes nvim)
- 1.1Mb   pkg/apis/application/v1alpha1/generated.pb.go (freezes nvim) 

To Reproduce

1. Clone argocd repo:

   git clone https://github.com/argoproj/argo-cd.git

2. Configure Neovim with nvim-treesitter and nvim-treesitter-refactor and enable highlight-definitions for Go syntax.

3. Open Neovim with Go files bigger than 50K

nvim ./argo-cd/pkg/apiclient/project/project.pb.gw.go  < noticeable lag
nvim ./argo-cd/pkg/apiclient/repository/repository.pb.go < unusable lag

Expected behavior

I would be great if highlight-definitions could run in a non-blocking (nvim ui) fashion. I found some past discussion about this but it seems that running it async wasn't implemented in that occasion. However I am not sure if async in that discussion would result in a non-blocking experience.

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()

Installation ~
- WARNING `tree-sitter` executable not found (parser generator, only needed for :TSInstallFromGrammar, not required for :TSInstall)
- OK `node` found v18.16.0 (only needed for :TSInstallFromGrammar)
- OK `git` executable found.
- OK `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
  Version: gcc (GCC) 12.2.0
- OK Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

OS Info:
{
  machine = "x86_64",
  release = "5.13.0-valve37-1-neptune",
  sysname = "Linux",
  version = "#1 SMP PREEMPT Thu, 17 Aug 2023 23:47:04 +0000"
} ~

Parser/Features         H L F I J
  - bash                ✓ ✓ ✓ . ✓
  - c                   ✓ ✓ ✓ ✓ ✓
  - go                  ✓ ✓ ✓ ✓ ✓
  - html                ✓ ✓ ✓ ✓ ✓
  - javascript          ✓ ✓ ✓ ✓ ✓
  - json                ✓ ✓ ✓ ✓ .
  - lua                 ✓ ✓ ✓ ✓ ✓
  - markdown            ✓ . ✓ ✓ ✓
  - markdown_inline     ✓ . . . ✓
  - python              x ✓ ✓ ✓ ✓
  - query               ✓ ✓ ✓ ✓ ✓
  - vim                 ✓ ✓ ✓ . ✓
  - vimdoc              ✓ . . . ✓
  - yaml                ✓ ✓ ✓ ✓ ✓
  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang} ~

The following errors have been detected: ~
- ERROR python(highlights): ...vpUJ/usr/share/nvim/runtime/lua/vim/treesitter/query.lua:273: query: invalid node type at position 727 for language python
  python(highlights) is concatenated from the following files:
  | [ERROR]:"/home/deck/.local/share/nvim/lazy/nvim-treesitter/queries/python/highlights.scm", failed to load: ...vpUJ/usr/share/nvim/runtime/lua/vim/treesitter/query.lua:273: query: invalid node type at position 727 for language python

### Output of `nvim --version`

```text
nvim --version
NVIM v0.10.0-dev-953+g1635c9e75
Build type: RelWithDebInfo
LuaJIT 2.1.1692716794
Run "nvim -V1 -v" for more info

### Additional context

Huge thanks to everyone involved in Treesitter. Awesome work!

[lucario387]

Does the lag here means lag while editing the file, or the initial lag while opening the file? Because the latter is kindof unavoidable, while the former is fixed(somewhat) on neovim nightly: neovim/neovim#25109, neovim/neovim#25115. Can you try a more up-to-date version of neovim?

Edit: I'm using nightly and it's still lagging. I honestly don't know what to do in this case 🤔

[b0o]

I am experiencing significant lag in TypeScript files while treesitter highlighting is enabled. Perhaps this is related?

Opening the file is nearly instantaneous, the lag doesn't start until I try to make certain edits. Doing something like entering insert mode and backspacing an opening { with a lot of nested code is an example of something that often causes the lag. It's not always so obvious, though - for example I just tried to change the path of an import statement and my editor completely hung.

I'm not using nvim-treesitter-refactor, but if my issue is related to this one, the common denominator seems to be highlighting. If I :TSDisable highlight, the lag goes away.

I am using the latest neovim nightly and the latest versions of treesitter and parsers. I'm happy to help debug or provide more info.

[llllvvuu]

Have y'all tried uninstalling the comment parser? #5057

(IDK if this is related, if it's about brackets and not //s)

Can anyone upload a file that can we can do profiling on? init.lua would be good too but I've been debugging tree-sitter perf lately and could probably figure that part out.

[lucario387]

Just treesitter highlighting with the files in this issue is enough to reproduce it for me.

[leoluz]

Does the lag here means lag while editing the file, or the initial lag while opening the file?

@lucario387 I clarified in the issue description.. It was while editing the file. Tks for pointing it out.

I am experiencing significant lag in TypeScript files while treesitter highlighting is enabled. Perhaps this is related?

@b0o not sure if this is directly related as we are talking about 2 different plugins (nvim-treesitter and nvim-treesitter-refactor). Maybe you are right as they are both using the same API at the end. It would be awesome if treesitter somehow had a non-blocking API.

Just treesitter highlighting with the files in this issue is enough to reproduce it for me.

@lucario387 Just navigating the biggest file (1.1M) with treesitter highlight enabled and highlight definition disabled is actually fine as long as I don't try to edit the file. If I try to do any modification the lag starts.

Have y'all tried uninstalling the comment parser?

@llllvvuu I don't think comment parser is the issue in my case. The pkg/apis/application/v1alpha1/generated.pb.go is a gigantic protobuf generated Go file with almost no comments in it. (sorry.. misunderstood what you meant by comment parser) Unrelated in my case as I don't use this parser.

[bugabinga]

@llllvvuu uninstalling comment parser fixed the issue for me.
the file I tested it on is https://gist.github.com/bugabinga/730dea632b94b2c54c48bd4eabf31481

[jamestrew]

I'm also having this issue with lua files with a large number of comments (anything more than ~300 lines of comments).

• The issue is strictly related to high comment line count
• I don't have treesitter-refactor installed
• I don't have the comment parser installed.
• Doing :TSBufDisable highlight fixes the issue
• I've bisected this to this commit on neovim nightly neovim/neovim@908843d

Funny thing is, I spent around 2 hours trying to make a minimal config to replicate this but I'm not able to. But doing { highlight = { enabled = false } } in my ts config fixes the issue.

I also tried reinstalling parsers.

[llllvvuu]

@jamestrew This is going to be fixed in neovim/neovim#25201 (which currently has a typo "integer" should be "number"). The effect you're seeing is the opposite of what was hypothesized, i.e. what's happening is that you don't have the comment parser installed and Neovim keeps trying to search your filesystem for it.

[wongjiahau]

Assuming the root cause is that the whole file is being passed to the Treesitter highlight function, I recently found a solution with a slight drawback.

The solution is to pass only the contents visible on the screen to the function instead, this can cause some incorrectness in the highlights, but surprisingly the errors are negligible, if not inexistent most of the time.

With this approach, the performance remains the same regardless of the size of the file, the only factor is the terminal size.

[clason]

Shouldn't this issue be reported at nvim-treesitter-refactor?