Tree-sitter discussion

[bfredl]

As a common question is what the status of tree-sitter is, I figure it would make sense with a tracking issue.

General:

• implement some kind of standard search path for tree-sitter grammars (compiled .so files) treesitter: add standard search path for parsers + bundle c parser properly #11757
• eliminate the need for users to compile their own grammars, by either:
  • ship common parsers as part of our binary packages
  • alternatively host these somewhere and provide a download mechanism (for common platforms)
• implement some demo plugin for text objects or folding or something simple, to help people get started : nvim-treesitter

Highlight specific:

• add a mechanism/search path for runtime and third-party highlight definitions
• consider a standard macro/template system for developing highlight queries (could involve fennel :)
• support @match predicate (by proper lua regex API) add regex support in treesitter queries #11880
• add basic matching of local variables, like tree-sitter highlight crate does
• Implement a proper lua-callback highlight interface based on "decorations" concept.
• Reduce excessive multi-line redraws by using byte-based buffer updates (partialy done in [RDY] Bytetrack #12568)
• support nested languages and language "blocks" (like lua, python blocks within vim script, markdown code blocks etc).
  • when we have extranges, such a region could be defined by an extrange

[JasonWoof]

Hi all,

I'm occasionally overcome with a burning desire for async coloring in vim.

I've hacked up something that I hope could be useful, or at least interesting to some of you.

I coded an experiment to see if it might be possible/practical to do this with stock vim and some vimscript to communicate with an external process that runs treesitter.

Currently it's pretty basic, but it colors!

If you have a large buffer, then vim continues to be responsive while the buffer is sent and parsed, but there's a lag when the vimscript applies the colors. In many cases this could be fixed by not applying all the colors immediately.

I've found vimscript functions to figure out which lines are on the screen, and to clear coloring from certain lines, but I don't see any functions for removing coloring from character ranges, so I'm not sure the current vim api is sufficient to gracefully handle cases with very long lines.

I've been testing in vim8.1 as shipped by debian unstable. I think/hope it'll work just as well in neovim.

https://github.com/JasonWoof/vim-treesitter

Enjoy! - Jason

[kyazdani42]

Hi,
first of all, thanks for all this amazing work !

I would like to know if you could give information on how to build our own parsers for the time being.

I tried multiple repositories under the treesitter organization, but they only build statically linked libraries, which cannot be loaded with vim.treesitter.require_language.

I started to play around with lua and its quite a great language to use ! I hope we'll see more cool stuff like that in the future !

[bfredl]

@kyazdani42 depends on your system, but on a POSIX-like system you can clone any language repo and run gcc -o parser.so -shared src/parser.c -Os -I./src . If there is a file like src/scanner.c or src/scanner.cc you might need to add it as well.

Then copy parser.so to parser/LANG.so in the runtimepath, where LANG should be replaced by the language name.

[kyazdani42]

perfect i wasn't sure which file to compile ! thanks

[resolritter]

There might be some weird interaction between the highlights from inccommand and the lua_attr attributes and/or TSHighlighter being spammed non-stop while ex mode is active.

I have a mapping which does something like this:

nmap <expr> <key> :s/<C-R><C-W>/

i.e. prepopulate the command line with the substitution command for the word under the cursor.

I had both inccommand and TSHighlighter active for that buffer and this happened about 3 hours into the coding session.

---

When I tried to type anything after executing the mapping, in the command line

:s/WORD/P <-

I noticed the editor simply froze and memory started to spike up massively. I finally got to exit after mashing <c-c> a few times, but then tried it again for good measure; the second time, memory had rose up to about 16GB.

Something I noticed is that the screen did not redraw after those freezes, since I did not see the P being replaced as an effect of inccommand.

If I'm not mistaken, I got to use that same mapping a few times without running into this issue, so I can't reproduce it 100% of the time on my end (if ever again).

[resolritter]

Unrelated to the previous issue, but something I also noticed after using the highlighter massively for about a day: making edits (presumably most frequently on insert mode) will mess up the highlighting.

When I load up the file, it seems to be rendered perfectly according to the rules.

However, say you have

const foo = #barBaz#

where the # denotes the highlighted area. Sometimes after editing the line in insert mode, the highlight won't shift together with the identifier, resulting in e.g.

const fooBar # = barB#az

Another issue: two things being highlighted better if they were far apart, but not one after the other. For instance, the following would not render correctly: only the first line and the ones above it would be fine, everything below would be messed up

import { foo } from "module"
import { bar } from "module"
(messed up stuff...)

but if you added a line between them, it would get a little better

import { foo } from "module"

import { bar } from "module"

Redoing the same actions would result in the exact same effects, which I think hints at the tree being somehow corrupted by that point. Similar to this, it'd sometimes notice things being highlighted inside of comments, like somehow two attributes were merged for the same row and column.

[bfredl]

yes, byte-resolution updates need to be improved. I plan to do a major refactor soon.

[resolritter]

Today I tried another approach

Only rendering it in specific modes

local mode = api.nvim_get_mode()
if mode.blocking or (mode.mode ~= "n" and mode.mode ~= "v") then
    return
end

Not redrawing in case the tick didn't change

local tick = api.nvim_buf_get_changedtick(buf)
if tick == _G.buf_ticks[buf] then
    return
end

Reparsing the entire buffer every time

  local lines = api.nvim_buf_get_lines(buf, 0, -1, false)
  local parser = vim.treesitter.get_parser(0, lang)
  local tree = parser:parse()
  for cid, node in self.query:iter_captures(tree:root(),buf,0,#lines) do

Using the nvim_buf_add_highlight API for coloring the buffer

api.nvim_buf_clear_namespace(buf, _G.treesitter_namespace, 0, -1)
-- other stuff...
api.nvim_buf_add_highlight(buf, _G.treesitter_namespace, hl_group, start_row, start_col, next_end_col)

While it seemed to fare a little bit better in some regards, I eventually ran into the same problems as the current runtime's approach

• Highlights being inconsistent after e.g. re-formatting the document (ALE uses setbufline). However, unlike the first approach, it seemed like the highlighting got fixed when I deleted the bugged visual range and undid it after.
• I ran into the same bug of my RAM getting completely eaten out and the computer freezing after some hours of usage. This time I did not have to do anything special for it to happen, so I think the idea of it interacting with "inccommand" loses strength. I'm unsure if it's actually a bug in the implementation I have because I don't presume an infinite loop would eat 16GB of RAM, but who knows...

[resolritter]

@bfredl do you think it would make a difference, on the inconsistency issue, if I debounced the callback and ran the highlighting function on the trailing end of the timer? I'm thinking that it gets messed up sometimes because it's too eager, but I don't know...

[bfredl]

Well, if the problems arise at certain types of edits, then the buffer update is likely wrong (tree-sitter already uses more info than the rplugin buffer updates, even if it is not proper byte-level updates yet).

Debouncing would only help with performance, not with correctness. It is possible to set a deadline for the parsing, so a large reparse does not freeze the editor. We should do this eventually, but it is unrelated to redraw bugs.

[maxbrunsfeld]

@bfredl @justinmk Just a follow up to our conversation in #10124 (comment) - there is now a Makefile in the Tree-sitter repository, which I believe conforms to the interface that your build system expects. See tree-sitter/tree-sitter#602.

[maxbrunsfeld]

@bfredl Also, the syntax for tree queries is changing slightly in tree-sitter/tree-sitter#615. We will be updating the queries in grammar repos like tree-sitter-c, so I'm not sure if this affects you, but I thought I'd let you know because you were testing the query API early on.

[vigoux]

Hello everyone, lately I have been working on tree-sitter integration.

I just wanted to give some news on this. I (and many others) started working on some items in the list at the top of this issue :

• Implement a proper lua-callback highlight interface based on "decorations" concept : [RDY] Treesitter: fix highlight, attempt 2 #12590
• add a mechanism/search path for runtime and third-party highlight definitions : treesitter: add runtime queries feature #12639
• implement some demo plugin for text objects or folding or something simple, to help people get started : https://github.com/nvim-treesitter/nvim-treesitter

[tristan957]

@vigoux is tree-sitter going to be in neovim natively in 0.5 or will it always live as a separate extension?

[vigoux]

@tristan957 as I am not part of the core team, I can't tell anything about it and I don't have any clue. My personal goal though is to bring (at least for C) some kind of built-in, and easy-to use highlighting (cc @tjdevries).

[tristan957]

Awesome. I'll install it on my dev machine. Thanks for your work :).

[vigoux]

Personally I use nvim-treesitter on a dayly basis, but highlighting has some (pretty big tbh...) issues, they are being worked on though.

[tjdevries]

@tristan957 I'm not sure I understand your question. Tree sitter support is built-in and compiled into Neovim -- it would not be possible to use otherwise.

I'm not sure about long term support for various different languages and how/if we'll ship them, but I just wanted to at least clarify the question there.

[tristan957]

TJ thanks for the clarification. I was naiive when I first entered this issue, but now I understand the relationship between the package and neovim much better.

[tjdevries]

Good :)

As a personal desire (read: not necessarily as a goal of the core team) I think it would be cool to ship at least C, Vim & Lua treesitter support natively with neovim. For a couple of reasons:

1. It shows off what treesitter can do
2. It gives people the "standards" of what we want to do with treesitter in neovim
3. Those are the main languages required to develop for neovim, so it'd be fun to be able to have them all very well supported.
4. They could provide experimental playgrounds for highlighting, folding and completion that could work well with no external dependencies.

I think we could ship the various associated query files for these languages that nvim-treesitter has been standardizing as well.

As I mentioned, this is just a personal preference at the moment I have been thinking about recently, made possible by @vigoux great work 😁

[vigoux]

The vim treesitter parser does not exists (yet) fully, but I have been putting up some work here.
Here is the roadmap I see given your thoughts :

1. Move treesitter as a bundled dependency (that's not the case today)
2. Build and bundle the Lua parser
3. Move the queries from nvim-treesitter to here (there is issues with lua highlight though, need to investigate)
4. Provide an lsp-like omnifunc (we might look into completion-treesitter for that)
5. Provide a foldexpr function (nvim-treesitter has one working)

Thanks for the kind words !

[theHamsta]

I feel not comfortable about 3.

Move the queries from nvim-treesitter to here (there is issues with lua highlight though, need to investigate)

The configurations for nvim-lsp also have their own repo which can move quicker that polluting the neovim repo. Maybe we can have a minimum everyone can agree on here and update it every half year or so. The vim highlights are also quite minimalistic and almost never change and people use plugins for additional fanciness.

[euclidianAce]

are there any plans to separate out the lua tree-sitter bindings to their own module? as lua doesn't really have official tree sitter bindings it would be nice to have it as a module on luarocks or something as an officially endorsed binding to tree-sitter. I'm wanting to write some tooling in lua with tree-sitter, but would rather build on top of some work already done rather than rewrite my own bindings from scratch.

[vigoux]

Hi, there is effectively at least the willing to build out of tree Lua bindings.
We just did not had time to do so but I think that we tj ught about using LuaNativeObjects to generate everything.