[proposal] Introducing TextMate grammar system for syntax highlighting ?

[skywind3000]

Current problem

The current syntax highlighting system is very slow, and there are noticeable lags when scrolling large C++ files which contain complex syntax elements.

Issues of tree-sitter

Previously, most people suggest something like nvim-treesitter which will analyze source code in a background treesitter process and render keywords in the foreground with text-property.

But is it a good idea? I don't really think so,
at least 4 disadvantages for treesitter solutions:

• power consumption: an extra background job is required, causing less battery life and more carbon dioxide.
• buffer sync: code must be written very carefully to guarantee that the source code in two processes (vim/treesitter) are the same, coc/nvim-treesitter sends the whole buffer to the background every time changetick increase to prevent such things, which is a little flaky.
• reliability: an external program installed by the user is not reliable enough, they are plenty of issues in version compatibilities and environment misconfig, people need to take extra efforts to get syntax highlighting work.
• poor parser quality: treesitter is a great project, but language parsers are contributed by people all over the world and their quality is not under control (performance issues or inaccurate results in certain languages).

Background syntax highlighter is still immature, there are still many other strange issues in nvim-treesitter:

https://github.com/nvim-treesitter/nvim-treesitter/issues

If we introduce something like this, we shall take all these issues into account.

TextMate grammar system

Syntax highlighting is the most important part of an editor, better not rely on any uncontrollable external programs.

We need some new things that can satisfy such goals below:

• good performance
• robust and reliable
• accuracy
• low power consumption
• native and work in the same process (not require external programs)

And TextMate's grammar engine is really a good candidate which is widely used in many IDE/editors, including vscode (see syntax-highlight-guide for details), sublime and many others.

VS Code uses TextMate grammars as the syntax tokenization engine. Invented for the TextMate editor, they have been adopted by many other editors and IDEs due to the large number of language bundles created and maintained by the Open Source community.

TextMate grammars rely on Oniguruma regular expressions and are typically written as a plist or JSON. You can find a good introduction to TextMate grammars here, and you can take a look at existing TextMate grammars to learn more about how they work.

The grammar can be defined in JSON, that means can be translated into viml or just plain JSON files.

Possible Solution

We can specify which grammar engine to use for the given buffer:

• default engine: current vim's regex grammar
• textmate engine: textmate grammar system.

And some new command can be used to change grammar engine:

:syntax grammar textmate
:syntax grammar default
:syntax load ~/.vim/syntax/cpp.json

for example, the snippet below can be included in the head of syntax files:

if has('textmate')
    syntax grammar textmate
    syntax load syntax/cpp.json
    finish
endif

....

And lots of existing vscode/textmate syntax files can be reused with minimal modification.

[brammool]

Thank you for starting this discussion. I had a vague plan to look into integrating treesitter, it is good to know it also has disadvantages. Vscode is widely used, thus if it uses TextMate then there must be something good about it.

Comments welcome.

[clason]

I'll just comment that I would take these comments about tree-sitter with a significant heap of salt.

[bfredl]

The might be some misunderstanding here. Tree-sitter in neovim doesn't use an external process like coc.nvim. The parser runtime is a C library embedded into the editor itself (in total not more LOC:s than syntax.c + highlight.c in vim itself), and parses the buffer in memory without copy and produces a syntax tree that in-process plugins can use (for highlighting but also for other purposing like text objects).

[imranZERO]

Right now the biggest problem with syntax highlighting is how inconsistent and unpredictable it is among different languages. An unified interface will be more than worth the effort.

TextMate will probably be better for keeping the syntax system more integrated & backwards compatible than using something like treesitter. Also the modular and overengineered plugin architecture of treesitter would be a huge departure from the way it is done right now, so we should be a little cautious about how much functionality to reimplement.

[bfrg]

@bfredl How much longer does it take to load a larger file like src/evalfunc.c in Neovim when tree-sitter is enabled, compared to the default syntax highlighting? I'm assuming that default syntax highlighting is disabled for filetypes where tree-sitter is supported.

[bfredl]

@bfrg src/evalfunc.c from vim (10 000 lines) takes 80 ms more time with tree-sitter enabled for the initial parse (--startuptime 200ms compared to 120ms in my config)

[mg979]

treesitter is more than just syntax highlighting, it's also useful for text objects for example.

TextMate system is old, Sublime Text has been mentioned but it left it years ago to use its own syntax engine. Does it make sense to adopt a system that is already waning? And how big is its library if it must be included?

Also when saying that a system is more performant, some source/benchmark should be provided. Is it TextMate more performant than treesitter? Who says so?

[skywind3000]

@bfredl , thanks for figuring it out, and I made a new revision:

list of tree-sitter disadvantages for syntax highlighting:

• power consumption: yes tree-sitter is powerful, it will generate AST in real-time, but I am just talking about syntax highlighting, not textobj or indentation. Do we really need an AST for just syntax highlighting ? If someone really care about the semantic highlighting, they can use coc & lsp. AST generation has its price (see Horrible input lag on 2MB C header file. nvim-treesitter/nvim-treesitter#1292).
• reliability: nvim-treesitter need to load an external shared library as the parser for each language, the shared library must be downloaded and compiled into .so files (I know :TSInstall can simplify these steps), building progress can break if gcc/clang is not installed, the plugin or neovim itself may break due to any common dynamic link library problems, eg: version incompatible when the plugin has updated but parser .so files not, dependency conflict when loading the shared library.
• poor parser quality: tree-sitter is a great project, but language parsers are contributed by people all over the world and their quality is not under control (performance issues or inconsistent behavior in different languages).

The biggest risk is the horrible parser quality, over 100+ open issues for parsers:

• https://github.com/nvim-treesitter/nvim-treesitter/issues?q=is%3Aissue+is%3Aopen+parser

examples for quality issues:

• Python treesitter parser should never be enabled doom-neovim/doom-nvim#223
• Python and Vim highlighters are broken nvim-treesitter/nvim-treesitter#1976
• cpp highlighter is broken on macOS 11.6 with newest nightly build neovim. nvim-treesitter/nvim-treesitter#1968
• Python: base type inside a multi level type square bracket is detected as variable nvim-treesitter/nvim-treesitter#1899
• HTML/svelte syntax highligh disappears when scrolling a large file nvim-treesitter/nvim-treesitter#1788
• JS template literals in HTML are not highlighted as JS when in HTML string nvim-treesitter/nvim-treesitter#1688
• Indent: In C/C++, o doesn't work in else statements nvim-treesitter/nvim-treesitter#1568
• Hangs nvim uncancellably when encounters medium-sized binary files nvim-treesitter/nvim-treesitter#1530
• Inconsistent highlighting for JSX opening elements and class identifiers nvim-treesitter/nvim-treesitter#1547
• highlighting disappears after performing a sort nvim-treesitter/nvim-treesitter#1250

examples for performance issues:

• Horrible input lag on 2MB C header file. nvim-treesitter/nvim-treesitter#1292
• Input lag on large javascript files nvim-treesitter/nvim-treesitter#1313
• terrible elixir performance nvim-treesitter/nvim-treesitter#1957
• Slow tsx and typescript nvim-treesitter/nvim-treesitter#1396

The parser quality problem is totally out of control, nearly impossible for us to fix all the parsers one by one.

It appears that the performance issues of traditional vim syntax highlighting are not fixed by tree-sitter, but more issues are introduced by it. Therefore, syntax highlighting should not rely on such unreliable parsers.

One more thing, parsers are not only hard to implement but also hard to extend/improve. most of the time, only the author can do this. But json syntax files are much easier for everyone.

One last thing, parser binaries are big in size:

The average size of each parser is around 500KB, vim 8.2.3582 has 644 syntax files, that means you need extra 322MB disk space to save the parse binaries.

Vim is shipped every where, including systems that do not have gcc/clang compilers installed, and systems with limited storage space, that means it is hard to build the parsers on demand or ship the pre-build binaries with vim itself.

[skywind3000]

@mg979 For modern editors, regex based syntax highlighting is still the foundation, and semantic highlighting is just the decoration. I am talking about the foundation not the decoration.

Also when saying that a system is more performant, some source/benchmark should be provided. Is it TextMate more performant than treesitter? Who says so?

I think I made my point in the previous post: #9087 (comment)

Everyone knows, but nobody dares to say, that NeoVim users are struggling in the mud of treesitter parsers right now, no need for vim users to experience such horrible things again. If someone really care about semantic syntax highlighting, indentation or textobj, they can still use LSP as well, LSP does a good job for all these things. There are plenty of LSP solutions for both vim & neovim.

Let's back to textmate, the core part of textmate syntax system is oniguruma, which is open source and well maintained by the community.

known editors / ides supporting textmate grammar:

• vscode
• textmate itself
• eclipse
• jetbrains

The syntax rendering of those editors/IDEs above is really fluent and proved by both time and massive users, while treesitter are still in test and can freeze nvim when parsing large files. (also check the performance issues in the previous post)

Monarch was initially built to support languages in VS Code. Then, they decided to switch for TextMate as well because of reasons listed above: microsoft/vscode#174 (comment) .

Some details:

VS Code's tokenization engine is powered by TextMate grammars. TextMate grammars are a structured collection of regular expressions and are written as a plist (XML) or JSON files. VS Code extensions can contribute grammars through the grammar contribution point.

The TextMate tokenization engine runs in the same process as the renderer and tokens are updated as the user types. Tokens are used for syntax highlighting, but also to classify the source code into areas of comments, strings, regex.

Starting with release 1.43, VS Code also allows extensions to provide tokenization through a Semantic Token Provider. Semantic providers are typically implemented by language servers that have a deeper understanding of the source file and can resolve symbols in the context of the project. For example, a constant variable name can be rendered using constant highlighting throughout the project, not just at the place of its declaration.

Highlighting based on semantic tokens is considered an addition to the TextMate-based syntax highlighting. Semantic highlighting goes on top of the syntax highlighting. And as language servers can take a while to load and analyze a project, semantic token highlighting may appear after a short delay.

it is easy to implement textmate syntax highlighting

The tokenizer of vscode/textmate is:

• https://github.com/kkos/oniguruma

And here is the wrapper in javascript, it's neatly written and not hard to understand:

• https://github.com/microsoft/vscode-textmate

All we need to do is rewriting the javascript wrapper in C,

No more than 4854 lines (including comments) in javascript/typescript

And thousands of textmate syntax files are ready to use.

[lacygoill]

No more than 4854 lines (including comments) in javascript/typescript

Tests excluded, it's 3779 lines of code (source: cloc(1)).
Tests included, it's 5074 lines of code.

[lacygoill]

Why not Sublime grammar instead of TextMate grammar? It seems more powerful, and easier to read.

I think .sublime-syntax is more easy to write and readable.

source

Sublime text 3 has implemented a new grammar format that seems much better than the traditional textmate grammar.

source

Is it because there have been fewer .sublime-syntax files written than .tmLanguage ones? Is there a licensing issue with these files?

[skywind3000]

@lacygoill maybe textmate grammar is a little easier ? because there are reference implementations:

• https://github.com/kkos/oniguruma
• https://github.com/microsoft/vscode-textmate

But sublime is closed source ? we need write everything from scratch ??

edit: if sublime 3 grammar is also based on oniguruma, maybe things can become a little easier ? anyway, both textmate/sublime solution are better than tree-sitter.