Slow tsx and typescript

[jinh0]

Describe the bug

TSX and TypeScript files take a noticeably longer time to load than JavaScript files. I have looked at --startuptime and :profile logs, and even when I delete all default regex-based syntax files loaded for TypeScript by NeoVim, treesitter is slow to load TS / TSX files.

I saw on issues like #1082 , people are having the same issue as me. I've seen that it might be due to the newer versions of tree-sitter?

To Reproduce

Steps to reproduce the behavior:

1. Install latest versions of NeoVim, treesitter, and nvim-treesitter.
2. Install treesitter parsers for typescript and tsx.
3. Open file and observe slow loading time.
4. Compare with opening JavaScript or JSX files with their respective treesitter parsers.

Expected behavior

Fast loading time, comparable to JSX files.

Output of :checkhealth nvim_treesitter

Output of nvim --version

Additional context

[vigoux]

As you say in your issue description, it might be out of our reach. We'll try our best to enhance performances, but we can't guarantee anything.

[jinh0]

I'm just wondering if other people are still having this issue and if they have any workarounds for the time being.

[jacksonludwig]

I think I do have this issue. Using treesitter highlighting makes things noticeably slower when both opening and moving around in a typescript file.
If you're using a fast PC you might not notice, but on some machines it's very obvious (don't notice it on my desktop, but on my laptop it's too slow to really be used comfortably).

[timsofteng]

Same here

[stsewd]

Hi, I think this might be related to the treesitter runtime update, can you check if compiling neovim with a previous version fix this issue? See neovim/neovim#14897

[folke]

Can you check if #1442 resolves this?

[jinh0]

I updated Treesitter to the latest version, including #1442 , but sadly nothing's changed.

[jinh0]

@stsewd , apologies for the late response. I tried @clason 's solution from neovim/neovim#14897 which merged with neovim 3 days ago. Sadly, typescript files are still slow to load. However, one thing to note is that JavaScript files load incredibly quick.

And these typescript files I'm talking about are not crazy big files. I'm talking <100 lines of typescript.

[benwainwright]

Can confirm I'm getting the same experience. My laptop is pretty fast so its not a big deal, but where JavaScript files in the same project are near instantaneous to load, there is a slight but definitely discernible delay in TS files when they are loaded.

[trixnz]

This was bothering me, as the delay can be quite annoying when switching between many buffers. I'm still fairly new to nvim and nvim-treesitter, but I decided to take a look and see why it was happening.

My assumption was that the highlights query should only need to be parsed once, and cached for every other buffer that will be opened, but it seems like that isn't the case. Swapping between buffers with <C-i> and <C-o> gives me the 100ms hit every time.

After digging into TSHighlighter:get_query in runtime/lua/vim/treesitter/highlighter.lua in nvim, I see that queries are cached at the scope of the TSHighlighter object, which suggests that nvim-treesitter is creating a separate highlighter instance per buffer, but worse yet, even when switching buffers?

If I change TSHighlighter:get_query to store it's cache at the file level rather than per-instance I get my smooth buffer switching back, but this is an obvious hack that I wouldn't recommend.

I don't know what the real solution should be, maybe nvim-treesitter needs to make better re-use of TSHighlighter objects so it can share the same instance between multiple files of the same language type? Or is this a limitation of the API exposed by nvim?

[jinh0]

@trixnz can you share this fix of yours? I don't personally have problems with lag switching buffers but it might be a clue in how to fix the general speed problem.

Also, I believe one important aspect about this issue is that, at least in my experience, the speed of Treesitter varies between languages. From what I've observed, opening JavaScript and Lua files are incredibly fast, but Typescript and Python (the two languages I most frequently use) are terribly slow.

This might be an issue on the language-specific implementations. Maybe the TypeScript and Python parsers are using some unnecessary laggy stuff. Is there anyone familiar to these parsers' development that can weigh in?

[folke]

@trixnz I dont' see that slow-down at all with .ts files, but I did disable treesitter folding. That still has performance issues. Have you tried disabling that?

[trixnz]

@trixnz can you share this fix of yours? I don't personally have problems with lag switching buffers but it might be a clue in how to fix the general speed problem.

Also, I believe one important aspect about this issue is that, at least in my experience, the speed of Treesitter varies between languages. From what I've observed, opening JavaScript and Lua files are incredibly fast, but Typescript and Python (the two languages I most frequently use) are terribly slow.

This might be an issue on the language-specific implementations. Maybe the TypeScript and Python parsers are using some unnecessary laggy stuff. Is there anyone familiar to these parsers' development that can weigh in?

Sure, but I wouldn't advise doing it this way. This was purely speculative changes to see where the bottleneck was. I believe this will break the moment you use more than one query, but as I was only working in a single .ts file with only highlights.scm, it satisfied my curiousity.

diff --git a/runtime/lua/vim/treesitter/highlighter.lua b/runtime/lua/vim/treesitter/highlighter.lua
index 84b6a5f13..17ae14b6c 100644
--- a/runtime/lua/vim/treesitter/highlighter.lua
+++ b/runtime/lua/vim/treesitter/highlighter.lua
@@ -215,15 +215,18 @@ function TSHighlighter:on_changedtree(changes)
   end
 end

+-- Store built queries globally, rather than per TSHighlighter instance
+local _queries = {}
+
 --- Gets the query used for @param lang
 ---
 --- @param lang A language used by the highlighter.
 function TSHighlighter:get_query(lang)
-  if not self._queries[lang] then
-    self._queries[lang] = TSHighlighterQuery.new(lang)
+  if not _queries[lang] then
+    _queries[lang] = TSHighlighterQuery.new(lang)
   end

-  return self._queries[lang]
+  return _queries[lang]
 end

 ---@private

@trixnz I dont' see that slow-down at all with .ts files, but I did disable treesitter folding. That still has performance issues. Have you tried disabling that?

I've tried disabling all modules except for highlighting and couldn't get it to go any faster. However, I did find some interesting results when comparing the performance of the highlights.scm present in this repo vs those that exist in the tree-sitter-typescript and tree-sitter-javascript repos.

Note: At the time of writing, I used the latest master revisions of each grammar repo and tree-sitter v0.19.0.

Benchmark #1: node bench_treesitter.js
  Time (mean ± σ):      74.8 ms ±   1.0 ms    [User: 70.8 ms, System: 5.0 ms]
  Range (min … max):    73.8 ms …  80.0 ms    39 runs

Benchmark #1: node bench_nvim_treesitter.js
  Time (mean ± σ):     126.2 ms ±   0.8 ms    [User: 119.8 ms, System: 7.3 ms]
  Range (min … max):   125.0 ms … 127.4 ms    23 runs

bench_nvim_treesitter.js

const Parser = require("tree-sitter");
const { Query } = Parser;

const TypeScript = require("tree-sitter-typescript");

const fs = require("fs");

const highlightsJs = Buffer.from(
  fs
    .readFileSync("/home/cam/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/queries/ecma/highlights.scm")
    .toString()
    .replaceAll("vim-match", "match")
);

const highlightsTs = Buffer.from(
  fs
    .readFileSync("/home/cam/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/queries/typescript/highlights.scm")
    .toString()
    .replaceAll("vim-match", "match")
);

const query = new Query(TypeScript.typescript, Buffer.concat([highlightsJs, highlightsTs]));

bench_treesitter.js

const Parser = require("tree-sitter");
const { Query } = Parser;

const TypeScript = require("tree-sitter-typescript");

const fs = require("fs");

const highlightsJs = fs.readFileSync("./node_modules/tree-sitter-javascript/queries/highlights.scm");
const highlightsTs = fs.readFileSync("./node_modules/tree-sitter-typescript/queries/highlights.scm");

const query = new Query(TypeScript.typescript, Buffer.concat([highlightsJs, highlightsTs]));

So while there are performance differences in the upstream grammer highlights vs those in this repo, it still takes a good amount of time to build the query in both cases. Almost all of that time can be found spent in ts_query__analyze_patterns. If nvim-treesitter creates a new highlight instance for each buffer, but also when switching buffers, it seems like the hit is unavoidable unless more aggressive caching of queries is implemented.

Again, this codebase is new to me so I might be up the wrong tree entirely but just figured I'd throw some braindumps down in the hopes that it might help someone smarter than me :).

[jinh0]

@trixnz I dont' see that slow-down at all with .ts files, but I did disable treesitter folding. That still has performance issues. Have you tried disabling that?

I don't have any explicit settings set for treesitter folding and I have my foldmethod as marker. Do I need to explicitly disable folding?

[theHamsta]

@trixnz I also saw that the queries for highlighting and injections don't get cached like out modules. Eventually, we should upstream our caching logic (we don't load folding/textobjects/indent queries twice)

I saw that ts_query__analyze_patterns was causing problems, this seemed to be introduced by 0.19.X tree-sitter/tree-sitter#973

[trixnz]

After a little bit more experimentation, I've come to a solution that solves my needs for the time being. I found that vim.treesitter.query allows users to override the queries via set_query and get_query caches the parsed queries globally, rather than per-highlighter instance.

So, when my LSP attaches I warm up typescript/javascript queries by parsing them up and overriding them. Essentially i'm (ab)using the user overrides as a cache mechanism. Probably a good place for a cache for anyone brave enough to take it on 😄.

Attached the prebuild snippet below for anyone who sees this as a reasonable hack:

local TSPrebuild = {}
local has_prebuilt = false

TSPrebuild.on_attach = function(client, bufnr)
    if has_prebuilt then return end

    local query = require("vim.treesitter.query")

    local function safe_read(filename, read_quantifier)
        local file, err = io.open(filename, "r")
        if not file then error(err) end
        local content = file:read(read_quantifier)
        io.close(file)
        return content
    end

    local function read_query_files(filenames)
        local contents = {}

        for _, filename in ipairs(filenames) do table.insert(contents, safe_read(filename, "*a")) end

        return table.concat(contents, "")
    end

    local function prebuild_query(lang, query_name)
        local query_files = query.get_query_files(lang, query_name)
        local query_string = read_query_files(query_files)

        query.set_query(lang, query_name, query_string)
    end

    local prebuild_languages = {"typescript", "javascript", "tsx"}
    for _, lang in ipairs(prebuild_languages) do
        prebuild_query(lang, "highlights")
        prebuild_query(lang, "injections")
    end

    has_prebuilt = true
end

return TSPrebuild

[theHamsta]

We should really tackle the caching issue ASAP. I have time next weekend but if anyone has time it would be nice to port our caching system (or a better version) to Neovim core. So that highlight queries and injections don't get loaded multiple times.

Anyone who has time now, please go ahead! ;-)

[jacksonludwig]

@trixnz Would you mind sharing where in my configuration that snippet of code would go?

[trixnz]

@trixnz Would you mind sharing where in my configuration that snippet of code would go?

It's really dependent on how your LSP is setup, but you just need to call TSPrebuild.on_attach(client, bufnr) in your tsserver on_attach callback.

[lewis6991]

Can someone confirm if this is still an issue? Closing in the meantime since there hasn't been any updates in some time.

[dovvakkin]

Can someone confirm if this is still an issue? Closing in the meantime since there hasn't been any updates in some time.

I think I still have it
Latest nvim nighly, and in 1k+ lines js file ui gets noticeably laggy

[clason]

That sounds like a different issue (which is already opened).

[kleczekr]

Can someone confirm if this is still an issue? Closing in the meantime since there hasn't been any updates in some time.

Still an issue in my case. Working on Debian, opening a .tsx file is impossibly slow and spikes CPU to 100% (and that with 300-line file).

[clason]

"Me too" comments without reproducible context (Neovim version, exact file) are useless. Performance is actively being worked on in Neovim core (not nvim-treesitter; there is nothing that can be done here except disable expensive query patterns and injections).