treesitter incredible slow with single line big json due to hotspot in nvim_buf_set_extmark

[comicfans]

• nvim --version:
  v0.5.0-dev+1357-g192f89ea1
  Build type: RelWithDebInfo
  LuaJIT 2.1.0-beta3

• Operating system/version:
  archlinux x64

• Terminal name/version:
  xterm(also happened when using nvim-qt)

[ ] vim -u DEFAULTS (version: ) behaves differently (not tried)

Steps to reproduce using nvim -u NORC

only happen if using with nvim-treesitter, while using nvim -u NORC, nvim highlight head parts of json and not hang.

Actual behaviour

while using with tree-sitter, nvim incredible slow when opening very big json file(in one line), profile shows nvim_buf_set_extmark consume most of CPU time. after a while, even after file is highlighted, cursor move still very slow, especially when cursor hit object start '{'

Expected behaviour

tree-sitter highlight should be responsive.

minimal config:

set nocompatible
filetype off


call plug#begin('~/.vim/bundle')

"vim-scripts

Plug 'nvim-treesitter/nvim-treesitter', {'do': ':TSUpdate'}
call plug#end()


lua << EOF


require'nvim-treesitter.configs'.setup {
    ensure_installed = "maintained",
    highlight = {
        enable = true
    },
    indent = {
        enable = true
    }
}

EOF

while nvim hanging, I've run profiler attached, profile result shows that nvim_buf_set_extmark consume 96% cpu time.
source code shows that src/nvim/api/buffer.c 1497 and 1687 each consume about 50% cpu time.

  1492   size_t len = 0;
  1493   if (line < 0 || line > buf->b_ml.ml_line_count) {
  1494     api_set_error(err, kErrorTypeValidation, "line value outside range");
  1495     return 0;
  1496   } else if (line < buf->b_ml.ml_line_count) {
  1497     len = STRLEN(ml_get_buf(buf, (linenr_T)line+1, false));             <------------------------ hotspot1
  1498   }



  1685  if (col2 >= 0) {
  1686     if (line2 >= 0 && line2 < buf->b_ml.ml_line_count) {
  1687       len = STRLEN(ml_get_buf(buf, (linenr_T)line2 + 1, false));          <--------------------------hotspot2
  1688     } else if (line2 == buf->b_ml.ml_line_count) {
  1689       // We are trying to add an extmark past final newline
  1690       len = 0;
  1691     } else {
  1692       // reuse len from before
  1693       line2 = (int)line;
  1694     }

[vigoux]

Nice work in profiling this.
I'll try to look into it when I have time.

[bfredl]

We could add a ml_get_buflen variant and cache the last requested len, though a lot of other things are O(N^2) in line length so it will punt the issue somewhere else.

[vigoux]

Working on it then. I think this can be marked as a bugfix for 0.5

[vigoux]

Reopenning after nvim-treesitter/nvim-treesitter#1708, I think I found a fix for that, by using the new api function added in #15248 and some optimizations in TSHighlighter:on_line