Sluggish with big projects

[tricktux]

Hi,
Thanks for awesome plugin. My issue is that neovim in windows because extremely sluggish when neotags is enabled for big projects. On the other side I dont even see any highlight difference. I assume is because is busy _parseLine() and not done yet. The neotags file that it creates is ~8.8kb. However, if I use the default options:

let g:neotags_ctags_args = [
            \ '-L -',
            \ '--fields=+l',
            \ '--c-kinds=+p',
            \ '--c++-kinds=+p',
            \ '--sort=no',
            \ '--extra=+q'
            \ ]

Then neotags ctags file is created but always with size 0.
This below is my configuration:

Plug 'c0r73x/neotags.nvim' " Depends on pip3 install --user psutil
	let g:neotags_enabled = 1
	let g:neotags_file = g:cache_path . 'neotags'

	let g:neotags_appendpath = 0
	let g:neotags_recursive = 0
	let g:neotags_ctags_bin = 'rg --files -t cpp "'. getcwd() .'" | ctags'
	let g:neotags_ctags_args = [
				\ '-L -',
				\ '--fields=+iaSl',
				\ '--c-kinds=+p',
				\ '--c++-kinds=+p',
				\ '--sort=no',
				\ '--extra=+q'
				\ ]

Thanks for the support

[c0r73x]

That sounds strange indeed. 8.8kb shouldn't be sluggish, I've tested it on the neovim source code that generates a 3.5mb neotags file, and it works but it's abit sluggish when all the syntax matchers are in.

Never tested it on windows though.

I will do some testing to see if I can replicate the problem.

[tricktux]

I am sorry I meant to say ~8.8mb. Thanks for the quick reply

[c0r73x]

Can you try with the latest commit, I did some tweaks using itertools.

[tricktux]

Now I cannot get it to create the tags file. It creates the file per say but is always of size 0.
I got neotags to use an existing ctags file of size 7.99 MB (8,378,846 bytes) by adding the file to 'tags'. And is still very sluggish. Can maybe threads come into play here? Thanks for the support

[c0r73x]

Very strange, I will add some debug functions.

I've tried to add thread support but I cant get it to work, it seems that whenever I spawn a thread the neovim api crashes.

I will do some more testing during the weekend.

[tricktux]

Sounds awesome. Maybe even oportunity to file for `neovim` bug. I'll try to look for some plugins that uses threads remotely.

[c0r73x]

I did manage to get some speed improvements. Now it reads the neovim-source tagfile in about 0.20s, i also added a new variable g:neotags_verbose to see what part of the plugin that runs slow. And i updated some of the error handling.

I also noticed that on linux if i set shell=False to the Popen (pythons function to open an new process) the ctags file is empty. It might be that we need some special tricks on windows.

[tricktux]

Waoo this almost super awesome:

done reading b'tags' (0.29s)
done reading b'tags' (0.32s)
	applied syntax for cpp#c (0.00s)
	applied syntax for cpp#g (0.01s)
	applied syntax for cpp#s (0.01s)
	applied syntax for cpp#t (0.01s)
	applied syntax for cpp#u (0.00s)
	applied syntax for cpp#e (0.01s)
	applied syntax for cpp#d (0.00s)
	applied syntax for cpp#f (0.01s)
	applied syntax for cpp#p (0.00s)
applied syntax for cpp#m (0.01s)

And that tag file is of size: 10.6 MB (11,154,497 bytes). The only thing is that I see no difference in the highlight. This is the command used to create the file:
--sort=yes --c++-kinds=+pl --fields=+iaS --extra=+q --language-force=C++
Any idea why no highlight?
Thanks again for the support

[c0r73x]

Awesome :D

You probably need to add some highlights for the tag syntax. I should probably add this to the documentation :P

For example, this is what i use for C/C++.

" colors {{{1
let s:bg         = { 'gui': '#0c0c0c', 'cterm': 'none' }
let s:fg         = { 'gui': '#424242', 'cterm': 'none' }

let s:black      = { 'gui': '#212121', 'cterm': '0' }
let s:grey       = { 'gui': '#424242', 'cterm': '8' }

let s:darkred    = { 'gui': '#d32f2f', 'cterm': '1' }
let s:red        = { 'gui': '#f44336', 'cterm': '9' }

let s:darkgreen  = { 'gui': '#689f38', 'cterm': '2' }
let s:green      = { 'gui': '#8bc34a', 'cterm': '10' }

let s:brown      = { 'gui': '#ffa000', 'cterm': '3' }
let s:yellow     = { 'gui': '#ffc107', 'cterm': '11' }

let s:darkblue   = { 'gui': '#0288d1', 'cterm': '4' }
let s:blue       = { 'gui': '#03a9f4', 'cterm': '12' }

let s:purple     = { 'gui': '#c2185b', 'cterm': '5' }
let s:pink       = { 'gui': '#e91e63', 'cterm': '13' }

let s:darkcyan   = { 'gui': '#0097a7', 'cterm': '6' }
let s:cyan       = { 'gui': '#00bcd4', 'cterm': '14' }

let s:lightgrey  = { 'gui': '#757575', 'cterm': '7' }
let s:white      = { 'gui': '#e0e0e0', 'cterm': '15' }

let s:darkergrey = { 'gui': '#121212', 'cterm': '233' }
let s:darkgrey   = { 'gui': '#1c1c1c', 'cterm': '234' }
" 1}}}
" util {{{1
function! s:h(group, style)
    if(has_key(a:style, 'link'))
        exec 'highlight! link ' a:group a:style.link
    else
        exec 'highlight' a:group
                    \ 'guifg='   (has_key(a:style, 'fg')    ? a:style.fg.gui   : 'NONE')
                    \ 'guibg='   (has_key(a:style, 'bg')    ? a:style.bg.gui   : 'NONE')
                    \ 'guisp='   (has_key(a:style, 'sp')    ? a:style.sp.gui   : 'NONE')
                    \ 'gui='     (has_key(a:style, 'deco')  ? a:style.deco     : 'NONE')
                    \ 'ctermfg=' (has_key(a:style, 'fg')    ? a:style.fg.cterm : 'NONE')
                    \ 'ctermbg=' (has_key(a:style, 'bg')    ? a:style.bg.cterm : 'NONE')
                    \ 'cterm='   (has_key(a:style, 'deco')  ? a:style.deco     : 'NONE')
    endif
endfunction
" 1}}}
" c/cpp {{{3
call s:h('cTypeTag',                { 'fg': s:brown })
call s:h('cPreProcTag',             { 'fg': s:purple })
call s:h('cFunctionTag',            { 'fg': s:darkred })

call s:h('cMemberTag',              { 'link': 'cMember' })
call s:h('cEnumTag',                { 'link': 'cEnum' })

call s:h('cppTypeTag',              { 'fg': s:brown })
call s:h('cppPreProcTag',           { 'fg': s:purple })
call s:h('cppFunctionTag',          { 'fg': s:darkred })

call s:h('cppMemberTag',            { 'link': 'cppMember' })
call s:h('cppEnumTag',              { 'link': 'cppEnum' })
" 3}}}

[tricktux]

I will help you with that as soon as I understand what the heck is going on 😄
Where that script goes? after/syntax/cpp(I tried this just copied it and nothing happend :( ), autocmd???
Lost :P.....

[c0r73x]

Oh, I just copy/pasted from my syntax file :P

But you can just place some highlights in your vimrc, for example

highlight cppFunctionTag ctermfg=2 guifg=#00ff00

You can see what highlights that gets created by looking in neovim.vim.

Hopefully this helps :)

[tricktux]

OMG man so dumb :( I can't get it to work :(

I placed this in my after/syntax/cpp.vim

highlight cppEnum ctermfg=2 guifg=#00ff00
highlight cppFunction ctermfg=2 guifg=#00ff00
highlight cppMember ctermfg=2 guifg=#00ff00

and got nothing

[c0r73x]

after/syntax/cpp.vim should work but you have to add Tag :)

like this.

highlight cppEnumTag ctermfg=2 guifg=#00ff00
highlight cppFunctionTag ctermfg=2 guifg=#00ff00
highlight cppMemberTag ctermfg=2 guifg=#00ff00

[tricktux]

😭 It still doesnt work 😭 I dunno. Maybe its neovim-qt. I should probably try it under unix. I use gruvbox as my colorscheme. Does that even matter. 😭 hahaha. I appreciate the help. I really want this to work. How would I go about debugging highlight? Any other super awesome tips?

[c0r73x]

Very strange, I just tested on neovim-qt on linux with the same highlight in after/syntax/cpp.vim and it works. I think I will have to setup a Windows VM to be able to test further :/

As for debugging, you could check :syntax to see if you have for example cppTypeTag xxx match /\C\<\%(test\)\>/. Also :hi cppFunctionTag to see if the cpp.vim file is loaded correctly.

[tricktux]

The :hi cppTypeTag are there. I need to investigate more on the syntax command. I have this powershell script to download and update neovim on windows. Also with chocolatey you could install python relativity easy on windows. Will keep investigating 👍

[c0r73x]

Ah nice, I will setup a VM this weekend and do some testing :)

[tricktux]

I made some discoveries:

1. On unix is slugish as well. But its a weird issue. Because the plugin its not doing anything per say. The times are fast still so.... maybe neovim its highlighting?? not sure. I think it could be related the fact that the plugin adds the g:neotags_file to the 'tags' options and this file doesnt exist in my system....maybe not sure. I tested it with neovim source code and tags loaded at startup. My setup has changed though. Im not letting neotags handle the tags file creation:

		let g:neotags_enabled = 1
		let g:neotags_file = g:cache_path . 'ctags/neotags'
		let g:neotags_verbose = 1
		let g:neotags_run_ctags = 0

Therefore, like I mention before even though the file neotags doesnt exist the plugin adds it to the 'tags' option. I dont know wild guess 😛
2. I discovered that universal-ctags the "new" ctags doesnt support legacy option --extra you must specify --extras. I thought I got it, this is why Im not getting any highlight.....wrong...still not getting highlights... 💯 😭

P.S: Hmm.....it just occurred to me that maybe the new universal-ctags are not the same as the legacy highliting ctags????? Im pretty sure I have universal ctags on Windows as well.
On arch they can be installed: pacaur -S universal-ctags-git

[c0r73x]

I use universal ctags aswell on gentoo. but if i check the --help it says

  --extra=[+|-]flags
      Include extra tag entries for selected information (flags: "Ffq.") [F].

Also this is the version I'm using.

Universal Ctags 0.0.0, Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Mar  4 2017, 22:13:31
  URL: https://ctags.io/
  Optional compiled features: +wildcards, +regex, +option-directory, +coproc

So that shouldn't be a problem. Yeah if it's running slow with big tag files it's probably highlighting but missing "highlighters". I dont think that it would matter if you add a none existing file to &tags, neovim and neotags should just skip it.

Just to test things, if you check with easytags do you get highlighting from the same tag files?

This is a very strang issue indeed.

[tricktux]

I got the highlighting!! 💯 😃
Silly me I forgot neovim is C code 😛
However, it still not smooth. I'm not sure what it is. I made a screencast

[c0r73x]

Ah nice!

Humm yeah it looks abit slow but unfortunately that slowness is not because of neotags, it's because of neovims highlight. You might be able to optimize it by disabling some of the highlight rules with g:neotags#cpp#order and only add those that you need.

[tricktux]

Got super improvements on speed with these:

set regexpengine=1
let g:neotags#cpp#order = 'ced'
let g:neotags#c#order = 'ced'

I remembered regexpengine from easytags
With these two settings on big projects there is some lag still. But its so little that is usable.
Closing the issue but I think that the plugin doesnt work with the legacy ctags because I did realize that that is what I had on windows. Maybe some testing should be done on this. And mentioned in the documentation. If it turns out to be an issue.
Thanks for all the support 😛

P.S: I had forgotten about the syntax file. I think its awesome. It should definitely be included in the documentation.