Request: Update ctags incrementally

[blip-lorist]

Hi there, mahalo for putting this project together. Only tested with ctags with autogen on since I'm not using any languages currently supported by gtags.

While making quick edits in vim, I noticed that my project's tags file would get stuck in an partially-written state (for example, a complete tags file of 50K lines would only have 3K lines after a quick edit to a project file)

Looks like every time :w is used, the plugin seems to regenerate ctags for the project from scratch while vim is open. If I quit vim before this process is complete, then I'm left with a partially-generated tags file.

Incremental ctag updates could have the following benefits:

• Preserves the bulk of the project tags generated earlier
• Reduced cpu time, ctags is a cpu hog
• More likely to capture updates before the user quits vim

[jsfaint]

I'm not sure which parameter is for incremental update in ctags.
Maybe you can try this, by setting let g:gen_tags#ctags_opts = '-a' in your vimrc

According to ctags help message, -a or --append maybe suit for you

  -a   Append the tags to an existing tag file.

BUT I don't see any difference on my side, when I run both command for Linux Kernel.

# jiasui @ ul80vt in ~/repos/linux-stable on git:master o [8:54:41]
$ time ctags -R
ctags -R  71.40s user 5.56s system 100% cpu 1:16.37 total

# jiasui @ ul80vt in ~/repos/linux-stable on git:master o [8:56:03]
$ time ctags -Ra
ctags -Ra  76.37s user 6.58s system 100% cpu 1:22.50 total

[blip-lorist]

The time similarity in both commands is likely due to all tags still being generated for the pwd both times. I believe the -a flag just impacts the output.

According to ctags docs:
Indicates whether tags generated from the specified files should be appended

To incrementally generate based on the current file being edited, is it possible to specify the current file for automatic generation? Such as:
ctags -Ra ./path/to/file_being_written_to

This would yield better incremental change times:

# Generate all tags once, up front
$ gtime ctags -R
0.67user 0.07system 0:00.77elapsed 97%CPU
$ wc -l tags
22376 tags

# Now I've declared a new function in ./current_file
# As the file is written to, only generate / append tags for that file
$ gtime ctags -Ra ./current_file
0.01user 0.01system 0:00.04elapsed 72%CPU
$ wc -l tags
22377 tags

[jsfaint]

Hmm, which means gen_tags.vim need to run ctags -a for single file?
But it looks -a will not delete the symbol from tags file which was already deleted?

[blip-lorist]

Yea, it is one possibility. Looks like the issue with -a would be keeping track of subtractive updates - the append with a single file seems like it would handle additive updates just fine.

Got curious about how vim-gutentags handles this. Their docs describe this workflow:

Usually, ctags can only append tags to an existing tag file, so Gutentags removes the tags for the current file first, to make sure the tag file is always consistent with the source code.

[jsfaint]

I read the source code of vim-gutentags, it use platform dependent script to do this job..
I need to check how to do this with vimscript.

[jsfaint]

@lorainekv
I have a dirty hack for this issue at increament_update branch
gen_tags.vim supports incremental update now.
But for prune exist tags, I added a option g:gen_tags#ctags_prune for it.
If g:gen_tags#ctags_prune is 1, gen_tags.vim will prune exist tags first.

The prune action will impact the performance when tagfile is big. (like linux kernel, >500MB)

Can you have a try? Thanks

[blip-lorist]

Nice, thanks for updating this. The subtractive and additive updates to tag files seem to be working now. :)

I am running into some surprising behavior though - I seem to now have tag files in both a project root (for a github repo) and in the ~/.cache/... path. This doesn't seem intended, since it's duplicate tag files. So when I attempt to find a function definition it asks me to choose from in ~/project_folder/file and also in ./project_folder/file.

When I run a vimdiff on the ~/.cache tagfile and the git repo tag file, I notice that one says !_TAG_FILE_SORTED 0 and the other says !_TAG_FILE_SORTED 1

[jsfaint]

What do you mean about 'git repo tag file'?
Did you generate tags file manually?

gen_tags.vim will only generate tags in ~/.cache/tags_dir by default.
If you don't set g:gen_tags#ctags_use_cache_dir to 0

[blip-lorist]

My bad, regarding the repo tag file, I had forgotten to uninstall another plugin I was testing out yesterday. Thanks for the increment update!

[jsfaint]

You're welcome, thanks for the report and clarify 😄

[jsfaint]

I merged increment_update branch to master branch

[jeromedalbert]

I like the idea and I would like to use this prune+increment feature, but there a few problems currently that don't make it very usable.

@jsfaint Would it be possible to make the "hack" in s:ctags_prune async by using async jobs? Because otherwise it defeats the purpose of an async plugin. On Unix it would be by using grep or sed I guess.

Also, if I have g:gen_tags#ctags_prune set to 1 and I save a file outside of the project directory, for example my vimrc, I have this error: Pattern not found: \/Users\/jerome\/.dotfiles\/.vim\/init.vim. I have this problem with the following Neovim configuration:

let g:loaded_gentags#gtags = 1
let g:gen_tags#ctags_auto_gen = 1
let g:gen_tags#ctags_opts = '-a -u'
let g:gen_tags#ctags_prune = 1

[jsfaint]

@jeromedalbert Sorry for the late reply.
It's known issue which is limitation of pure vim script.

Yeah, I know sed is very efficient, but it only works on *nix platform.
I want gen_tags.vim work on both windows and *nix platform

So I need to find a way which do same thing with sed on windows, or rewrite this part in Python.
Sorry, I do not have spare time to look into this currently.
PR will be appreciated 😄

[jsfaint]

Due to the the exuberant ctags is not maintained anymore, the successor recommendation
is universal ctags from now on.

There is feature request in universal-ctags/ctags#1421
If this feature is implemented, the ugly prune in vimscript can be removed and the performance should be improved in same time.

Let's see what will be happened. 😄