Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

ENH: Avoid tag updating when ctags output has identical fingerprint. #43

Merged
merged 2 commits into from

2 participants

@inkarkat

We can avoid the lengthy and blocking update of the tags database when the ctags output returns the same information as before; i.e. nothing tag-relevant has been changed since the last update.
Since 7.3.816, Vim has a sha256() function that allows us to quickly calculate a fingerprint over the ctags output. When that is not available, only handle the special case of a (covered) file that has no tags at all (as calculating a hash in Vimscript is costly and would probably defeat the intended speedup).

inkarkat added some commits
@inkarkat inkarkat ENH: Avoid tag updating when ctags output has identical fingerprint.
We can avoid the lengthy and blocking update of the tags database when the ctags output returns the same information as before; i.e. nothing tag-relevant has been changed since the last update.
Since 7.3.816, Vim has a sha256() function that allows us to quickly calculate a fingerprint over the ctags output. When that is not available, only handle the special case of a (covered) file that has no tags at all (as calculating a hash in Vimscript is costly and would probably defeat the intended speedup).
d789e8e
@inkarkat inkarkat BUG: Avoid E713 when attempting fingerprinting without current file.
When creating a tags file for an entire directory, the a:cfile variable is empty. In that case, the cache doesn't work. Avoid "E713: Cannot use empty key for Dictionary" by checking for that.
92d1f2d
@xolox xolox referenced this pull request from a commit
@xolox Minor changes to pull request #43 on GitHub (thank you Ingo :-)
The pull request: #43
07437cf
@xolox xolox merged commit 92d1f2d into xolox:master
@xolox
Owner

Hi Ingo,

Thanks for the idea and code, this is very nice! You actually motivated me to get off my ass and compile an up to date Vim 7.3 (so I could test the code) instead of being stuck with Vim 7.2. I'm much happier now :-) (both with Vim 7.3 and a speedier vim-easytags).

- Peter

@inkarkat

Thanks Peter, and good to hear you're back! I had already started to worry you don't maintain your plugins any more. I hope you'll get around to consider my other two fixes (#36 and #37) as well.

I'm quite happy with this plugin; however, the fact that the tags update is still blocking Vim regularly for a few seconds is bothering me. Since Vim is re-reading the tags file on each tags jump, the tags update could be done completely asynchronously, couldn't it? (I imagine an asynchronous external script, or a separate thread in the built-in Python interpreter.) Why didn't you go this route?!

Did you do this to trigger the update of the highlighting (which I currently don't use to avoid more slowdown) immediately after the tags update? After dabbling with the code, I think it would have been a cleaner design and less complexity if those two parts were separate plugins. I would love to hear your thoughts on this!

@xolox
Owner

Thanks Peter, and good to hear you're back! I had already started to worry you don't maintain your plugins any more. I hope you'll get around to consider my other two fixes (#36 and #37) as well.

Hopefully I'll get around to merging most of your pull requests this weekend (assuming I agree with the changes ;-).

I'm quite happy with this plugin; however, the fact that the tags update is still blocking Vim regularly for a few seconds is bothering me. Since Vim is re-reading the tags file on each tags jump, the tags update could be done completely asynchronously, couldn't it? (I imagine an asynchronous external script, or a separate thread in the built-in Python interpreter.) Why didn't you go this route?!

In one word: portability.

It bothers me as well, and I have about a dozen different implementations of vim-easytags floating around on my hard disk, all solving the problem in different ways... One of them (a headless external program that forks to the background as you suggest) is the most advanced by far and I should probably just bite the bullet and publish that. From what I remember it makes the plug-in a dream to work with (no more locking up Vim, waiting for a 5 MB tags file to be written).

However the nice thing about vim-easytags right now is that people can drop it in their ~/.vim and it should Just Work (assuming Exuberant Ctags is installed of course). When I add this external program I see two choices:

  1. Keep two parallel implementations up to date, one in Vim script and the other in my language of choice. Always testing both is going to be a pain but it's manageable. This would make vim-easytags work out of the box but with the option of being faster once people install the easytags executable on their system.

  2. Force all users to always use the external program. This simplifies the Vim script code a lot, but the external program will have to be very well tested to guarantee that it works on all platforms and in all circumstances...

What do you think? Regardless of what I do with the plug-in published on Vim Online I can see about making the easytags executable available on GitHub so people can play around with it. I'm afraid the code has suffered some bit rot, e.g. I will probably have to make it compatible with file type specific tags files and such.

@xolox
Owner

By the way, I'm wondering if using the Python interface for Vim for fingerprinting in Vim 7.2 would be worthwhile. Have you considered this already, or maybe even tried it?

@inkarkat

Thanks for your insights; you've surely spent a lot of time thinking this through, that's great!

One of them (a headless external program that forks to the background as you suggest) is the most advanced by far and I should probably just bite the bullet and publish that. From what I remember it makes the plug-in a dream to work with (no more locking up Vim, waiting for a 5 MB tags file to be written).

I'm ambivalent about additional dependencies, too; for example, I'm still using ol' snipMate instead of the modern Ultisnips plugin because of its Python implementation. I've only started compiling my own Windows Vim binaries a few months ago; beforehand it was difficult to keep a matching Python implementation around.

I guess those most in need of easytags are hard-core programmers, not beginners, so additional dependencies aren't that problematic. I'm quite open to Python and Perl, a binary might work, too (compilation usually is a breeze on Linux, for Windows it would be nice to provide ready-made binaries). On the other hand, you'll invite more contributions from others (like me) if you stick to an easy scripting language.

One idea I had was to strip off the tags updating part of easytags into a separate Vimscript, and then just launch a non-interactive separate Vim process in the background (without loading .vimrc and other plugins). Though not as efficient as Python or binary, it might be a good compromise, and everybody probably has a terminal Vim available :-)

@inkarkat

RE: Python interface for Vim 7.2 for fingerprinting

I usually have up-to-date Vim binaries, and I don't consider the added fingerprinting to be that revolutionary, so IMO it's not worthwhile to do, though certainly possible without that much effort.

@xolox
Owner

One idea I had was to strip off the tags updating part of easytags into a separate Vimscript, and then just launch a non-interactive separate Vim process in the background (without loading .vimrc and other plugins). Though not as efficient as Python or binary, it might be a good compromise, and everybody probably has a terminal Vim available :-)

For some reason this seems really dirty to me :-) however it would definitely be portable and it would completely solve the code duplication issue. You definitely got me wondering, so I tried it out and it seems surprisingly fast:

peter@lucid-vm> time vim -u NONE -U NONE --noplugin --cmd 'call xolox#misc#msg#info("testing, 1, 2, 3") | scriptnames | quit'

… Vim startup screen snipped …

testing, 1, 2, 3

1: ~/Dropbox/Dotfiles/vim/autoload/xolox/misc/msg.vim

vim -u NONE -U NONE --noplugin --cmd   0.00s user 0.02s system 58% cpu 0.034 total

Hopefully I'll have some time to create a prototype to play around with in the next weekend.

@xolox
Owner

I usually have up-to-date Vim binaries, and I don't consider the added fingerprinting to be that revolutionary, so IMO it's not worthwhile to do, though certainly possible without that much effort.

You're right, I want to move away from Vim + Python, not make the duplication worse :-)

@xolox xolox referenced this pull request from a commit
@xolox Remove Python highlighting code (to be replaced with Vim script..)
These are the first steps (ripping out the Python code) towards
implementing asynchronous tag updates using only Vim as outlined
in pull request #43:

  #43
aa65f9f
@xolox xolox referenced this pull request from a commit
@xolox Remove update detection (using sha256())
In an asynchronous configuration, update detection seems to require
bidirectional communication between the two Vim instances (which will
get messy). Update detection was nice to have, but in the branch where
I'm currently working, asynchronous tag updates have priority :-).
I'll see about resurrecting the update detection later.

                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
b918cd4
@xolox xolox referenced this pull request from a commit
@xolox Remove special casing of "first run"
I removed the special casing of the "first run" that used Exuberant Ctags
to directly create non-existing tag files. This additional code path
complicated the plug-in for an exceptional circumstance that wouldn't
be slow even if it was handled using the generic code. In other words,
out it goes in the quest for asynchronous tag updates!

                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
3408ba9
@xolox xolox referenced this pull request from a commit
@xolox Split xolox#easytags#update() into 2 phases: initialization & execution
                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
f8f99c8
@xolox xolox referenced this pull request from a commit
@xolox Initial implementation of asynchronous tag updates*
Based on the refactorings in the last four change sets, this change set
introduces initial support for asynchronous tag updates using only Vim
as outlined in pull request #43:

  #43

* I butchered a lot of code (including a few features) to get
  asynchronous tag updates working correctly. I'm not sure yet how
  successful the asynchronous tag updates feature will be (ideally it
  can become the default on all platforms) so I'm also not sure yet if I
  will re-implement the butchered features :-)
6ab882d
@xolox xolox referenced this pull request
Closed

Asynchronous tag updates #49

@xolox
Owner

Hey Ingo,

I created the branch async-cleanup and have started working on asynchronous tag updates for vim-easytags using only Vim (as we discussed previously in this pull request). If you're interested you can find an overview of the changes I made so far in pull request #49.

@inkarkat

Peter,

that's great news! I especially like the fact that you were able to remove considerable portions of code; of course, the promise of no more Vim blocking during the tags update makes this feature highly desirable!

I've managed to get your async branch to work on Vim 7.3.823 on Windows/x64; you've probably already noticed the pull requests and issues I had to submit. I sincerely hope we can get this straightened out in the near future!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 28, 2013
  1. @inkarkat

    ENH: Avoid tag updating when ctags output has identical fingerprint.

    inkarkat authored
    We can avoid the lengthy and blocking update of the tags database when the ctags output returns the same information as before; i.e. nothing tag-relevant has been changed since the last update.
    Since 7.3.816, Vim has a sha256() function that allows us to quickly calculate a fingerprint over the ctags output. When that is not available, only handle the special case of a (covered) file that has no tags at all (as calculating a hash in Vimscript is costly and would probably defeat the intended speedup).
Commits on Apr 17, 2013
  1. @inkarkat

    BUG: Avoid E713 when attempting fingerprinting without current file.

    inkarkat authored
    When creating a tags file for an entire directory, the a:cfile variable is empty. In that case, the cache doesn't work. Avoid "E713: Cannot use empty key for Dictionary" by checking for that.
This page is out of date. Refresh to see the latest.
Showing with 40 additions and 2 deletions.
  1. +40 −2 autoload/xolox/easytags.vim
View
42 autoload/xolox/easytags.vim
@@ -99,8 +99,11 @@ function! xolox#easytags#update(silent, filter_tags, filenames) " {{{2
let tagsfile = xolox#easytags#get_tagsfile()
let firstrun = !filereadable(tagsfile)
let cmdline = s:prep_cmdline(cfile, tagsfile, firstrun, a:filenames, context)
- let output = s:run_ctags(starttime, cfile, tagsfile, firstrun, cmdline)
+ let [output, has_updates] = s:run_ctags(starttime, cfile, tagsfile, firstrun, cmdline)
if !firstrun
+ if !has_updates
+ return 1
+ endif
if have_args && !empty(g:easytags_by_filetype)
" TODO Get the headers from somewhere?!
call s:save_by_filetype(a:filter_tags, [], output, context)
@@ -201,12 +204,46 @@ function! s:prep_cmdline(cfile, tagsfile, firstrun, arguments, context) " {{{3
return have_args ? join(cmdline) : ''
endfunction
+if exists('*sha256')
+function! s:get_fingerprint(cfile, output)
+ return sha256(a:output)
+endfunction
+else
+function! s:get_fingerprint(cfile, output)
+ " Don't want to re-implement a costly hashing function in Vimscript. Just
+ " handle files that never had any tags.
+ if empty(a:output)
+ return get(s:fingerprints, a:cfile, 1)
+ else
+ return ''
+ endif
+endfunction
+endif
+
+let s:fingerprints = {}
+function! s:has_updates(cfile, output)
+ if empty(a:cfile)
+ " The cache doesn't work when tags aren't created for the current file.
+ return 1
+ endif
+
+ let fingerprint = s:get_fingerprint(a:cfile, a:output)
+ if ! empty(fingerprint) && get(s:fingerprints, a:cfile, '') ==# fingerprint
+ return 0
+ endif
+
+ let s:fingerprints[a:cfile] = fingerprint
+ return 1
+endfunction
+
function! s:run_ctags(starttime, cfile, tagsfile, firstrun, cmdline) " {{{3
let lines = []
+ let has_updates = 1
if a:cmdline != ''
call xolox#misc#msg#debug("easytags.vim %s: Executing %s.", g:xolox#easytags#version, a:cmdline)
try
let lines = xolox#shell#execute(a:cmdline, 1)
+ let has_updates = a:firstrun || s:has_updates(a:cfile, join(lines, "\n"))
catch /^Vim\%((\a\+)\)\=:E117/
" Ignore missing shell.vim plug-in.
let output = system(a:cmdline)
@@ -215,6 +252,7 @@ function! s:run_ctags(starttime, cfile, tagsfile, firstrun, cmdline) " {{{3
throw printf(msg, fnamemodify(a:tagsfile, ':~'), strtrans(output))
endif
let lines = split(output, "\n")
+ let has_updates = a:firstrun || s:has_updates(a:cfile, output)
endtry
if a:firstrun
if a:cfile != ''
@@ -225,7 +263,7 @@ function! s:run_ctags(starttime, cfile, tagsfile, firstrun, cmdline) " {{{3
return []
endif
endif
- return xolox#easytags#parse_entries(lines)
+ return [xolox#easytags#parse_entries(lines), has_updates]
endfunction
function! s:filter_merge_tags(filter_tags, tagsfile, output, context) " {{{3
Something went wrong with that request. Please try again.