ENH: Avoid tag updating when ctags output has identical fingerprint.

[inkarkat]

We can avoid the lengthy and blocking update of the tags database when the ctags output returns the same information as before; i.e. nothing tag-relevant has been changed since the last update.
Since 7.3.816, Vim has a sha256() function that allows us to quickly calculate a fingerprint over the ctags output. When that is not available, only handle the special case of a (covered) file that has no tags at all (as calculating a hash in Vimscript is costly and would probably defeat the intended speedup).

[xolox]

Hi Ingo,

Thanks for the idea and code, this is very nice! You actually motivated me to get off my ass and compile an up to date Vim 7.3 (so I could test the code) instead of being stuck with Vim 7.2. I'm much happier now :-) (both with Vim 7.3 and a speedier vim-easytags).

• Peter

[inkarkat]

Thanks Peter, and good to hear you're back! I had already started to worry you don't maintain your plugins any more. I hope you'll get around to consider my other two fixes (#36 and #37) as well.

I'm quite happy with this plugin; however, the fact that the tags update is still blocking Vim regularly for a few seconds is bothering me. Since Vim is re-reading the tags file on each tags jump, the tags update could be done completely asynchronously, couldn't it? (I imagine an asynchronous external script, or a separate thread in the built-in Python interpreter.) Why didn't you go this route?!

Did you do this to trigger the update of the highlighting (which I currently don't use to avoid more slowdown) immediately after the tags update? After dabbling with the code, I think it would have been a cleaner design and less complexity if those two parts were separate plugins. I would love to hear your thoughts on this!

[xolox]

Thanks Peter, and good to hear you're back! I had already started to worry you don't maintain your plugins any more. I hope you'll get around to consider my other two fixes (#36 and #37) as well.

Hopefully I'll get around to merging most of your pull requests this weekend (assuming I agree with the changes ;-).

I'm quite happy with this plugin; however, the fact that the tags update is still blocking Vim regularly for a few seconds is bothering me. Since Vim is re-reading the tags file on each tags jump, the tags update could be done completely asynchronously, couldn't it? (I imagine an asynchronous external script, or a separate thread in the built-in Python interpreter.) Why didn't you go this route?!

In one word: portability.

It bothers me as well, and I have about a dozen different implementations of vim-easytags floating around on my hard disk, all solving the problem in different ways... One of them (a headless external program that forks to the background as you suggest) is the most advanced by far and I should probably just bite the bullet and publish that. From what I remember it makes the plug-in a dream to work with (no more locking up Vim, waiting for a 5 MB tags file to be written).

However the nice thing about vim-easytags right now is that people can drop it in their ~/.vim and it should Just Work (assuming Exuberant Ctags is installed of course). When I add this external program I see two choices:

1. Keep two parallel implementations up to date, one in Vim script and the other in my language of choice. Always testing both is going to be a pain but it's manageable. This would make vim-easytags work out of the box but with the option of being faster once people install the easytags executable on their system.
2. Force all users to always use the external program. This simplifies the Vim script code a lot, but the external program will have to be very well tested to guarantee that it works on all platforms and in all circumstances...

What do you think? Regardless of what I do with the plug-in published on Vim Online I can see about making the easytags executable available on GitHub so people can play around with it. I'm afraid the code has suffered some bit rot, e.g. I will probably have to make it compatible with file type specific tags files and such.

[xolox]

By the way, I'm wondering if using the Python interface for Vim for fingerprinting in Vim 7.2 would be worthwhile. Have you considered this already, or maybe even tried it?

[inkarkat]

Thanks for your insights; you've surely spent a lot of time thinking this through, that's great!

One of them (a headless external program that forks to the background as you suggest) is the most advanced by far and I should probably just bite the bullet and publish that. From what I remember it makes the plug-in a dream to work with (no more locking up Vim, waiting for a 5 MB tags file to be written).

I'm ambivalent about additional dependencies, too; for example, I'm still using ol' snipMate instead of the modern Ultisnips plugin because of its Python implementation. I've only started compiling my own Windows Vim binaries a few months ago; beforehand it was difficult to keep a matching Python implementation around.

I guess those most in need of easytags are hard-core programmers, not beginners, so additional dependencies aren't that problematic. I'm quite open to Python and Perl, a binary might work, too (compilation usually is a breeze on Linux, for Windows it would be nice to provide ready-made binaries). On the other hand, you'll invite more contributions from others (like me) if you stick to an easy scripting language.

One idea I had was to strip off the tags updating part of easytags into a separate Vimscript, and then just launch a non-interactive separate Vim process in the background (without loading .vimrc and other plugins). Though not as efficient as Python or binary, it might be a good compromise, and everybody probably has a terminal Vim available :-)

[inkarkat]

RE: Python interface for Vim 7.2 for fingerprinting

I usually have up-to-date Vim binaries, and I don't consider the added fingerprinting to be that revolutionary, so IMO it's not worthwhile to do, though certainly possible without that much effort.

[xolox]

One idea I had was to strip off the tags updating part of easytags into a separate Vimscript, and then just launch a non-interactive separate Vim process in the background (without loading .vimrc and other plugins). Though not as efficient as Python or binary, it might be a good compromise, and everybody probably has a terminal Vim available :-)

For some reason this seems really dirty to me :-) however it would definitely be portable and it would completely solve the code duplication issue. You definitely got me wondering, so I tried it out and it seems surprisingly fast:

peter@lucid-vm> time vim -u NONE -U NONE --noplugin --cmd 'call xolox#misc#msg#info("testing, 1, 2, 3") | scriptnames | quit'

… Vim startup screen snipped …

testing, 1, 2, 3

1: ~/Dropbox/Dotfiles/vim/autoload/xolox/misc/msg.vim

vim -u NONE -U NONE --noplugin --cmd   0.00s user 0.02s system 58% cpu 0.034 total

Hopefully I'll have some time to create a prototype to play around with in the next weekend.

[xolox]

I usually have up-to-date Vim binaries, and I don't consider the added fingerprinting to be that revolutionary, so IMO it's not worthwhile to do, though certainly possible without that much effort.

You're right, I want to move away from Vim + Python, not make the duplication worse :-)

[xolox]

Hey Ingo,

I created the branch async-cleanup and have started working on asynchronous tag updates for vim-easytags using only Vim (as we discussed previously in this pull request). If you're interested you can find an overview of the changes I made so far in pull request #49.

[inkarkat]

Peter,

that's great news! I especially like the fact that you were able to remove considerable portions of code; of course, the promise of no more Vim blocking during the tags update makes this feature highly desirable!

I've managed to get your async branch to work on Vim 7.3.823 on Windows/x64; you've probably already noticed the pull requests and issues I had to submit. I sincerely hope we can get this straightened out in the near future!