Asynchronous tag updates #49

Closed
wants to merge 13 commits into
from

Conversation

Projects
None yet
7 participants
Owner

xolox commented May 20, 2013

I would love to see asynchronous tag updating implemented and working so reliably that it can be made the default. In pull request #43 I discussed with @inkarkat how to implement this in a portable manner (without too much dependencies and machinery). He pointed out that I could always fork Vim to the background:

@inkarkat said:
One idea I had was to strip off the tags updating part of easytags into a separate Vimscript, and then just launch a non-interactive separate Vim process in the background (without loading .vimrc and other plugins). Though not as efficient as Python or binary, it might be a good compromise, and everybody probably has a terminal Vim available :-)

@xolox said:
For some reason this seems really dirty to me :-) however it would definitely be portable and it would completely solve the code duplication issue. You definitely got me wondering, so I tried it out and it seems surprisingly fast:

peter@lucid-vm> time vim -u NONE -U NONE --noplugin --cmd 'call xolox#misc#msg#info("testing, 1, 2, 3") | scriptnames | quit'

… Vim startup screen snipped …

testing, 1, 2, 3

1: ~/Dropbox/Dotfiles/vim/autoload/xolox/misc/msg.vim

vim -u NONE -U NONE --noplugin --cmd   0.00s user 0.02s system 58% cpu 0.034 total

Hopefully I'll have some time to create a prototype to play around with in the next weekend.

So a bit later than estimated, but here's a start. This code definitely still has some issues, so instead of simply releasing it as the default I'm keeping it in a feature branch until I trust it a bit more :-).

xolox added some commits May 19, 2013

@xolox xolox Remove Python highlighting code (to be replaced with Vim script..)
These are the first steps (ripping out the Python code) towards
implementing asynchronous tag updates using only Vim as outlined
in pull request #43:

  #43
aa65f9f
@xolox xolox Remove update detection (using sha256())
In an asynchronous configuration, update detection seems to require
bidirectional communication between the two Vim instances (which will
get messy). Update detection was nice to have, but in the branch where
I'm currently working, asynchronous tag updates have priority :-).
I'll see about resurrecting the update detection later.

                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
b918cd4
@xolox xolox Remove special casing of "first run"
I removed the special casing of the "first run" that used Exuberant Ctags
to directly create non-existing tag files. This additional code path
complicated the plug-in for an exceptional circumstance that wouldn't
be slow even if it was handled using the generic code. In other words,
out it goes in the quest for asynchronous tag updates!

                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
3408ba9
@xolox xolox Split xolox#easytags#update() into 2 phases: initialization & execution
                                * * *

These are the first steps towards implementing asynchronous tag updates
using only Vim as outlined in pull request #43:

  #43
f8f99c8
@xolox xolox Initial implementation of asynchronous tag updates*
Based on the refactorings in the last four change sets, this change set
introduces initial support for asynchronous tag updates using only Vim
as outlined in pull request #43:

  #43

* I butchered a lot of code (including a few features) to get
  asynchronous tag updates working correctly. I'm not sure yet how
  successful the asynchronous tag updates feature will be (ideally it
  can become the default on all platforms) so I'm also not sure yet if I
  will re-implement the butchered features :-)
6ab882d
Owner

xolox commented May 20, 2013

The first three commits simply rip out the Python code and any functionality that conflicted with (or significantly complicated) the asynchronous update process. What's nice about this is the diff --stat output:

 README.md                   |   41 +-----
 autoload/xolox/easytags.vim |  386 +++++++++++++++++++++----------------------
 doc/easytags.txt            |  105 ++----------
 misc/easytags/highlight.py  |   55 ------
 plugin/easytags.vim         |   21 +--
 5 files changed, 220 insertions(+), 388 deletions(-)

So by adding asynchronous tag updates I've actually reduced the amount of code significantly :-). That's not to say that the feature is finished and ready to release, but I digress.

Owner

xolox commented May 20, 2013

Issues I noticed with this code so far:

  1. Once or twice now while I was working on vim-easytags, an empty Vim window popped up without any indication as to why that happened. Of course this Vim window was meant to be used for an asynchronous tag update, but somehow it didn't happen.
  2. In 3408ba9 I removed the special casing of the "first run" without thinking about it too much, but I guess it was there for a reason: When Exuberant Ctags reports tags to standard output, it doesn't include a header, so tags files will now be created without any headers...?
  3. In aa65f9f I ripped out the Python code to make syntax highlighting faster, but I did not implement an alternative yet. I'm considering moving most of the highlighting process to the asynchronous background processes as well, but have to give it a bit more thought...
  4. A couple of times now I've had Vim crash on me since I started using the asynchronous branch during unrelated work. I used the verbosefile option to capture a message log when it happened, but the log was buffered so it stopped before the crash ಠ_ಠ. I guess I'll have to dig out my notes on compiling Vim with debugging symbols and running it under gdb. Already looking forward to the experience ಠ_ಠ.

Edit: So far I've been putting of recompiling Vim with debugging symbols, but I do have a suspicion what the problem might be: Maybe remote_expr() is not always save to call? The only problem then would be: How can I tell when it's safe and when it's not? (before crashing Vim :-)

xolox and others added some commits May 20, 2013

@xolox xolox Don't update tags when loading a session (too disruptive) c8e2ab5
@inkarkat inkarkat Move the shellescaping of the command to stage 2.
On Windows, I get the following error
,----
| Error detected while processing pre-vimrc command line:
| E115: Missing quote: 'ctags --fields=+l --c-kinds=+p --c++-kinds=+p --sort=no -f-
`----
from the following command
    "C:\Program Files\vim\vim73\gvim.exe" -u NONE -U NONE --noplugin -N --cmd "let &rtp = 'D:/A/.vim,C:\Program Files\vim/vimfiles,C:\Program Files\vim\vim73,C:\Program Files\vim/vimfiles/after,D:/A/.vim/after' | call xolox#easytags#update_remote({'cmdline': 'ctags --fields=+l --c-kinds=+p --c++-kinds=+p --sort=no -f- ""--language-force=vim"" ""D:\A\Unixhome\.vim\autoload\xolox\misc\path.vim""', 'by_filetype': '', 'tagsfile': '\A\Unixhome\.vim\tags', 'servername': 'GVIM', 'verbose': 1, 'have_args': 0, 'starttime': [400, 326843564], 'silent': 1, 'filter_tags': 0, 'cache': {}})"

The problem is in the doubled double quotes (""), resulting from xolox#misc#escape#shell() applied twice, one time for the shell execution in stage 2, and once for launching stage 2 from stage 1. Unfortunately, shell escaping is a hairy mess. Let's work around that by passing the individual command args as the original list to stage 2, and apply the shell escaping and joining there.
a126c30
@xolox xolox Merge branch 'dev' into async-cleanup 0639181
@xolox xolox Merge branch 'master' into async-cleanup e5358cb
@inkarkat inkarkat In stage 2, don't touch the .viminfo file and don't create swap files.
I've seen the tags update clobbering my ~/.viminfo history, especially because stage 2 doesn't run with my .vimrc settings and therefore with default sizes for the retained history.
Also, turn off swap files; though stage 2 doesn't :edit any files, don't run the risk of accidentally littering the file system with swap files.
38b955a
@xolox xolox Merge pull request #50 into async branch
Pull request #50 on GitHub:
  #50
85fbc29
@xolox xolox Extract xolox#easytags#call_remote() from xolox#easytags#update() 07cc283
Owner

xolox commented Jun 3, 2013

@inkarkat: I just posted an addition to my list of issues above:

  1. After fixing xolox/vim-easytags#51 I was able to try the asynchronous tag updates on Windows, but it turns out to be quite annoying: When Vim has no interface to show a message printed with :echomsg, it will pop up a dialog showing the message. For me it does this every single time an asynchronous tag update runs. I'll have to silence those messages somehow (preferably without losing the ability to easily debug the plug-in).

I assume you are testing the asynchronous branch on Windows. Have you noticed this issue? Maybe you found an easy way to fix it? I guess I can always use something like the verbosefile option, but I'd rather not :-)

Contributor

inkarkat commented Jun 3, 2013

Yes, GVIM does that. The way I've worked around that is by (first hard-coding, now configuring)

let g:xolox#misc#os#vim_progname = 'vim'

This always uses terminal Vim, which does not show this behavior. Why do you need to always echo messages from stage 2 (I could understand during debugging); that whole process isn't visible to the user, anyway?

Owner

xolox commented Jun 3, 2013

You're right that it doesn't make sense to echo messages in an invisible window, however easytags relays the messages to the user's Vim instance. I was kind of abusing the existing :echomsg infrastructure for that :-). However I like your approach of always running the console Vim; that definitely won't have any graphical interaction, which is exactly what we need. I changed xolox#misc#os#find_vim() to accept the string vim or gvim so callers can indicate their preference. I also changed the async branch to pass the vim argument to xolox#misc#os#find_vim(). Thanks for the tip :-)

Contributor

inkarkat commented Sep 3, 2013

I've noticed some recent fixes going into master that haven't yet been merged into the async-cleanup branch yet. I hope you're not abandoning that branch; I've been using it without problems for the past 3 months!

My vote is clearly for including this in the mainline now. Because maintaining both synchronous and async modes is a lot of (testing) effort, I would recommend to completely switch over to async (in a new alpha release), and only go down the route of configurable modes if we encounter many async problems that cannot be fixed easily. As I said, I found the original plugin too slow, but I'm happily running the async branch (mostly with a 5 MB tags database of > 2000 Vimscript files).

jcf commented Jan 25, 2014

Have you considered adding support for vimproc? Something like the system_bg function would handle shelling out to ctags, without the need to support directly in easytags.

You could detect if it's installed, and when available use the vimproc function to run ctags, and when it's not run the call synchronously…

If I were to create a PR with this feature, would you consider merging it?

sjdrodge commented Feb 1, 2014

+1 for @jcf's suggestion. vim-easytags is currently unusable for me because it sometimes blocks for several seconds after I open a new file. Vimproc support is by far the easiest way to implement async, and even if you don't like it as a permanent solution, it would be great in the meantime.

Contributor

inkarkat commented Feb 1, 2014

@sjdrodge Try out this async branch; it works very well for me (no more annoying blocking indeed)!

And another bump @xolox to please have this integrated with the mainline!

aktau commented Feb 27, 2014

This does not seem to work on osx (mavericks) homebrew vim on the commandline. Most likely because vim remote doesn't work on it. When I run vim --servername ... --remote-expr ... it just gives me that --servername is not recognized... I've done some reading and it appears X11 is required for vim remote. A bit sad as it looked really nice.

async branch doesn't work for me either on osx mavericks. I get a bunch of errors about not being able to find syntax files.

Contributor

inkarkat commented Jun 17, 2014

I'm happy to see you updating the master branch again. Do you plan to merge those updates to this async branch, too, or have you abandoned it?! I still urge you to integrate this soon; I'm still happily using it (and can't imagine going back to the old blocking behavior)!!!

Owner

xolox commented Jun 17, 2014

@jcf, @sjdrodge

The problem is not with asynchronous command execution so using vimproc is not a solution. That part was solved when I opened this feature branch.

@inkarkat

I haven't exactly abandoned this feature branch, but I'm also not very enthusiastic about merging it into master as is, because the last status update from me was that it had a tendency to sporadically crash my GVim on Linux, which is totally unacceptable and a blocker for merging it into master.

I just tried merging master into the feature branch and this has already become too complex for my taste. I have to resolve merge conflicts in my day job as well and having to do it when I get paid to do it is one thing, but this is another :-). What I'm considering now: Merging async-cleanup (maybe selectively) into master and hiding it behind feature flags. That would solve the most urgent problems:

  • All users can decide for themselves whether to test the async mode or keep using blocking mode;
  • The feature / code won't suffer bit rot
  • No more merge hell for me

That is not to say that this will be an easy undertaking, but I do still believe that this (async mode) is the future for vim-easytags (assuming it has a future, but given that it's by far my most popular plug-in together with vim-notes I expect it does) so it should be worth the effort (I would personally love it if Vim never blocks / hangs again).

@ everyone

I hope to post an update here soon. Feel free to bug me if I don't ;-)

Contributor

inkarkat commented Jun 18, 2014

I fully agree that a merge to master as an optional feature is the way to go; in fact, I would have done so earlier, before the branches diverged so much. I hope you'll find the time to go through this; I know it won't be trivial. I'm definitely willing to test the result.

As there's already quite a lot of code, I'd recommend to create separate autoload/xolox/easytags/async.vim and .../sync.vim, and only leave the general functions in the original autoload/xolox/easytags.vim.

@xolox xolox added a commit to xolox/vim-misc that referenced this pull request Jun 22, 2014

@xolox xolox Generalized asynchronous Vim script evaluation (for vim-easytags)
The idea of forking Vim into the background to perform long running tasks
that would otherwise block the user's editing session originates from the
'async-cleanup' feature branch of vim-easytags, where I picked it up from
a tip by Inko Karkat (@inkarkat). Because I can see this being useful in
several of my Vim plug-ins I decided to generalize the concept and
implement it in vim-misc.

One thing that may not be immediately obvious from the commit is why I
implemented two ways for the results of asynchronous processes to feed
back into the main Vim process. The reason is twofold, both having to do
with Vim's client/server implementation:

 1) Reading through the pull request of the mentioned feature branch it
    looks like Vim's client/server support on Mac OS X is either missing
    or problematic.

 2) Running the 'async-cleanup' branch (which always used client/server
    communication) I saw several crashes on Linux. I have no proof but
    suspected the client/server communication to be the cause of this.

So now there's an alternative if the client/server mechanism starts to
act up (which it may very well do because it's a complex part of Vim).

See also the pull request for the mentioned feature branch:
  xolox/vim-easytags#49
81fdec8
Owner

xolox commented Jun 22, 2014

Hi all,

I'm almost done reimplementing this on top of master. I took a very different route this time, pushing most of the complexity over to vim-misc where I can use it in my other Vim plug-ins. I just committed xolox/vim-misc@81fdec8 (reformatted commit message below). My next commit will integrate this new functionality into easytags (mostly done already).


Generalized asynchronous Vim script evaluation (for vim-easytags)

The idea of forking Vim into the background to perform long running tasks that would otherwise block the user's editing session originates from the 'async-cleanup' feature branch of vim-easytags, where I picked it up from a tip by Ingo Karkat (@inkarkat). Because I can see this being useful in several of my Vim plug-ins I decided to generalize the concept and implement it in vim-misc.

One thing that may not be immediately obvious from the commit is why I implemented two ways for the results of asynchronous processes to feed back into the main Vim process. The reason is twofold, both having to do with Vim's client/server implementation:

  1. Reading through the pull request of the mentioned feature branch it looks like Vim's client/server support on Mac OS X is either missing or problematic.
  2. Running the 'async-cleanup' branch (which always used client/server communication) I saw several crashes on Linux. I have no proof but suspected the client/server communication to be the cause of this.

So now there's an alternative if the client/server mechanism starts to act up (which it may very well do because it's a complex part of Vim).

@xolox xolox added a commit that referenced this pull request Jun 22, 2014

@xolox xolox Support for synchronous + asynchronous tags file updates (huge refact…
…oring)

See also pull request #49 for my previous and failed attempt:
  #49
27c29aa
Owner

xolox commented Jun 22, 2014

Hi all,

I created a new feature branch (async-take-two) and a new pull request (#84). Let's continue the discussion there. I'll close this now because the async-cleanup feature branch has suffered bit rot and I'm not going to revive it.

Thanks for all of the feedback!

xolox closed this Jun 22, 2014

@xolox xolox added a commit that referenced this pull request Jun 29, 2014

@xolox xolox Silence asynchronous tags file updates by default
This change is related to pull request #82 however that pull request
wasn't merged here (and won't be merged at all) because it was based on
the old/dead `async-cleanup' feature branch (see pull request #49 on
GitHub) instead of the new `async-take-two' feature branch (see pull
request #84 on GitHub). This change set implements the equivalent on the
new feature branch.

In addition to Ingo's comments in pull request #82, the asynchronous
message frequently disturbs me while typing a Vim command, which is kind
of annoying. If everything goes well and we can get the async mode to be
stable enough to become the default mode then the status messages will
only be interesting when debugging a problem anyway.
55d7e31

@xolox xolox added a commit that referenced this pull request Jun 29, 2014

@xolox xolox Disable automatic tags file updates during :vimgrep
This change is related to pull request #83 however that pull request
wasn't merged here (and won't be merged at all) because it was based on
the old/dead `async-cleanup' feature branch (see pull request #49 on
GitHub) instead of the new `async-take-two' feature branch (see pull
request #84 on GitHub). This change set implements the equivalent on the
new feature branch (without introducing another option).
6c7a663

xolox deleted the async-cleanup branch Jul 8, 2014

Owner

xolox commented Jul 8, 2014

In case anyone who participated in this discussion didn't notice yet: I created a new feature branch in #84. That feature branch is now merged so you can try out the feature by pulling the latest changes on master and enabling an option.

Reading through this discussion once more I notice the bug reports about Mac OS X and Vim's client/server support. The new feature branch can use Vim's client/server support but it can also work without it, so it might Just Work on Mac OS X. If it doesn't work out of the box on Mac OS X feel free to report a new issue on GitHub, I don't think it will be a lot of effort to make it work out of the box on Mac OS X (assuming it doesn't already; I can't easily test).

aktau commented Jul 9, 2014

@xolox just a heads up, for async updates it might be possible to defer to jobs or async_send on neovim to simplify the whole ordeal quite a bit:

Some issues that could be relevant:

I'm not entirely sure if the events or the jobs feature is most appropriate, @tarruda could probably expound on that :).

Owner

xolox commented Jul 9, 2014

@aktau: Thanks for the heads up. I was actually completely unaware of neovim until yesterday evening, then again I haven't been following vim-dev for quite a while. I assume these mechanisms are backwards incompatible with "stock" Vim, right? But I read something about it being 99% compatible, so I suppose I could probe for supported mechanisms using exists()? I would need to actually start using neovim then though... I hope it's easy to compile on Linux/GTK? :-)

One thing to note about the neovim proposal is that I would never break compatibility with "stock" Vim, because as I said I'd never heard of neovim until yesterday evening so I'm not going to hedge my bets on neovim just yet (with all due respect because I do think it's awesome what you guys are doing).

aktau commented Jul 9, 2014

One thing to note about the neovim proposal is that I would never break compatibility with "stock" Vim, because as I said I'd never heard of neovim until yesterday evening so I'm not going to hedge my bets on neovim just yet (with all due respect because I do think it's awesome what you guys are doing).

Don't worry, we don't expect anyone to drop vanilla Vim compatibility unless they're developing from scratch and really want to use neovim-only features. We do our utmost to maintain backwards compatibility.

But, there are (and will be) has and/or exists for features that neovim supports, so developers can make use of a simpler and hopefully faster/bugfree codepath when the feature is detected.

Perhaps, if it proves popular, Bram might consider integrating a specific feature in vanilla Vim (though I don't see it happening any time soon, especially the async features which would be quite difficult to implement with Vim's architecture).

tarruda commented Jul 9, 2014

@xolox You can test if a script is running under neovim using the has('neovim') expression. The best way to run asychronous tag updates would be to use the job control feature, I have published a demo script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment