Consider relying on vim-dispatch

[Diablo-D3]

tpope has a new vim plugin out for async jobs. Watch the introductory video.

http://vimeo.com/63116209

https://github.com/tpope/vim-dispatch#readme

[lcd047]

@scrooloose is still missing in action, but I'm pretty sure he mentioned vim-dispatch in the past. So yes, we're aware of it.

My personal take is that vim-dispatch is nice and very well written, but it only supports quickfix lists, while syntastic needs loclists. An older patch adding support for async checking (see #370) uses AsyncCommand instead, which can handle both quickfix lists and loclists. I'd still prefer vim-dispatch, mainly because the code is cleaner. But for that to happen we'd have to either persuade @tpope to support loclists, or change syntastic to use quickfix lists (which in turn would make it conflict with :make and friends), or do some ugly dance to bridge quickfix lists with loclists (and still conflict with :make).

Well, we should wait for @scrooloose to get back.

[tpope]

I don't really see these two plugins as a great fit together. Can you imagine a tmux pane opening for a split second every time you write a file? Gross.

[lcd047]

If it happens in background, why not? :)

[tpope]

You mean other than two resizes and a potential view port shift?

[lcd047]

Ah right, in tmux parlance "panes" are equivalents to Vim's windows. Ok, would that work better if the checkers would be sent to run in a background tmux window (equivalent to a background tab in Vim parlance)?

[tpope]

Dispatch relies on the resize event to trigger the callback. Without that, there's no point in bothering with tmux at all.

[justinmk]

What about vimproc? https://github.com/Shougo/vimproc.vim

[lcd047]

@justinmk After scratching my head about it for a while, I don't see any useful way to interface vimproc with syntastic. Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

[lcd047]

On a related topic: I played a little with AsyncCommand, and it isn't very robust. It appears to leave background processes behind, which can happily eat 99% of CPU cycles. And I wasn't even trying to crash it. :)

[justinmk]

@lcd047 I'm surprised that vimproc doesn't fit this use case, but I haven't looked closely at it. AFAIK, there is no way to get passively notified of an external event in vim (without --remote which requires gvim). But using vim's &updatetime and CursorHold/CursorHoldI have been used by other plugins to reasonable effect. Why wouldn't that work for syntastic?

FWIW, I just bumped into another option: xolox's vim-misc combined with vim-shell. It is used by easytags to provided asynchronous behavior. See xolox#misc#os#exec(options) for usage of the vim-shell asynchronous capability.

[lcd047]

Well, vim-dispatch mentioned above uses tmux to send Vim a SIGWINCH when a background command finishes. Also, I believe --remote requires +clientserver, not a GUI.

As for CursorHold, I know about people using it in creative ways to achieve async effects, but I never bothered to find out exactly how that works. If it's possible to emulate a poll loop that way, I'm not familiar with it. The burden of proof is on you, not me. :)

Anyway, personally I'm not going to waste any more brain cycles on this. Async make should be implemented in Vim, not emulated by some horrible contortions in third-party plugins. It's 2013, job control has been understood really, really well by now. Patching Vim to do that would be relatively straightforward --- except qf_init_ext() is ~700 lines long, and the chances of such a patch becoming official any time soon seem pretty slim. And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

[justinmk]

@lcd047 fair enough, I mostly agree.

For posterity: --remote does appear to require a GUI: https://code.google.com/p/macvim/issues/detail?id=431

[lcd047]

@justinmk: Apparently it requires either OLE (that is, Windows), or an X-like clipboard (UNIX, Mac OS, or VMS). This means Vim needs to run in a (specific) GUI, but Vim itself doesn't need to be compiled with GUI support. I'm using this:

VIM - Vi IMproved 7.4b BETA (2013 Jul 28, compiled Jul 29 2013 09:10:27)
Compiled by root@euler
Huge version without GUI.  Features included (+) or not (-):

It can do remote commands as long as both the server and the client run in xterms, but not in console.

[justinmk]

@lcd047 Very interesting! Good to know. It does work on Windows using any combination of {vim.exe, gvim.exe}.

Well, since it appears this github issue has become an accidental comprehensive survey of the state of "vim async support", perhaps it will save others the trouble in the future.

[maksimr]

+1 Need async for javascript :)

[idbrii]

@lcd047 : Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")? AsyncCommand doesn't do any process killing, so if you do something like AsyncShell yes, then it will run forever (but isn't that what you'd expect)?

---

On topic:

Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

Without using something like AsyncCommand (where you use --remote to send a command to vim), I don't think you can send an event, without polling for completion. So essentially, those are the same use cases except syntastic only fires one job. (Unless you mean vimproc blocks until all parallel jobs complete?)

I can't find the SIGWINCH call @lcd047 mentioned, but dispatch#callback is also using --remote. Maybe I'm missing something.

@tpope Wouldn't using syntastic and vim-dispatch's headless strategy make sense? I guess that would require adding a way to force a specific strategy to vim-dispatch? That seems like something that would be generally useful for other plugins to depend on too. (You could use it for an auto-exec repl, doc or tags lookup, and of course syntax checking.) Anywhere the user wants parsed output and not raw output.

[justinmk]

@pydave I would keep an eye on tarruda's Vim fork to see where it goes. It's showing more promise than the floobits "timers" fork from October.

[lcd047]

Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")?

Vanishingly small chances. As I said above, this proved to be a huge time sink, and I'm not going to waste any more brain cycles on it. If you care about it, I suppose you can retrieve the relevant setup in #370 and do your own testing. If you revive that patch (or if you come up with some other working solution), people test it extensively, and it proves to be stable, then I'll gladly include it. Until then, I'd rather spend my time doing something else.

I can't find the SIGWINCH call @lcd047 mentioned

The term to look for is VimResized. SIGWINCH is the signal that makes it possible.

[mitchmindtree]

Hey guys, any further word on a solid solution for async checking? If not, what is the best (/ least terrible) 3rd party solution that you've come across? I've recently been converted from Xcode to Vim land (a very liberating move), yet the one thing I miss a lot is Xcode's excellent multi-file async checking. Syntastic is already incredible, but I just wanted to add another "with async it would be totally awesome!". I hope the passion for this feature hasn't died yet :-) Or even better, if this has already been addressed with a recent feature that I've somehow managed to miss, let me know!

[lcd047]

If you're happy with the set of languages it supports, YouCompleteMe is pretty good at what it does. And if you're not married to Vim, InteliJ IDEA, PyCharm, and friends are rather useful, too.

[gilligan]

@lcd047 Sorry for reviving this but I was wondering if you are following neovim development and if you can say to what degree the async stuff they have included will be of use for syntastic at some point ?

[lcd047]

Sorry, I haven't followed neovim development recently. I'll probably look at it again when / if they actually produce some kind of release.

[s0undt3ch]

A look for the support if syntastic background checking led me to this ticket. Support for this would make syntastic even awesome'er!

[idbrii]

Awhile back I found a fork of syntastic that tried to use idbrii/AsyncCommand to run syntax checking in the background. The initial and only commit stgpetrovic/syntastic-async@b3e3a9ba22b54aac6483e3b5ea545363d10fb304 (from a year ago) says:

First version that is workable with.

Will improve everything this is just the initial hack.

@s0undt3ch: if you're interested, you could try figuring out the problems with it. (Or whether it could work with vim-dispatch.)

(I haven't had time to look into VimResized that lcd047 mentioned above or actually mess with syntastic-async myself.)

[lcd047]

I suppose I should update you guys on this. The short version is, it should be possible to make syntastic do async checks using vimproc alone, but there would be a lot of work involved, most checkers would have to be refactored to take advantage of it, and the result is likely to be pretty fragile. Also speed would likely suck for some checkers, despite them being run asynchronously. A longer version follows, if you care about that kind of details.

The discovery process went something like this: vimlint pointed me to vim-watchdogs, which does async checks. Its approach is very different from syntastic, so the trick there can't be applied directly to syntastic; however, it pointed me to vim-quickrun, which is essentially a higher level frontend to vimproc. This again isn't directly useful for syntastic, but digging through the sources reveals how to do polling for an external event from Vim, using CursorHold and feeding it invisible keystrokes with feedkeys(). So it would be possible (at least in principle) to run the checkers with vimproc, poll for results, and trigger syntastic notifications accordingly. This seems better than all other options listed above, for many technical reasons.

Now there are two main problems with this approach. First, we'd need an entire framework for dealing with tasks, processes, and queues. This can be done, but it's a lot of work, and the documentation for vimproc is less than useful.

Second, one can only run processes in background, not Vim functions. Since the results of most checkers are further munged by Vim before being put in the final loclist, dealing with this would involve a major rewrite of the entire checking mechanism. Also, having a single *_GetLocList() function per checker would no longer be enough, it would have to be split into a function that would submit the checker command line to vimproc, and another function that would post-process the results. Furthermore, this post-processing stage would be run in foreground (since it would be a Vim function, not a process), in a CursorHold handler. The same CursorHold handler would also be responsible for task bookkeeping. That is likely to suck, badly. A possible solution to speed things up would be to do the post-processing in something like Python instead of VimL, but that would mean re-writing all checkers to use Python, which would suck even worse, albeit in a different way.

Last but not least: using feedkeys() is almost guaranteed to be fragile. If you're a fast typist and you already run into Vim occasionally misreading some key combinations while you type, feedkeys() would complicate this problem for you exponentially. shrug

[blueyed]

@lcd047
Thanks for the update and plans for a bright future.

Have you thought about using Neomake as a base for the new Syntastic?
Syntastic could be kept in maintenance mode as-is, and instead its features would get ported over to Neomake. Neomake has a quite decent testing framework already, and has support for both Neovim and Vim's async modes.

I'd rather not waste development and support effort. So if you do not like going the Neomake route then I might just put Neomake (from my side) into maintenance mode and help you out.. :)
But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

[lcd047]

@blueyed

Thanks for the update and plans for a bright future.

Oh absolutely, maintaining a healthy dose of skepticism is always a good idea. 😄

Have you thought about using Neomake as a base for the new Syntastic?

I have. I still think syntastic is a lot easier as a foundation for what I'm trying to do than Neomake would be. The vast majority of syntastic is also my code, so I'm already pretty familiar with it.

Neomake has a quite decent testing framework already,

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author. I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days. Activating Travis-CI is simple but not completely trivial if you haven't done it in a long time, it might take a few exchanges back and forth.

and has support for both Neovim and Vim's async modes.

I'm not worrying about Neovim at this point. With the design I'm going to use adding support for Neovim shall be either trivial, or very hard. I'm reasonably confident it's going to be trivial rather than very hard.

But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

As far as I can tell the point of Neomake was to create an asynchronous :make for Neovim, completely integrated with compiler sets and directly extending them. I think the author of Neomake considered syntastic grossly over-engineered. I don't think Neomake was ever meant to be an async syntastic.

Now, is syntastic grossly over-engineered? I'm sure it is. I'm a scientist that happens to know a few things about computers, not a programmer, and when I started all this I didn't know substantially more about Vim than the average user (despite having already used Vim for 15+ years at the time). It probably shows. However, there is a number of things in syntastic that are the way they are for a number of reasons. The author of Neomake was not aware of these reasons (which is by no means surprising, since errorformat and friends are a mine field), and that also shows. I suppose people these days address this problem by writing blog posts about their projects, where they basically keep chronicles about their reasons and their designs. But I'm an old fart, and blogs are way past my time. 😄

[mgedmin]

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author. I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days.

It might be best to create a GitHub organization and ask the original maintainer to transfer repository ownership to the new org.

GitHub redirects old URLs when you transfer ownership, so existing links (and, more importantly, git checkouts) will continue to work.

[lcd047]

@mgedmin I'm not particularly unhappy with the current arrangement, and there are more pressing issues at the moment than enabling Travis-CI. I might consider it later though.

[scrooloose]

Hey guys, I have also been thinking it would be a good idea to move syntastic out into an org - for a while now. @lcd047: you have been the main maintainer for several years now and it is wrong for it be under my name after all this time. For one, I am getting credit for your hard work, and two, for the practical reasons you outlined above.

Thoughts?

One thing I would consider is putting the checkers and the core in different repos. Then you could have a very open policy with adding maintainers to the checker repo. For me, it was the tedium of maintaining the checkers that I burned out on. Perhaps that is a solution if you feel the same.

[lcd047]

@scrooloose You do deserve the credit, syntastic has been a very useful plugin for a years when I came around. I just tried to make it more maintainable, with some optimisations, improving the docs, and by adding some obvious features. But the overall design is yours, and it's beyond my league in creativity.

I'd be fine with whatever you decide, including a GitHub organisation. I'd also be fine with leaving it as it is, but yeah, there are a few practical problems that can only be solved by the owner of the project.

As for splitting the checkers from the core, that's probably a good idea. It would make sense to split them by groups of related languages, but that would leave the less popular ones in no man's land. In fact the long tail is probably more important than the main languages, since things like C/C++, Python, and Go already have plugins that can do much better than syntastic, simply by being more specialised.

[scrooloose]

OK, well that was scary but seems to have worked fine.

@lcd047 I have created the vim-syntastic org and added you as an owner (i.e. the highest permissions). I have then transferred syntastic into this org. All the links are redirecting fine (AFAICS) so it looks like a win.

Thanks for the kind words :-) I wouldn't undersell your hard work and persistence though. I think the syntastic project was very lucky that you came along just as I was running out of steam.

[lcd047]

@scrooloose Thank you, it seems to be working fine for me too.

[unphased]

Someone mentioned instability concerns; I use YCM with javascript and it already leaves tern stomping out 100% CPU semi-regularly. This is in no way encouraging this as something that is okay, because it so absolutely is not.

I really would prefer syntastic to stop blocking the UI because I have tmux binds that need Vim to stay responsive for the keybinds to stay responsive.

In fact, I'm so twitchy that even if I'm not using the binds I'll still end up typing movement characters or something, and it invariably makes the command line shift up and it garbles up vim's TUI render state, and all of this is because of blocking the UI.

Apologies for not reading the thread to the letter as that'd take the better half of a day which I'd be happy to devote if I could afford that at the moment. It sounds like architectural changes that will make syntastic even more amazing than it already is are in the pipes and they'll include robust async checkers. 👍

🎉

[mgedmin]

ALE is a Syntastic-like Vim plugin that works asynchronously. AFAICT it supports fewer languages than Syntastic, but you could check it out today instead of waiting for Syntastic to get support for async.

[JohanBjoerklund]

Neomake is also an alternative

[lcd047]

@unphased Sorry to disappoint you, but for the last few months I've been struggling with health problems. I'm a sick, old fart, and my priorities are those of a sick, old fart. Support for async will be ready when / if I can get myself together enough to finish it. Until then, you shouldn't really expect more than basic support for the existing code. In the mean time you could always:

1. submit a patch to add support for async,
2. fork syntastic, or
3. use something else.

Not sure what else I could tell you. shrug

[gilbertwyw]

@lcd047 Hope you get well soon, i'm grateful for your great work on syntastic. 🙏

[unphased]

@lcd047 Absolutely, your contributions to the Vim ecosystem have NOT gone unnoticed!

[bradleyfalzon]

I know +1s are frowned upon, but I also wanted to say how much I loved and used Syntastic, it was one of the first plugins I used with vim, and we used it extensively. Love your work, and thank you.

[wshanks]

Thank you, @lcd047 for all your work on Syntastic. I hope you feel better soon. If working on Syntastic adds stress to your life and asynchronous support will be a lot of work, you might want to try out ALE that @mgedmin suggested. I have been testing it recently and it works well -- for the languages I use it is like a drop-in replacement for Syntastic except it doesn't hang my terminal when I save. I worry that by the time Syntastic has asynchronous support that most of the userbase will have migrated to something else like it.

[bounceme]

#699 (comment)
there have already been several async linter runners made. ALE is one of the newest and will have the most changes and activity, sure, but tons of people have used syntastic for a long time and there is a huge amount of linter support here.

And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

[ErikBjare]

I just switched from syntastic to ALE which went pretty easy. I figured someone might wonder how to do that in a reasonable manner, so here's a basic port of my config: ErikBjare/dotfiles@2b946b5

[vitaly-zdanevich]

Still blocking only, with waiting a few seconds after file saving? :(