New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider relying on vim-dispatch #699

Open
Diablo-D3 opened this Issue Jun 20, 2013 · 79 comments

Comments

Projects
None yet
@Diablo-D3

Diablo-D3 commented Jun 20, 2013

tpope has a new vim plugin out for async jobs. Watch the introductory video.

http://vimeo.com/63116209

https://github.com/tpope/vim-dispatch#readme

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jun 20, 2013

@scrooloose is still missing in action, but I'm pretty sure he mentioned vim-dispatch in the past. So yes, we're aware of it.

My personal take is that vim-dispatch is nice and very well written, but it only supports quickfix lists, while syntastic needs loclists. An older patch adding support for async checking (see #370) uses AsyncCommand instead, which can handle both quickfix lists and loclists. I'd still prefer vim-dispatch, mainly because the code is cleaner. But for that to happen we'd have to either persuade @tpope to support loclists, or change syntastic to use quickfix lists (which in turn would make it conflict with :make and friends), or do some ugly dance to bridge quickfix lists with loclists (and still conflict with :make).

Well, we should wait for @scrooloose to get back.

@tpope

This comment has been minimized.

Contributor

tpope commented Jun 20, 2013

I don't really see these two plugins as a great fit together. Can you imagine a tmux pane opening for a split second every time you write a file? Gross.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jun 20, 2013

If it happens in background, why not? :)

@tpope

This comment has been minimized.

Contributor

tpope commented Jun 21, 2013

You mean other than two resizes and a potential view port shift?

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jun 21, 2013

Ah right, in tmux parlance "panes" are equivalents to Vim's windows. Ok, would that work better if the checkers would be sent to run in a background tmux window (equivalent to a background tab in Vim parlance)?

@tpope

This comment has been minimized.

Contributor

tpope commented Jun 21, 2013

Dispatch relies on the resize event to trigger the callback. Without that, there's no point in bothering with tmux at all.

@justinmk

This comment has been minimized.

justinmk commented Jul 22, 2013

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jul 22, 2013

@justinmk After scratching my head about it for a while, I don't see any useful way to interface vimproc with syntastic. Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jul 22, 2013

On a related topic: I played a little with AsyncCommand, and it isn't very robust. It appears to leave background processes behind, which can happily eat 99% of CPU cycles. And I wasn't even trying to crash it. :)

@justinmk

This comment has been minimized.

justinmk commented Jul 28, 2013

@lcd047 I'm surprised that vimproc doesn't fit this use case, but I haven't looked closely at it. AFAIK, there is no way to get passively notified of an external event in vim (without --remote which requires gvim). But using vim's &updatetime and CursorHold/CursorHoldI have been used by other plugins to reasonable effect. Why wouldn't that work for syntastic?

FWIW, I just bumped into another option: xolox's vim-misc combined with vim-shell. It is used by easytags to provided asynchronous behavior. See xolox#misc#os#exec(options) for usage of the vim-shell asynchronous capability.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jul 29, 2013

Well, vim-dispatch mentioned above uses tmux to send Vim a SIGWINCH when a background command finishes. Also, I believe --remote requires +clientserver, not a GUI.

As for CursorHold, I know about people using it in creative ways to achieve async effects, but I never bothered to find out exactly how that works. If it's possible to emulate a poll loop that way, I'm not familiar with it. The burden of proof is on you, not me. :)

Anyway, personally I'm not going to waste any more brain cycles on this. Async make should be implemented in Vim, not emulated by some horrible contortions in third-party plugins. It's 2013, job control has been understood really, really well by now. Patching Vim to do that would be relatively straightforward --- except qf_init_ext() is ~700 lines long, and the chances of such a patch becoming official any time soon seem pretty slim. And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

@justinmk

This comment has been minimized.

justinmk commented Jul 29, 2013

@lcd047 fair enough, I mostly agree.

For posterity: --remote does appear to require a GUI: https://code.google.com/p/macvim/issues/detail?id=431

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jul 29, 2013

@justinmk: Apparently it requires either OLE (that is, Windows), or an X-like clipboard (UNIX, Mac OS, or VMS). This means Vim needs to run in a (specific) GUI, but Vim itself doesn't need to be compiled with GUI support. I'm using this:

VIM - Vi IMproved 7.4b BETA (2013 Jul 28, compiled Jul 29 2013 09:10:27)
Compiled by root@euler
Huge version without GUI.  Features included (+) or not (-):

It can do remote commands as long as both the server and the client run in xterms, but not in console.

@justinmk

This comment has been minimized.

justinmk commented Jul 29, 2013

@lcd047 Very interesting! Good to know. It does work on Windows using any combination of {vim.exe, gvim.exe}.

Well, since it appears this github issue has become an accidental comprehensive survey of the state of "vim async support", perhaps it will save others the trouble in the future.

@maksimr

This comment has been minimized.

Contributor

maksimr commented Dec 17, 2013

+1 Need async for javascript :)

@idbrii

This comment has been minimized.

idbrii commented Jan 20, 2014

@lcd047 : Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")? AsyncCommand doesn't do any process killing, so if you do something like AsyncShell yes, then it will run forever (but isn't that what you'd expect)?


On topic:

Assuming I decrypted it properly, vimproc's main use of backgrounding is about running several things in parallel and polling for results. For syntastic we'd need a mechanism to run a single process in background, and get an event when the process is done. These are pretty different use cases.

Without using something like AsyncCommand (where you use --remote to send a command to vim), I don't think you can send an event, without polling for completion. So essentially, those are the same use cases except syntastic only fires one job. (Unless you mean vimproc blocks until all parallel jobs complete?)

I can't find the SIGWINCH call @lcd047 mentioned, but dispatch#callback is also using --remote. Maybe I'm missing something.

@tpope Wouldn't using syntastic and vim-dispatch's headless strategy make sense? I guess that would require adding a way to force a specific strategy to vim-dispatch? That seems like something that would be generally useful for other plugins to depend on too. (You could use it for an auto-exec repl, doc or tags lookup, and of course syntax checking.) Anywhere the user wants parsed output and not raw output.

@justinmk

This comment has been minimized.

justinmk commented Jan 20, 2014

@pydave I would keep an eye on tarruda's Vim fork to see where it goes. It's showing more promise than the floobits "timers" fork from October.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jan 20, 2014

Any chance you could file a bug to AsyncCommand for how you produced abandoned background processes ("which can happily eat 99% of CPU cycles.")?

Vanishingly small chances. As I said above, this proved to be a huge time sink, and I'm not going to waste any more brain cycles on it. If you care about it, I suppose you can retrieve the relevant setup in #370 and do your own testing. If you revive that patch (or if you come up with some other working solution), people test it extensively, and it proves to be stable, then I'll gladly include it. Until then, I'd rather spend my time doing something else.

I can't find the SIGWINCH call @lcd047 mentioned

The term to look for is VimResized. SIGWINCH is the signal that makes it possible.

@mitchmindtree

This comment has been minimized.

mitchmindtree commented Jun 3, 2014

Hey guys, any further word on a solid solution for async checking? If not, what is the best (/ least terrible) 3rd party solution that you've come across? I've recently been converted from Xcode to Vim land (a very liberating move), yet the one thing I miss a lot is Xcode's excellent multi-file async checking. Syntastic is already incredible, but I just wanted to add another "with async it would be totally awesome!". I hope the passion for this feature hasn't died yet :-) Or even better, if this has already been addressed with a recent feature that I've somehow managed to miss, let me know!

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jun 3, 2014

If you're happy with the set of languages it supports, YouCompleteMe is pretty good at what it does. And if you're not married to Vim, InteliJ IDEA, PyCharm, and friends are rather useful, too.

@gilligan

This comment has been minimized.

gilligan commented Jul 2, 2014

@lcd047 Sorry for reviving this but I was wondering if you are following neovim development and if you can say to what degree the async stuff they have included will be of use for syntastic at some point ?

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Jul 3, 2014

Sorry, I haven't followed neovim development recently. I'll probably look at it again when / if they actually produce some kind of release.

@s0undt3ch

This comment has been minimized.

s0undt3ch commented Nov 3, 2014

A look for the support if syntastic background checking led me to this ticket. Support for this would make syntastic even awesome'er!

@idbrii

This comment has been minimized.

idbrii commented Nov 3, 2014

Awhile back I found a fork of syntastic that tried to use idbrii/AsyncCommand to run syntax checking in the background. The initial and only commit stgpetrovic/syntastic-async@b3e3a9ba22b54aac6483e3b5ea545363d10fb304 (from a year ago) says:

First version that is workable with.

Will improve everything this is just the initial hack.

@s0undt3ch: if you're interested, you could try figuring out the problems with it. (Or whether it could work with vim-dispatch.)

(I haven't had time to look into VimResized that lcd047 mentioned above or actually mess with syntastic-async myself.)

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Nov 4, 2014

I suppose I should update you guys on this. The short version is, it should be possible to make syntastic do async checks using vimproc alone, but there would be a lot of work involved, most checkers would have to be refactored to take advantage of it, and the result is likely to be pretty fragile. Also speed would likely suck for some checkers, despite them being run asynchronously. A longer version follows, if you care about that kind of details.

The discovery process went something like this: vimlint pointed me to vim-watchdogs, which does async checks. Its approach is very different from syntastic, so the trick there can't be applied directly to syntastic; however, it pointed me to vim-quickrun, which is essentially a higher level frontend to vimproc. This again isn't directly useful for syntastic, but digging through the sources reveals how to do polling for an external event from Vim, using CursorHold and feeding it invisible keystrokes with feedkeys(). So it would be possible (at least in principle) to run the checkers with vimproc, poll for results, and trigger syntastic notifications accordingly. This seems better than all other options listed above, for many technical reasons.

Now there are two main problems with this approach. First, we'd need an entire framework for dealing with tasks, processes, and queues. This can be done, but it's a lot of work, and the documentation for vimproc is less than useful.

Second, one can only run processes in background, not Vim functions. Since the results of most checkers are further munged by Vim before being put in the final loclist, dealing with this would involve a major rewrite of the entire checking mechanism. Also, having a single *_GetLocList() function per checker would no longer be enough, it would have to be split into a function that would submit the checker command line to vimproc, and another function that would post-process the results. Furthermore, this post-processing stage would be run in foreground (since it would be a Vim function, not a process), in a CursorHold handler. The same CursorHold handler would also be responsible for task bookkeeping. That is likely to suck, badly. A possible solution to speed things up would be to do the post-processing in something like Python instead of VimL, but that would mean re-writing all checkers to use Python, which would suck even worse, albeit in a different way.

Last but not least: using feedkeys() is almost guaranteed to be fragile. If you're a fast typist and you already run into Vim occasionally misreading some key combinations while you type, feedkeys() would complicate this problem for you exponentially. shrug

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Oct 19, 2016

Here's an update for you guys. The plan goes something like this:

  • I'll make a 3.8 release, provided that no new bugs are reported the next few days. This shall be the stable branch. If new bugs are reported after the release I'll fix them, but I won't be adding any new features (or new checkers, for that matter). This may eventually evolve into a stable 3.9 if needed.
  • Right now I'm fixing the API for the checkers. Checkers will be able to use custom parsing functions instead of errorformat. Right now that means moving all JSON parsers from preprocess to the new parser sections of each checker, but there will be new parsing functions based on if_python and if_perl. This means some checkers will depend on Vim being compiled with Python or Perl.
  • Most internal checkers (hopefully all) will be migrated to the new API, but external third-party checkers will stop working. In particular checkers that rely on calling a Vim function instead of running a program (such as tsuquyomi and omnisharp) will break. I plan to allow old-style checkers to run synchronously (along with the async checkers), and I'll eventually provide some upgrade path for checkers running Vim functions, but neither would happen at the beginning.
  • The async functions still require a lot of bookkeeping. I'll upload a new branch named async when I have them actually doing something useful. They'll run with Vim 8.0, but they'll require a more recent Vim snapshot to work reliably (a few critical problems have been fixed in Vim in the mean time). At first they won't run with Neovim. There will be hooks for Neovim though, provided that somebody in the know is willing to fill in the necessary code.
  • A number of defaults will change. If you want to suggest changes to defaults to your favourite checkers, now is the time to do it. When the new branch gets released the defaults will again get locked for the next few years. 😄 Of course, I'll keep a detailed log with all incompatible changes, and I'll eventually update the docs, write an upgrade guide, etc.
  • There are a number of other problems to be addressed, f.i. how to handle the IsAvailable() functions. This is important because some checkers (hello, rubocop) take forever and a day to return a version. So version queries might need to be run asynchronously -- but doing so would turn the API for IsAvailable() into a pain in the rear. Let's hope some useful approach pops up eventually.
  • If you plan to write new checkers for syntastic after the 3.8 release, you'll have to use to the new API. And you'll have to do that based on whatever docs are available at the time. I don't plan to offer much hand-holding until the API stops changing. Sorry about that. Please feel free to explore using other checker engines in the mean time.
  • When the new API in the async branch stops changing I'll call the result syntastic 4.0, and merge it to master. All major issues would have to be fixed for that to happen. And you probably want to stay with the stable branch in the mean time.

That's the basic outline. Of course, that's just a plan. Plans do typically change.

@blueyed

This comment has been minimized.

Contributor

blueyed commented Oct 20, 2016

@lcd047
Thanks for the update and plans for a bright future.

Have you thought about using Neomake as a base for the new Syntastic?
Syntastic could be kept in maintenance mode as-is, and instead its features would get ported over to Neomake. Neomake has a quite decent testing framework already, and has support for both Neovim and Vim's async modes.

I'd rather not waste development and support effort. So if you do not like going the Neomake route then I might just put Neomake (from my side) into maintenance mode and help you out.. :)
But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Oct 21, 2016

@blueyed

Thanks for the update and plans for a bright future.

Oh absolutely, maintaining a healthy dose of skepticism is always a good idea. 😄

Have you thought about using Neomake as a base for the new Syntastic?

I have. I still think syntastic is a lot easier as a foundation for what I'm trying to do than Neomake would be. The vast majority of syntastic is also my code, so I'm already pretty familiar with it.

Neomake has a quite decent testing framework already,

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author. I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days. Activating Travis-CI is simple but not completely trivial if you haven't done it in a long time, it might take a few exchanges back and forth.

and has support for both Neovim and Vim's async modes.

I'm not worrying about Neovim at this point. With the design I'm going to use adding support for Neovim shall be either trivial, or very hard. I'm reasonably confident it's going to be trivial rather than very hard.

But please consider that Neomake had been created (not by me though) as a proof-of-concept to make Syntastic async.

As far as I can tell the point of Neomake was to create an asynchronous :make for Neovim, completely integrated with compiler sets and directly extending them. I think the author of Neomake considered syntastic grossly over-engineered. I don't think Neomake was ever meant to be an async syntastic.

Now, is syntastic grossly over-engineered? I'm sure it is. I'm a scientist that happens to know a few things about computers, not a programmer, and when I started all this I didn't know substantially more about Vim than the average user (despite having already used Vim for 15+ years at the time). It probably shows. However, there is a number of things in syntastic that are the way they are for a number of reasons. The author of Neomake was not aware of these reasons (which is by no means surprising, since errorformat and friends are a mine field), and that also shows. I suppose people these days address this problem by writing blog posts about their projects, where they basically keep chronicles about their reasons and their designs. But I'm an old fart, and blogs are way past my time. 😄

@mgedmin

This comment has been minimized.

mgedmin commented Oct 21, 2016

On a side note, syntastic doesn't have tests for a single reason: I'm not the owner of this repository, and activating Travi-CI for it would require the intervention of the initial author. I contacted him in the past and, while he was always nice, I did get a feeling he wants to put all this behind him, and he wouldn't appreciate being bothered with anything related to Vim these days.

It might be best to create a GitHub organization and ask the original maintainer to transfer repository ownership to the new org.

GitHub redirects old URLs when you transfer ownership, so existing links (and, more importantly, git checkouts) will continue to work.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Oct 21, 2016

@mgedmin I'm not particularly unhappy with the current arrangement, and there are more pressing issues at the moment than enabling Travis-CI. I might consider it later though.

@scrooloose

This comment has been minimized.

Collaborator

scrooloose commented Oct 21, 2016

Hey guys, I have also been thinking it would be a good idea to move syntastic out into an org - for a while now. @lcd047: you have been the main maintainer for several years now and it is wrong for it be under my name after all this time. For one, I am getting credit for your hard work, and two, for the practical reasons you outlined above.

Thoughts?

One thing I would consider is putting the checkers and the core in different repos. Then you could have a very open policy with adding maintainers to the checker repo. For me, it was the tedium of maintaining the checkers that I burned out on. Perhaps that is a solution if you feel the same.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Oct 21, 2016

@scrooloose You do deserve the credit, syntastic has been a very useful plugin for a years when I came around. I just tried to make it more maintainable, with some optimisations, improving the docs, and by adding some obvious features. But the overall design is yours, and it's beyond my league in creativity.

I'd be fine with whatever you decide, including a GitHub organisation. I'd also be fine with leaving it as it is, but yeah, there are a few practical problems that can only be solved by the owner of the project.

As for splitting the checkers from the core, that's probably a good idea. It would make sense to split them by groups of related languages, but that would leave the less popular ones in no man's land. In fact the long tail is probably more important than the main languages, since things like C/C++, Python, and Go already have plugins that can do much better than syntastic, simply by being more specialised.

@scrooloose

This comment has been minimized.

Collaborator

scrooloose commented Nov 3, 2016

OK, well that was scary but seems to have worked fine.

@lcd047 I have created the vim-syntastic org and added you as an owner (i.e. the highest permissions). I have then transferred syntastic into this org. All the links are redirecting fine (AFAICS) so it looks like a win.

Thanks for the kind words :-) I wouldn't undersell your hard work and persistence though. I think the syntastic project was very lucky that you came along just as I was running out of steam.

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Nov 3, 2016

@scrooloose Thank you, it seems to be working fine for me too.

tdooner added a commit to tdooner/.files that referenced this issue Nov 14, 2016

Switch from syntastic to neomake
Neomake supports Vim 8 background jobs, which makes life a lot better
for me.

Although it looks like Syntastic will likely eventually support
background jobs, I think it's worth switching now to neomake since it
was architected asynchronously from the ground up (no backwards
compatibility concerns) and also apparently has little-to-no test
coverage. Also, the maintainer [seems burned out][1].

It's been a good run, syntastic! Syntastic has saved me tons of time and
I owe a ton of productivity to the short feedback cycle it's provided me
for years now.

[1]: vim-syntastic/syntastic#699 (comment)

This was referenced Nov 22, 2016

@lcd047 lcd047 referenced this issue Dec 2, 2016

Merged

Hadolint #1921

lencioni added a commit to lencioni/dotfiles that referenced this issue Feb 20, 2017

Vim: remove Syntastic plugin
I'd like to try out Ale, a Vim plugin for syntax checking that works
asynchronously. It has been a good run with Syntastic, and perhaps I'll
return some day once it gets async support.

vim-syntastic/syntastic#699
@unphased

This comment has been minimized.

unphased commented Mar 8, 2017

Someone mentioned instability concerns; I use YCM with javascript and it already leaves tern stomping out 100% CPU semi-regularly. This is in no way encouraging this as something that is okay, because it so absolutely is not.

I really would prefer syntastic to stop blocking the UI because I have tmux binds that need Vim to stay responsive for the keybinds to stay responsive.

In fact, I'm so twitchy that even if I'm not using the binds I'll still end up typing movement characters or something, and it invariably makes the command line shift up and it garbles up vim's TUI render state, and all of this is because of blocking the UI.

Apologies for not reading the thread to the letter as that'd take the better half of a day which I'd be happy to devote if I could afford that at the moment. It sounds like architectural changes that will make syntastic even more amazing than it already is are in the pipes and they'll include robust async checkers. 👍

🎉

@mgedmin

This comment has been minimized.

mgedmin commented Mar 8, 2017

ALE is a Syntastic-like Vim plugin that works asynchronously. AFAICT it supports fewer languages than Syntastic, but you could check it out today instead of waiting for Syntastic to get support for async.

@JohanBjoerklund

This comment has been minimized.

JohanBjoerklund commented Mar 8, 2017

Neomake is also an alternative

@lcd047

This comment has been minimized.

Collaborator

lcd047 commented Mar 8, 2017

@unphased Sorry to disappoint you, but for the last few months I've been struggling with health problems. I'm a sick, old fart, and my priorities are those of a sick, old fart. Support for async will be ready when / if I can get myself together enough to finish it. Until then, you shouldn't really expect more than basic support for the existing code. In the mean time you could always:

  1. submit a patch to add support for async,
  2. fork syntastic, or
  3. use something else.

Not sure what else I could tell you. shrug

@gilbertwyw

This comment has been minimized.

gilbertwyw commented Mar 8, 2017

@lcd047 Hope you get well soon, i'm grateful for your great work on syntastic. 🙏

@unphased

This comment has been minimized.

unphased commented Mar 8, 2017

@lcd047 Absolutely, your contributions to the Vim ecosystem have NOT gone unnoticed!

@bradleyfalzon

This comment has been minimized.

bradleyfalzon commented Mar 8, 2017

I know +1s are frowned upon, but I also wanted to say how much I loved and used Syntastic, it was one of the first plugins I used with vim, and we used it extensively. Love your work, and thank you.

@willsALMANJ

This comment has been minimized.

willsALMANJ commented Mar 9, 2017

Thank you, @lcd047 for all your work on Syntastic. I hope you feel better soon. If working on Syntastic adds stress to your life and asynchronous support will be a lot of work, you might want to try out ALE that @mgedmin suggested. I have been testing it recently and it works well -- for the languages I use it is like a drop-in replacement for Syntastic except it doesn't hang my terminal when I save. I worry that by the time Syntastic has asynchronous support that most of the userbase will have migrated to something else like it.

@bounceme

This comment has been minimized.

bounceme commented Mar 12, 2017

#699 (comment)
there have already been several async linter runners made. ALE is one of the newest and will have the most changes and activity, sure, but tons of people have used syntastic for a long time and there is a huge amount of linter support here.

And, last but not least, I have exactly zero use for all this, I'm pretty happy with the sync checks. shrug

@ErikBjare

This comment has been minimized.

ErikBjare commented Nov 28, 2017

I just switched from syntastic to ALE which went pretty easy. I figured someone might wonder how to do that in a reasonable manner, so here's a basic port of my config: ErikBjare/dotfiles@2b946b5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment