How about joining up for Python code completion? (YouCompleteMe) #119

Open
Valloric opened this Issue Mar 25, 2013 · 40 comments

Comments

Projects
None yet
10 participants

[Replicating our private email discussion in a more public forum]

I'm the author of a YouCompleteMe (YCM), a recently released (Feb 2013) OSS code completion system for Vim. It's become very popular very quickly (~1300 stars on GitHub).

YCM comes with out-of-the box support for semantic completion for C-family languages like C/C++/Objective-C/Objective-C++ (using libclang as the engine) and also has a language-agnostic identifier completer. The GIF on the project page is a thousand words and there's also an extensive introduction there so I'll spare you the details that you can easily grab.

Getting to the point, an external developer contributed a Jedi-based semantic completer for YCM today. YCM actually provides a generic framework for writing custom semantic completers (for any language) that can then take advantage of the code completion infrastructure that YCM provides (non-blocking operation that lets Vim continue to process events, a smart caching layer, the subsequence-based completion filtering system and the advanced completion ranking algorithm etc). You can see just how small the file for jedi-based semantic Python completion is in YCM.

I see that Jedi (which is freaking awesome btw, congrats on that) also has other features too, like goto etc. Any chance on us teaming up to bring those features to YCM? I see that you have written the jedi-vim plugin. While I haven't tried it I'm sure it's awesome (hey, it's using Jedi) but I think it's sad that we have this duplication of work. I'd also like to see YCM become a common framework for code completion in Vim, much like Syntastic is the common framework for displaying compilation/lint errors.

YCM will expand into more IDE features with go-to-definition/declaration. Basically all the useful features that one can build when one has an engine that provides a semantic understanding of the codebase. These features would be accessed through the new :YcmCompleter [completer sub-command] [sub-command arguments] interface that also recently landed. That command calls OnUserCommand on the Completer object with the arguments the user provided after the sub-command name. From then on, the Completer writer can do anything he wishes.

So my idea is to expand the functionality of YCM's jedi_completer until it does everything jedi-vim can. That's it. I'll probably end up doing all this myself if no one else does but I'd love it if you could contribute pull-requests. Correct me if I'm wrong, but your goal is to build the best possible Python code completion system. You shouldn't have to spend time on the VimScript parts and other Vim plugin boilerplate. You should be able to focus on making Jedi, the library, even better.

If you're not interested, that's perfectly fine. I just think it's sad that there's this duplication of efforts; I spent a ton of time writing the ycm_core library in C++ so that completions in Vim can be super-fast. I'd love to see others leverage that work to improve editing in Vim even more.

Owner

davidhalter commented Mar 25, 2013

This was my answer:

Hey Val!

I don't have much time right now, I will answer you later in detail.

I really thank you for bringing this up. That's basically the idea of open source. Working together and improving things. It also comes in quite handy for me, because I don't really have time to maintain jedi-vim a lot, because of all the things going on in Jedi.

My main question would be here: Where does jedi-vim fit in? Would you deprecate jedi-vim in favor of YouCompleteMe? Or would it be more intelligent to let them co-exist and let YouCompleteMe do the work if available? The question is also relevant because some features in jedi-vim are probably Python-specific. Additionally jedi-vim plans to integrate a few refactoring things. What do you think about this?

~ David

PS: Could I post this mail and my answer on github? So that everybody can be a part of the discussion?

My main question would be here: Where does jedi-vim fit in? Would you deprecate jedi-vim in favor of YouCompleteMe? Or would it be more intelligent to let them co-exist and let YouCompleteMe do the work if available? The question is also relevant because some features in jedi-vim are probably Python-specific. Additionally jedi-vim plans to integrate a few refactoring things. What do you think about this?

I'd love to see refactoring commands in YCM! This is the reason why the OnUserCommand API was created. I know I'll be building more IDE features for C-family languages in YCM using libclang. I think this makes complete sense: since YCM already has an in-memory AST of your C/C++/ObjC file and has access to a library that provides a complete semantic understanding of the codebase, why not expose those features?

It would be great if features like these were also implemented in YCM for non-C-family languages like Python too. Naturally, those features would be Python specific; this is fine, there will be C++-specific commands too. The user would access them through :YcmCompleter [subcommand] [args]. That call would then be forwarded to the currently active Completer object (which is jedi_completer in Python files, clang_completer in C-family files etc). So there's nice decoupling here. The user can always add a mapping to reduce typing.

If you decide to start contributing to the jedi_completer in YCM, what happens to jedi-vim is up to you. It seems like you don't have time to maintain it and I've mentioned that I think there's a better way than duplicating the work for a good code completion Vim infrastructure. It's already built in YCM.

So jedi-vim... I don't know. Do you want to continue to develop it in parallel? That's certainly up to you. If you'd rather focus on jedi the library and jedi_completer in YCM and would rather turn down jedi-vim, put a deprecation notice at the top of your README pointing out that it's not being maintained anymore and point people to YCM (I feel like an egotistical dick just mentioning this).

Again, it's up to you. If you want to work on both in parallel, I'm fine with that.

Collaborator

dbrgn commented Mar 25, 2013

I haven't contributed much to jedi-vim, but I think this makes a lot of sense. I haven't tested YCM yet, but the description and GIF look awesome.

One important thing is not yet answered though: Is it possible to install YCM without all the C/C++ specific parts (e.g. Clang)? I'm using those things from time to time, but I'm pretty sure there are many people that don't, and that wouldn't want to install unneeded dependencies to complete their Python code.

As for the "migration process", I'd try to fix any important open issues and then add a notice that jedi-vim is not being developed anymore and that people should migrate to YCM.

Anyways, I'll try out YCM when I find the time to do so, but generally I'm all in favor of replacing jedi-vim with YCM, as long as the issue above is guaranteed. If there are any issues with YCM, I'll post them here or at the YCM github page.

One important thing is not yet answered though: Is it possible to install YCM without all the C/C++ specific parts (e.g. Clang)? I'm using those things from time to time, but I'm pretty sure there are many people that don't, and that wouldn't want to install unneeded dependencies to complete their Python code.

Yes, the default install does not build the clang parts. YCM still has a compiled component (ycm_core.so) but that's what makes YCM completions super-fast (installation is trivial and amounts to running ./install.sh). You can't get subsequence-based matching in VimScript and have it work fast (you may want to read my blog post on YCM for more details). Also you can't run a background thread in VimScript because that doesn't exist (you can in a native binary). The background worker threads prevent YCM from blocking Vim's GUI thread when you're querying for completions; AFAIK every other completion plugin for Vim blocks the whole UI until the answer comes back.

Collaborator

dbrgn commented Mar 25, 2013

Sounds great :)

Owner

davidhalter commented Mar 26, 2013

AFAIK every other completion plugin for Vim blocks the whole UI until the answer comes back.

You could probably use Python threads. But I never used them.

At the moment I'm not sure if it's a good idea to close jedi-vim (because YCM) is nowhere where jedi-vim is (pydoc support, function display, etc.). The first step would be to use YCM if available (if you know what I mean).

My personal problem is at the moment that I don't even have the time to do that (look at my commits in the last few months in jedi-vim). I only do Jedi stuff at the moment. So for me the big issue at the moment is my time.

A thought for you @Valloric: YouCompleteMe is VIM specific, why not create an engine (like Jedi), that could be of general use? This wouldn't necessarily mean that you need to split YCM. I guess there are quite a few IDEs which would love to have good autocompletion and do that all by themselves.

You could probably use Python threads. But I never used them.

The jedi_completer does in fact use a Python background thread. But you still need a native code binary to get fast subsequence-based matching.

At the moment I'm not sure if it's a good idea to close jedi-vim (because YCM) is nowhere where jedi-vim is (pydoc support, function display, etc.). The first step would be to use YCM if available (if you know what I mean).

Yep, those features will come.

My personal problem is at the moment that I don't even have the time to do that (look at my commits in the last few months in jedi-vim). I only do Jedi stuff at the moment. So for me the big issue at the moment is my time.

I completely understand; YCM takes up a lot of my time too. Feel free to send a pull request at some point if you feel the need/have time.

A thought for you @Valloric: YouCompleteMe is VIM specific, why not create an engine (like Jedi), that could be of general use? This wouldn't necessarily mean that you need to split YCM. I guess there are quite a few IDEs which would love to have good autocompletion and do that all by themselves.

I haven't discounted this option actually. Some emacs guys I know have talked about getting something similar for emacs by building on YCM as a base. But it's not really a current goal.

Owner

davidhalter commented Mar 27, 2013

subsequence-based matching

what's that?

Collaborator

dbrgn commented Mar 27, 2013

From the docs:

A critical thing to notice is that the completion filtering is NOT based on the input being a string prefix of the completion (but that works too). The input needs to be a subsequence match of a completion. This is a fancy way of saying that any input characters need to be present in a completion string in the order in which they appear in the input. So abc is a subsequence of xaybgc, but not of xbyxaxxc.

Yes, subsequence-based matching is awesome. It's actually the main reason why I started YCM; in a past life I used IDEs and a lot of the good ones have subsequence-based matching. The GIF on YCM's page shows how it works in practice: I can type gua and have GetUserAccount offered as a completion option. YCM also has a pretty smart completion ranking algorithm that will rank GetUserAccount higher in the completion menu for gua than, say, GetSamsungAccount because G, U and A are all word-boundary characters in GetUserAccount but aren't in GetSamsungAccount. There's a lot more logic to it, but the main gist is that YCM tries very, very hard to minimize the number of characters you have to type to reach a completion string you're looking for and minimize the number of TAB hits to get to that completion in the menu (with the goal of pretty much always having the completion you wanted as the first item in the menu).

+1 for getting jedi through YCM!
But as long as most jedi-vim features (goto and docstring at least) will not be available in YCM, I will not switch from jedi-vim.

inducer commented May 11, 2013

👍

While this is happening, I'd love it if I could at least load YCM and jedi-vim simultaneously and get the best of both worlds, without the two stepping on each other's feet. (Currently, this doesn't work properly--completions appear out of nowhere and such.)

Owner

davidhalter commented May 12, 2013

While this is happening, I'd love it if I could at least load YCM and jedi-vim simultaneously and get the best of both worlds, without the two stepping on each other's feet.

Absolutely. That's the important goal for now.

Just an update: since a few days ago, YCM now supports GoTo commands for Python, using the Jedi library (same as jedi-vim). See the YCM README for details.

Am I correct in saying that the last major features that jedi-vim has for Python that YCM needs to gain are:

  • Python documentation extraction and display
  • Inline function parameter display
  • Refactoring commands
Owner

davidhalter commented May 12, 2013

Refactoring commands have not been implemented yet. But that will probably soon happen. I think your right. That's about it.

Collaborator

dbrgn commented May 12, 2013

@Valloric great, in that case I need to update immediately :)

Collaborator

dbrgn commented May 27, 2013

I added YCM to the plugin list in the Jedi README: davidhalter/jedi@06cd975

:YcmCompleter GotoDefinitionElseDeclaration works great btw.

Collaborator

dbrgn commented May 27, 2013

Blogpost: http://blog.dbrgn.ch/2013/5/27/using-jedi-with-ymc/ (Correct me if something's wrong or missing.)

Thanks for adding it to the docs! I don't see anything wrong in the blog post (thanks for writing that too).

Valloric referenced this issue in Valloric/YouCompleteMe Jun 11, 2013

Merged

Semantic C# completer #365

Owner

davidhalter commented Aug 27, 2013

Just as a side-note: jedi-vim will not be deprecated. I'm probably including more features than YCM will. Also I'm much more flexible to test things.

That doesn't mean that I don't think YCM is great, it is!

Are there any issues at the moment, that those two wouldn't work well together?

YCM uses Jedi internally, but conflicts with jedi-vim. I don't see how it would be possible for the user to have both plugins running at the same time in a Python filetype.

YCM is still missing a few jedi-vim features, but I intend to add them all eventually.

how can i make sure i enable jedi-vim for python and others and ycm for ruby?

Owner

davidhalter commented Dec 17, 2013

I don't know. It's easy to disable jedi-vim features. I don't know how that would work with ycm.

rhlobo commented Mar 16, 2014

I am using jedi-vim together with YCM...

In order to prevent conflicts, I've disabled most of jedi-vim that is related to completion.
Doing that, I have not being facing any issues yet.

    " jedi-vim {
        let g:jedi#auto_vim_configuration = 0
        let g:jedi#popup_on_dot = 0
        let g:jedi#popup_select_first = 0
        let g:jedi#completions_enabled = 0
        let g:jedi#completions_command = ""
        let g:jedi#show_call_signatures = "1"

        let g:jedi#goto_assignments_command = "<leader>pa"
        let g:jedi#goto_definitions_command = "<leader>pd"
        let g:jedi#documentation_command = "<leader>pk"
        let g:jedi#usages_command = "<leader>pu"
        let g:jedi#rename_command = "<leader>pr"
    " }

I get to fully use YCM and still earn some extra nice jedi-vim features such as better argument completion.

To disable YCM for python i've put the following snippet in my .vimrc:

let g:ycm_filetype_specific_completion_to_disable = { 'python' : 1 }
let g:ycm_filetype_blacklist = { 'python' : 1 }

So far seems like everything is going as planned.

How is the conversation about a merger/integration going? You probably already thought about this, but how hard it is to make jedi-vim a module of YCM?

Owner

davidhalter commented Apr 2, 2014

@josecostamartins Both jedi-vim and ycm are using Jedi. Jedi is doing almost all of the work. The difference of jedi-vim and ycm in behavior and features is small.


I wrote a small answer detailing why I like jedi-vim to be a different project, you can read it below. However, the reason why I won't merge those two projects is basically my limited time. I'm working like 99% on Jedi and 1% on jedi-vim. I'm not really doing anything on jedi-vim, because I think Jedi is way more important for the future of the Python ecosystem and nobody else is going to do that work in the near future.
There will always be people that write VIM plugins, I don't fear that :-)

I'm very happy to see people improving YCM and jedi-vim. Seriously. I don't even look at them as competitors anymore. They are two ways of implementing Jedi. Both are fine. But if you're really interested in good autocompletion you'll start improving Jedi :-)


  1. YCM is quite a huge project, it involves compiling C++ code. I personally don't like compiling something that can also be done without compiling. Clearly that's not a big issue, but I just like to copy my VIM configs around and be ready to work.
  2. Jedi-vim is currently a very lightweight and simple Python autocompletion library. I like things that do one thing very well. It's just that those tool tend to stick with me for a long time. I tried a couple of the well known VIM plugins, but now I'm really back with a simple VIM configuration of just jedi-vim, tagbar, gundo, syntastic and supertab. That has been enough for me in the last two years. I would probably think about using YCM if I did more C-style programming.
  3. With its simplicity, jedi-vim has been a great way for testing and improving Jedi. It's going to be very easy for me to add new features or debugging capabilities to jedi-vim, since it's a very small code base and I have complete control over it.

On a side note, I still think that YCM hasn't tried to "reach the stars", by limiting itself to be a VIM autocompletion engine. I would much rather have see a library/engine like Jedi that drives autocompletions for all IDEs/editors to the next level. It's just stupid to write a hundred different autocompletion systems for each new editor.

Valloric commented Apr 2, 2014

On a side note, I still think that YCM hasn't tried to "reach the stars", by limiting itself to be a VIM autocompletion engine. I would much rather have see a library/engine like Jedi that drives autocompletions for all IDEs/editors to the next level. It's just stupid to write a hundred different autocompletion systems for each new editor.

I agree with this 100%. Several months ago YCM was split into a client-and-server model where the vast majority of the YCM logic lives in the ycmd completion server which has nothing to with Vim. There's also a thin client that runs in Vim that talks to ycmd.

Some time in the near future, ycmd will be split into a separate project so that other editors can build clients that talk to it.

I see ycmd as the one completion server that editors talk to; ycmd itself ships with and connects to various semantic engines (Jedi, libclang, omnisharp etc.) but provides one common interface to talk to all of them.

That's the idea at least.

Owner

davidhalter commented Apr 2, 2014

Awesome! :-) Well maybe one day we can merge our efforts... However, davidhalter/jedi#385 is going to be implemented first anyways.

Valloric commented Apr 2, 2014

Awesome! :-) Well maybe one day we can merge our efforts... However, davidhalter/jedi#385 is going to be implemented first anyways.

davidhalter/jedi#385 sounds like a great idea! I'd love to connect to a Jedi server instead of importing Jedi as a Python library. YCM already uses OmniSharpServer instead of calling into the OmniSharp library directly.

Having a server for this is the right approach. Some advice: use JSON over HTTP as the RPC interface. It's very easy to use and the overhead is negligible for local calls. This combo works great for ycmd, OmniSharpServer and Tern.js (server for JS code completion).

Owner

davidhalter commented Apr 3, 2014

Thank you for the detailed analysis. Doesn't surprise me at all. Since I know what Jedi does internally (it's like a crazy amount of things), the server won't eat too much. The big issue is actually implementing this for different Python versions and setting up a good protocol. Is there a protocol for autocompletion that you think looks like it should? If not, what's typically missing in those protocols? Examples?

Valloric commented Apr 3, 2014

Is there a protocol for autocompletion that you think looks like it should? If not, what's typically missing in those protocols?

Honestly I'd recommend looking at the ycmd's protocol for inspiration. It isn't documented anywhere but eventually will be when ycmd becomes it's own repository. Parts of the API won't be relevant, but a lot of it will be.

If you want to examine it, the server handlers are here. You should be able to follow the logic from there. The tests are certainly also worth taking a look. Here's what the completion tests look like.

This API is sound and has been working well for months now. The one thing I want to change before releasing the server and the API externally is make line and column numbers in the API be 1-based instead of 0-based since editor API's are all 1-based as are all diagnostic messages (compilation errors & warnings) shown to the user.

Valloric commented Apr 3, 2014

I'd also like to stress how important it is to focus on making your server API easy to interface with for clients. This is why HTTP+JSON is such a good combo here: every (sensible) language has great support for HTTP and JSON and both of those are very well understood by developers worldwide. Going with something like Thrift, zeromq or any kind of binary protocol makes no sense since you do not need that kind of performance (as proven by both the benchmarks I linked to and experience with ycmd over the last 6 months), but would still have to pay the cost of ease of use.

Owner

davidhalter commented Apr 3, 2014

Going with something like Thrift, zeromq or any kind of binary protocol makes no sense since you do not need that kind of performance (as proven by [..]

Yes I fully agree. Thank you for bringing this up again, it will make my life way easier defending this position. Just to quote Knuth again:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

HTTP/JSON is a good option. I don't even think there's a lot of good alternatives to transfer data like this. The only thing I would actually consider is to use Python's pickle, which is comfortable if your client is also written in Python (but I would prefer something language agnostic).

Honestly I'd recommend looking at the ycmd's protocol for inspiration.

Any other protocols? Or do you have any examples of how no to do it?

tkf commented Apr 4, 2014

Jumping in the thread via davidhalter/jedi#385

@Valloric

Some time in the near future, ycmd will be split into a separate project so that other editors can build clients that talk to it.

Sounds great. It would be nice to have Emacs plugin to talk to ycmd.

This is why HTTP+JSON is such a good combo here: every (sensible) language has great support for HTTP and JSON and both of those are very well understood by developers worldwide. Going with something like Thrift, zeromq or any kind of binary protocol makes no sense since you do not need that kind of performance

The reason why I proposed ZMQ in davidhalter/jedi#385 is that you need many-to-many communication not for performance, especially for Python/Jedi. Read davidhalter/jedi#385 for why. That said, I agree it is a good idea to have HTTP server "frontend" to communicate with editors ("translator") for easily accessible API. Probably that can be in ymcd: Jedi wrapper written around ZMQ and HTTP-based frontend in ymcd translating ZMQ to HTTP.

Owner

davidhalter commented Apr 4, 2014

Why do we need many-to-many connections? Sorry I don't understand the argument in #385.

tkf commented Apr 4, 2014

Let's say you have two processes running Jedi using Python 2 and 3, P2 and P3, and you have two editor processes E1 and E2. Connecting to E1 to P2 and P3 and E2 to P2 and P3 requires many-to-many connection. If you don't care about memory then you can launch a set of P2 and P3 for each editor but you still need many-to-one. Furthermore, if you want to mix the result of P2/P3 and doing subseq match on-the-fly then you'd want to something like iterator via RPC. I don't say it's impossible to do that with HTTP. It's just it's better to use a right tool at the right place; as I said, for editor-service communication HTTP is a good idea.

tkf commented Apr 4, 2014

@Valloric BTW, if you consider refactoring, then maybe you'd need a dialog-like UI. In that case, bi-directional RPC rather than HTTP could be a better option. Again, you can do that in HTTP with some more code. But I think HTTP is not a right tool to bring in this case too.

Valloric commented Apr 4, 2014

@tkf You're assuming that you want to have one Jedi server for several clients. I thought of the same for YCM, and it was unnecessarily complex (and limiting). The simplest and easiest way is for each client to start up its own copy of the completion server. This approach has been working for months without any issues for YCM.

If you try to have a single (or multiple) shared copies of the completion server, this limits what each client can do with it. Each client may want to start the completion server with different options to serve different needs. You may also want (or more likely, be forced) to store state on the server between completion requests. If you do store state, knowing that there's only one client talking to you simplifies things greatly.

My advice boils down to the KISS rule and YAGNI. Don't try to engineer something that works with multiple clients that you're unlikely to have; also, don't use a complicated protocol like zeromq that's designed for super-low overhead which isn't your concern.

In other words, don't design a cathedral. They're really hard to build. Design a small hut and build that, and if you need a cathedral later on (unlikely), extend the hut.

tkf commented Apr 4, 2014

@Valloric I consider your point already in my comment. See the last half after "If you don't care about memory..." about many-to-one connection. And don't miss the next comment about bi-directional RPC.

BTW, I understand and aware of most of your point about simplicity. My aim in davidhalter/jedi#385 was to start discussion. Nobody puts crazy ideas so I had to put ones.

In other words, don't design a cathedral.

Funny you say that. Isn't ycmd going to be a cathedral? I mean, if it is going to kill huge amount of birds (tools for many languages and many editors/IDEs) then it's a good cathedral :-)

spillz commented Apr 16, 2014

(Came here from jedi and found the discussion interesting)

Re http, isn't there an issue that on some systems (e.g. windows on a heavily locked-down network) using http even on localhost runs afoul of anti-virus/firewalls. Perhaps you could offer the option of using pipes as an alternative to http for RPC?

tssm referenced this issue in neovim/neovim Apr 30, 2014

Open

Question: Autocomplete Improvements #396

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment