deoplete-jedi doesn't recognize some candidates in large packages as numpy

[monkoose]

Problem summary

#89 partially fixes deoplete-jedi candidates recognition compare to other editors, but tested with numpy(even larger package then django and more comlex) and seems like it still cant show some candidates that JEDI(and jedi-vim) can. Tested in atom and vscode too. They both can complete(atom need like 5-10 sec to actually recognize this candidates. Yes numpy is huge), vscode much faster but still delays 2-3 seconds before comletions show up.
I dunno - maybe it's too much for deoplete-jedi and deoplete itself? Because my neovim even freeze for few seconds when im trying to get this completions with jedi-vim. And even when i dont complete with jedi-vim and just type something like np.cos() for the first time per session neovim freezes(i guess because of g:jedi#show_call_signatures = 2). Will test it later.
Some screenshots
deoplete-jedi

jedi-vim

atom

Atom an VScode after they actually detect this candidates, then completion is fast(i suppose because of cache). Jedi-vim too. Deoplete-jedi still cant detect them even when you already have them in your file(deoplete for sure completes them as simple keyword).

As you can see from jedi-vim screenshot actual path for sin is numpy.umath.sin. And as expected deoplete-jedi completes it without a problem

So i dont really know should we try to fix this. Because it's laggy. Just posted for the record.

• Neovim version:
  0.1.7

[zchee]

@monkoose Thanks for creating new issue. I'll check it.

[Shougo]

I think deoplete-jedi server is freezed by numpy analyses.

[zchee]

@Shougo Ah, but as I said before on #89 (comment), deoplete-jedi using json cache for the each module first method.
So, I think not freeze, just wrong cached the module methods. or wrong behavior at as import.

[monkoose]

As i said i guess neovim freezes because of jedi-vim call signatures. Will test it tomorrow.
I dont think that deoplete-jedi freezes neovim if it doesn't even "collect" this complex candidates.

[tweekmonster]

deoplete-jedi isn't freezing. The problem is that deoplete-jedi is (correctly) caching the results. I'm not sure why this is, but Jedi doesn't return the imports that are in numpy/core/__init__.py on the first pass. If I force numpy to refresh the cache on each completion, it shows np.sin and all of the other imported entities after 3 completion attempts.

With the following:

diff --git a/rplugin/python3/deoplete/sources/deoplete_jedi.py b/rplugin/python3/deoplete/sources/deoplete_jedi.py
index 14a2f20..26cd14c 100644
--- a/rplugin/python3/deoplete/sources/deoplete_jedi.py
+++ b/rplugin/python3/deoplete/sources/deoplete_jedi.py
@@ -199,6 +199,9 @@ class Source(Base):
             # refresh cached items that did not have associated module files.
             refresh = True

+        if cache_key and cache_key[-1] == 'package' and 'numpy' in cache_key[-2]:
+            refresh = True
+
         # Extra options to pass to the server.
         options = {
             'cwd': context.get('cwd'),

In a script like:

import numpy as np

np.si|

Where | is the cursor, press A<esc>A<esc>A<esc> and wait about 5+ seconds (or watch the logs), then press A, all of numpy's members that start with si will appear in the completions.

[zchee]

@tweekmonster Thanks detail and debug. Hmm, strange...
The statically cache is one of the features of deoplete-jedi, such as in particular for numpy. troubled :(
I'll also dig it.

[blueyed]

Might be related to davidhalter/jedi#784?!

[zchee]

@blueyed Good reference for me.
Might be successful if using your bisect-ed commit hash...? that's too old commit, but it's worth a try.

Edit: I saw added ClearCache command commit to jedi-vim related that issue thread. It means if clear jedi's cache, works fine?
But if so, we already disable jedi's cache. Hmm...

[tweekmonster]

Might be related to davidhalter/jedi#784?!

I'm not sure. I don't know enough about how Jedi works, but I believe something else is happening internally. Like the internal state incrementally changes on each call until it's able to resolve the imports at numpy/core/__init__.py's depth.

Also, because of that linked issue, I thought maybe changing call_signatures_validity to a higher value might help, because when numpy.sin finally can be resolved, it take almost 3 seconds (the default) before completing. It had no effect. Then, I started messing around and increasing other settings until I got to this point:

settings.fast_parser = False
settings.max_executions_without_builtins = 50000000
settings.max_executions = 50000000
settings.scale_call_signatures = 10000000
settings.max_until_execution_unique = 50000000
settings.max_function_recursion_level = 50000000
settings.max_executions = 50000000
settings.star_import_cache_validity = 50000000
settings.call_signatures_validity = 50000000
settings.use_filesystem_cache = False

I made it so that deoplete-jedi's caches were always invalid. All of these high values did basically nothing from my tests. Completions always finished in about the same amount of time, including numpy (but it still requires 3 completions before resolving numpy.sin). That isn't right, is it? If Jedi was able to recurse that many times into numpy it should take a long time, right?

[monkoose]

@tweekmonster Yes, you are right about refreshing the cache. Actually atom editor and it autocomplete-python(with jedi) behaves similarly. I didn't describe it in my post. You should type np.sin, then delete sin and type again and wait some time(5+ seconds). And then it can complete all other staff like np.arange, np.cos, etc. without a problem.

I checked my assumption about call_signatures of jedi-vim. And yes it is it that cause freeze of neovim. Maybe because it isn't async or i don't know.
If you have jedi-vim you can test it with g:jedi#call_show_signatures = 2 print np.cos then open round bracket np.cos( and neovim should freeze(at least it does for me) for few seconds. And then you can see cos function signatures in command line by jedi-vim.

[tweekmonster]

I checked my assumption about call_signatures of jedi-vim. And yes it is it that cause freeze of neovim. Maybe because it isn't async or i don't know.

What's happening is Jedi stalling on numpy because it needs to resolve the object before it can display the call signatures. Neovim's async becomes irrelevant since jedi-vim isn't designed to work asynchronously.

[monkoose]

@tweekmonster @blueyed Do you know does @davidhalter have plans to rewrite jedi-vim with some async features? Should i request it in his repo?

[blueyed]

@monkoose
Planned for Jedi itself: davidhalter/jedi#385.
As for jedi-vim: it could by async using jobs/timers I suppose.

What about adding a call signature feature to deoplete/deoplete-jedi btw? :)

[monkoose]

@blueyed I don't know what @zchee and @Shougo think about it. Myself i like linux way. So completion engine/helper/plugin should always stay only as completion plugin. Or may be i wrong and completion and refactoring staff is the same thing(as we use jedi for both) and its easier to have and maintain in the same place. So i dunno.
But for sure it would be cool to get rid of jedi-vim and have async plugin(deoplete-jedi itself?) for features as GoTo, Rename, Show Usage and call signatures(and do not forget about Show Documentation). i think it's then perfect time for me to learn more viml and jedi library.

[ghost]

Hey, what's the status of this issue? For me, it seems to be happening, also with numpy. Other things that use jedi such as jonathanslenders/ptpython complete everything with near-zero delay. I don't know exactly what's happening here.

[tweekmonster]

@EshanSingh What you linked is a REPL, which gets the benefit of actually executing code at your request. Live objects can be inspected a lot more easily since you already loaded things. What we're doing is inspecting the scripts without executing anything and Jedi has to emulate some execution to get the completions.

In a REPL you're controlling the execution step-by-step. So, you wouldn't be surprised if what you did caused a file to be written or deleted. In source editing, you wouldn't want code to execute before you consider it usable.

As for the status, I haven't kept up with Jedi's development. Besides what I posted above, I'm not sure where this stands.

[ghost]

@tweekmonster That makes sense, thanks a lot.

[ocehugo]

any way this days of avoiding the kludge of typing,erasing and waiting to make deoplete show the right candidates?

[blueyed]

@ocehugo
Are you hitting this issue exactly?
Otherwise, please create a new one and invest some time to triage, debug and fix it.

[ocehugo]

@blueyed,
yes.

I cant invest time on that now, but I can say that I'm still hitting this issue with numpy for example, where candidates are completely wrong and/or not loaded at all (np.zeros,np.sin,np.tan are not pop ed up and other candidates not related to methods of the package are shown) . All packages on HEAD (neovim,jedi,bla bla bla).

[blueyed]

@ocehugo
Maybe updating Jedi helps?! #105
Apart from that: it's required for issues to being reported and debugged.

I can see that this is an issue for people using pandas & co, but then you should also have experience with coding, optimization, data structures etc., so please consider digging into the source of the software you are using.

[ocehugo]

@blueyed
I tried with updated jedi and the error persist. The fix from @monkoose did not work anymore so useless for me/Need to travel back to the previous version.

Again , the issue is super easy to reproduce:

import numpy as np

#trying to complete np.zeros
x = np.zer #press tab or activate the completion engine

Also, doesn't matter if it's like:

import numpy

or

import numpy as np

candidates are wrong in all of them/completely missing.

This is not only with numpy, but with mostly all popular python packages (matplotlib,scipy,astropy,etc). I could say all packages cause none of them work here. Completions for imports are also a problem but i'm digressing...

[zchee]

@ocehugo Hi,
If you not try it, could you try following commands?

ls -la $XDG_CACHE_HOME/deoplete/jedi/your_python_version
# if there is json cache file...
rm -rm $XDG_CACHE_HOME/deoplete/jedi/your_python_version

that's clear before server cache data.
If you do not try it, it seems worth it.

[ocehugo]

Aha! @zchee worked! thx for that

[zchee]

@ocehugo 🎉
@blueyed Thanks follow up! 🙇

[theFool32]

Same issue here and no help to clear old cache, it seems to output same cache as old.
While numpy can get a part of candidates, cv2(opencv) get nothing.
I check cv2.json and find

{"version": 11, "cache_key": ["cv2", "package"], "time": 1488198528.8552847, "modules": {}, "completions": []}

both python2 and python3 on ArchLinux with jedi v0.10.0-1 failed.

[zhou13]

I can confirm the bug. It still fails with cache cleared.

[zhou13]

Update Jedi to 0.10.2 fixes this.

[monkoose]

Yep. Seems working for me too. So this issue can be closed i think.
But it is still slow. You need wait alot(jedi problem itself with large libraries i suppose) and sometimes delete entered letters of a word that you want to complete. But then when deoplete-jedi cached module all works fast and fine.
But then #106 is maybe needed, I don't know will you need to clear the cache in the future with some jedi or deoplete-jedi updates.

[tweekmonster]

@monkoose When #115 is merged, completions will return a lot faster. For modules that take a long time, the completion menu will refresh automatically so you no longer need to backspace to refresh. But it depends on Shougo/deoplete.nvim#467

Video: https://share.esdf.io/PTpzroecdt/

There's still some work that needs to be done with Deoplete's async refresh, though. I think nvim changes the b:changedtick when something in the menu matches the input text which invalidates the refresh timer.

I don't know will you need to clear the cache in the future with some jedi or deoplete-jedi updates.

That's a problem on our end. Any changes to the cache format or Jedi submodule requires a cache version bump in the code to invalidate the current cache.