Code completion in buildout.cfg based projects
#1325
Comments
|
If it actually freezes completion, I'd be happy to know what causes it. Are there too many modules? Is the freeze permanent? Does the freeze reoccur when completing again? I'd like to fix the bug instead of creating some workaround options that almost nobody will ever use, buildout itself is pretty out of fashion and an ignore option... That would probably just be one user: you ;-). |
|
Hi @davidhalter, thanks for your fast reply. There are actually a lot of modules in the project, but without the It seems like the freeze is not permanent, but runs in the 30s timeout of The freeze affects all completions and is ongoing, probably because these scripts manipulate the I've seen that the duplicates are removed here: https://github.com/davidhalter/jedi/blob/master/jedi/api/project.py#L124 In my case each script has about 300 path insertions and inside the Anyhow, I would indeed completely remove the buildout integration, because I do not see any value in having code completion for auto-generated Python scripts in my project folder – actually I think this project integration (the Django one as well) shouldn't be in the domain of Jedi, but from the user. But when these Just guessing, I didn't go into much detail and I was a bit lost in how to setup Jedi from the command line and make it call that buildout integration code to put a PDB there... |
They don't cause much trouble, they are actually a help for most people. Even the buildout integration. If you're adding 300 paths, you are probably the exception and not the rule. I can see that this probably doesn't perform very well. Jedi will open all eggs and unzip them?! I would need a better understanding of what happens to limit this. Can you use a straight Jedi call and reproduce it this way? Then I would need a |
|
Yes of course I can try to do so, but I wasn't able to write the code in such that this method was called: https://github.com/davidhalter/jedi/blob/master/jedi/api/project.py#L109 In my Emacs I did a very simple test and added a import json
json.lo|Inside the project folder containing the Removing the scripts one by one of the What I tried was to do it like described here: https://jedi.readthedocs.io/en/latest/docs/api.html#module-jedi However, this works pretty fast but does not seem to execute the same code like here: I'm using a Miniconda environment, so no Virtualenv involved here. |
Please try again, Make sure to be in the buildout folder and add a path to |
|
Ok, this are the contents of the script import jedi
print jedi.__version__
source = """
import json
json.lo
"""
script = jedi.Script(source, 3, len("json.lo"), "foo.py")
print script
print script.completions()The $ ls -lah bin
total 576K
drwxr-xr-x 49 rbartl staff 1,6K Mai 20 08:43 .
drwxr-xr-x 27 rbartl staff 864 Mai 21 14:23 ..
-rwxr-xr-x 1 rbartl staff 2,8K Mai 19 08:53 backup
-rwxr-xr-x 1 rbartl staff 313 Mai 19 08:52 buildout
-rwxr-xr-x 1 rbartl staff 1,3K Mai 20 08:43 check-manifest
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-chameleon-lint
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-check-manifest
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-clean-lines
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-csslint
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-dependencychecker
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-find-untranslated
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-flake8
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-importchecker
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-jscs
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-jshint
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-scsslint
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-xmllint
-rwxr-xr-x 1 rbartl staff 3,4K Mai 20 08:43 code-analysis-zptlint
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 createfontdatachunk.py
-rwxr-xr-x 1 rbartl staff 391 Mai 19 08:53 develop
-rwxr-xr-x 1 rbartl staff 23K Mai 20 08:43 enhancer.py
-rwxr-xr-x 1 rbartl staff 24K Mai 20 08:43 explode.py
-rwxr-xr-x 1 rbartl staff 1,4K Mai 20 08:43 flake8
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 gifmaker.py
-rwxr-xr-x 1 rbartl staff 677 Mai 20 08:43 i18ndude
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 instance
-rwxr-xr-x 1 rbartl staff 23K Mai 20 08:43 painter.py
-rwxr-xr-x 1 rbartl staff 24K Mai 20 08:43 pilconvert.py
-rwxr-xr-x 1 rbartl staff 36K Mai 20 08:43 pildriver.py
-rwxr-xr-x 1 rbartl staff 24K Mai 20 08:43 pilfile.py
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 pilfont.py
-rwxr-xr-x 1 rbartl staff 24K Mai 20 08:43 pilprint.py
-rwxr-xr-x 1 rbartl staff 23K Mai 20 08:43 player.py
-rwxr-xr-x 1 rbartl staff 722 Mai 19 21:27 repozo
-rwxr-xr-x 1 rbartl staff 2,8K Mai 19 08:53 restore
-rwxr-xr-x 1 rbartl staff 21K Mai 20 08:43 robot
-rwxr-xr-x 1 rbartl staff 21K Mai 20 08:43 robot-server
-rwxr-xr-x 1 rbartl staff 2,8K Mai 19 08:53 snapshotbackup
-rwxr-xr-x 1 rbartl staff 2,8K Mai 19 08:53 snapshotrestore
-rwxr-xr-x 1 rbartl staff 258 Nov 7 2018 sync_dbs.sh
-rwxr-xr-x 1 rbartl staff 20K Mai 20 08:43 test
-rwxr-xr-x 1 rbartl staff 23K Mai 20 08:43 thresholder.py
-rwxr-xr-x 1 rbartl staff 283 Mai 20 08:43 update_sources
-rwxr-xr-x 1 rbartl staff 2,9K Mai 20 08:43 update_translations
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 viewer.py
-rwxr-xr-x 1 rbartl staff 2,5K Mai 20 08:43 write_code_headers
-rwxr-xr-x 1 rbartl staff 21K Mai 20 08:43 zope_health_watcher
-rwxr-xr-x 1 rbartl staff 22K Mai 20 08:43 zopepyThis is the output of the call
```
0.13.3
>
[, ]
2116014 function calls (1997982 primitive calls) in 3.599 seconds
Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 0.13.3 Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) |
|
Hmm then you have to figure out why takes even longer. A this point we're just speculating, what's going wrong. Once you have the output, please add a |
|
ping. |
|
Hi @davidhalter, here comes the output sorted by internal time:
```
python -m cProfile -s tottime foo.py
0.13.1
>
[, ]
3255690 function calls (3141862 primitive calls) in 4.593 seconds
Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) python -m cProfile -s tottime foo.py Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) Traceback (most recent call last): |
|
Actually I would be happy to have the whole buildout integration dropped from Jedi, because it is as you said, that it is kind of old-school and I do not see any additional value in having the scripts loaded for completion. |
|
I'm also a buildout user and buildout support in jedi also caused me troubles. It took me some time to understand this was the problem. For literally years I have been thinking that jedi does not work because I have a buildout at the root of my home directory. My workaround is when I have a directory structure like: I add an empty |
|
How does your buildout.cfg look like? |
|
My buildout.cfg is very complex, it compiles python interpreter and dependencies from source and runs for several hours, so I tried to narrow it down. What seem to cause problem is that scripts that are more complex than simple assignments to [buildout]
parts = supervisor
[supervisor] # any egg installing scripts should do
recipe = zc.recipe.egg:scripts
initialization =
def repro(a):
a.append('')run buildout with something like: with this, buildout generates a #!/tmp/repro_jedi_buildout/env2/bin/python2.7
import sys
sys.path[0:0] = [
'/tmp/repro_jedi_buildout/eggs/supervisor-4.1.0-py2.7.egg',
]
def repro(a):
a.append('')
import supervisor.supervisord
if __name__ == '__main__':
sys.exit(supervisor.supervisord.main())and then this seem to cause an issue when jedi reads import jedi
jedi.Script('i', path='dummy.py').completions()This is one reproduction, but depending on the code in the script it seems we can have different errors. One egg that cause problems for me is https://github.com/rdiff-backup/rdiff-backup . As we can see in setup.py long scripts like this rdiff-backup-statistics are added in bin. ( buildout will generate a script with the a sys.path addition header then original script content ) It's a bit complex to reproduce in buildout ( the egg is not on pypi and it needs rsync development packages installed to compile extension ... ), maybe something like: [buildout]
parts = rdiff-backup
# before running buildout, run:
# git clone https://github.com/rdiff-backup/rdiff-backup/
develop = rdiff-backup
[rdiff-backup]
recipe = zc.recipe.egg:scriptsbut we can simulate without actually installing the egg, just create a import sys; sys.path[0:0] = [ '/tmp/', ]Maybe what this show is that parsing scripts in bin can fail and |
|
I limited the gathering of paths to 10. Since @ramonski had about 300 paths in there, I think this should be solved now. About potential tracebacks and other bugs: Please open new issues, I'm really happy to have them and fix them. But the ones you posted are so old that they probably were fixed already (so much has changed).
That is definitely a bad idea ;-). Never use exceptions if you don't have to. Catch them at the "edge" where something actually happens and not just randomly somewhere. If you catch Exception or other generalized exceptions, even specialized ones, you're just masking all exceptions of that type and that is very very annoying to debug. |
|
Hi @davidhalter Thanks for coming back to this issue and being persistent on that. However, breaking out of the loop after Imagine you get for one module a completion and for the other not. Might cause headaches as well... I would rather remove all the project integration approaches from the core codebase and consider a plugin based approach for people to integrate their own project structures. I will probably continue just returning no additional paths to keep code completion fast, because I need none of the scripts to be considered for completion: def discover_buildout_paths(inference_state, script_path):
# XXX: ramonski
return set()
buildout_script_paths = set()
for buildout_script_path in _get_buildout_script_paths(script_path):
for path in _get_paths_from_buildout_script(inference_state, buildout_script_path):
buildout_script_paths.add(path)
return buildout_script_pathsBest Regards, Ramon |
|
I found some "maybe off topic" information: There was recently in vscode-python issue tracker a feature request for implementing this thing that jedi is doing, adding eggs installed by buildout in python path for completions ( microsoft/vscode-python#6992 ). I don't know why this was raised there since jedi is already supposed to do it. Maybe because vscode can use either jedi or microsoft's python-language-server so maybe the reporter there was not using jedi. I don't know the exact relationship between jedi and vscode, so maybe my comment here is off topic, in that case, I'm sorry. Also, some plone users seem to be using https://github.com/nazrulworld/collective.recipe.vscode , a buildout recipe that generates a vscode config to set python path to use for completions. I don't know why jedi's built in support of buildout was not enough in their case. There's a discussion on plone forums at https://community.plone.org/t/vs-code-python-buildout-support/8959 I posted there a few days ago and linked to our issue here so that people can provide feedback if they have. |
This is pretty much a typical issue for Jedi. Jedi has to make sacrifices at a lot of points to be "fast". In a lot of ways Jedi doesn't find everything - not because it couldn't - it doesn't want to. There are a lot of artificial limits like the 10 loaded modules. Thanks for the research! The problem is a bit that some people might be really happy with the buildout support and they just never get to see this discussion. So I'm a bit hesitant in just removing the feature. I'm pretty open to removing it once Jedi has So if you think that removing this feature at that point really is a benefit for you, I might reconsider.
AFAIK Jedi is still the de-facto standard in VSCode. It is used in about 80% of their downloads, while the language server is used in the rest. However they aim to replace Jedi (they have been trying for years). So they focus on debugging for the language server (IMO unfortunately). I have however met with the people that work on the extension and they seem pretty cool :). |
Hi all,
I encountered some strange behaviour in my
buildout.cfgbased Plone projects, which brought me here.I'm using Spacemacs as my editor with anaconda-mode and company-anaconda for code completion.
Everything works fine outside of my zc.buildout based projects and I get code completion really fast for my Python code.
However, inside these project folders the process seem to hang and now I found out, that it is because of the scripts inside the
binfolder, which get automatically added in thesys.pathby this code:https://github.com/davidhalter/jedi/blob/master/jedi/evaluate/sys_path.py#L179
Most of these scripts modify the
sys.pathas well and look like this:Having all these scripts inside, seem to block the process and no further completion is possible.
Renaming the
binfolder to_binsolves all issues and I get the normal behaviour as outside of the project folder.I hadn't the chances yet to look into detail yet, but it doesn't seem that this can be turned off by a configuration option, so my question would be, if someone else encountered this issue as well and if it would make sense to make this behaviour optional. Maybe a
.ignoremarker inside a folder to skip traversing?Thanks
The text was updated successfully, but these errors were encountered: