[WIP/RFC] Improve selection of environments #836

blueyed · 2018-06-23T09:38:34Z

TODO:

keep g:jedi#force_py_version (s:init_python: handle g:jedi#force_py_version again #834 (comment))

blueyed · 2018-06-23T09:45:49Z

@davidhalter
This is WIP, but quite usable already.
You can use let g:jedi#environment_paths = [$HOME . '/.pyenv/versions]' to get all pyenv version in the completion for :JediUseEnvironment.. :) (a bit slow though!)

Using :JediUseEnvironment! … sets it for the current buffer.

davidhalter · 2018-06-23T12:54:00Z

BTW: I actually wanted to list the virtualenvs and system pythons in a buffer anyway so a user could choose. I think improved environment selection will be a great feature.

blueyed · 2018-06-23T18:16:04Z

autoload/jedi.vim

@@ -233,6 +242,14 @@ call jedi#init_python()  " Might throw an error.
 " ------------------------------------------------------------------------
 " functions that call python code
 " ------------------------------------------------------------------------
+function! jedi#use_environment(...) abort


Rename to set_environment?

blueyed · 2018-06-23T18:16:33Z

autoload/jedi.vim

+  PythonJedi jedi_vim.set_environment(*vim.eval('a:000'))
+endfunction
+
+function! jedi#use_environment_for_buffer(env) abort


Same as above.
There should also be setters (and getters) for tab- and window-local settings.

blueyed · 2018-06-23T18:16:55Z

autoload/jedi.vim

+" files?!).
+function! jedi#complete_environments(argl, cmdl, pos) abort
+    PythonJedi jedi_vim.complete_environments()
+EOF


Remove EOF.

blueyed · 2018-06-23T18:17:18Z

plugin/jedi.vim

@@ -48,6 +48,21 @@ endif
 " Pyimport command
 command! -nargs=1 -complete=custom,jedi#py_import_completions Pyimport :call jedi#py_import(<q-args>)

+" Transfer bang into current buffer number.
+" TODO: use window instead!?


See above - should support global, tab, window and buffer.

blueyed · 2018-06-23T18:17:55Z

pythonx/jedi_vim.py

@@ -158,42 +158,84 @@ def wrapper(*args, **kwargs):
    return func_receiver


+# XXX: might use lru cache, given that you can configure it per buffer?!


Probably all the caching (and invalidation) should be done by Jedi.

blueyed · 2018-06-23T18:18:30Z

pythonx/jedi_vim.py

 current_environment = (None, None)


+def set_environment(executable=None, bufnr=None):


Needs support for tab, window, and bufnr.
Could be scope.

While you can do that, I don't think this is what most users need/want. The scope most users want to use is probably global, but I'm not against it - I'm just saying that you're probably doing more work than necessary if you use a bufnr scope. :)

Yeah, it is not really necessary, but a good example for getting the scoping right from the beginning.
More important would be a tab-local setting then I guess.
This is kind of useful in an IDE-like environment where you would work on different projects/venvs in parallel - but yes, it is a bit constructed.

blueyed · 2018-06-23T18:18:52Z

pythonx/jedi_vim.py

+        vim_command("let g:jedi#use_environment = %r" % executable)
+    current_environment = (executable, environment)
+
+
 def get_environment(use_cache=True):


Add scope here then, too. But it would default to preferring bufnr over window over tab over globals.

blueyed · 2018-06-23T18:20:33Z

pythonx/jedi_vim.py

+    vim.command("return '%s'" % '\n'.join(
+        ('%s (*)' if env.executable == current_executable else '%s') % (
+            env.executable
+        ) for env in envs))


TODO: display more information (after the name, similar to the currently selected one), e.g. the version.

blueyed · 2018-07-15T15:41:14Z

Updated to use davidhalter/jedi#1176.

blueyed · 2018-07-15T21:57:51Z

@davidhalter
Would appreciate some feedback on this early stage for the next iteration.

davidhalter · 2018-07-19T07:16:37Z

pythonx/jedi_vim.py

+
+
+def complete_environments():
+    envs = get_known_environments()


Especially since the latest changes we need to cache those, otherwise this is going to be very very slow.

davidhalter · 2018-07-19T07:18:41Z

I like the general direction of this PR. For me personally I have always envisioned a "list environments" feature where you could then choose which environment you would want to use. However this is a feature that might also be interesting for some people!

This is taken out of davidhalter#836.

Improve JediDebugInfo for envs This is taken out of #836.

codecov-io · 2018-08-06T18:30:01Z

Codecov Report

Merging #836 into master will decrease coverage by 1.56%.
The diff coverage is 17.85%.

@@            Coverage Diff             @@
##           master     #836      +/-   ##
==========================================
- Coverage   47.68%   46.11%   -1.57%     
==========================================
  Files           3        3              
  Lines         799      798       -1     
  Branches      164      165       +1     
==========================================
- Hits          381      368      -13     
- Misses        384      396      +12     
  Partials       34       34

Flag	Coverage Δ
#py27	`45.23% <17.85%> (-0.56%)`	⬇️
#py37	`44.36% <17.85%> (-0.54%)`	⬇️
#py38	`?`

Impacted Files	Coverage Δ
pythonx/jedi_vim.py	`47.33% <17.85%> (-1.66%)`	⬇️
test/test_integration.py	`86.66% <0%> (-0.57%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2abec2...54f30f6. Read the comment docs.

Fenkiou · 2018-10-06T13:38:50Z

pythonx/jedi_vim.py

+            # XXX: create_environment is not deterministic
+            #      (https://github.com/davidhalter/jedi/issues/1155)
+            # jedi.create_environment(executable, safe=False)
+            environment = jedi.api.environment.Environment(executable)


Just for information, I had to comment this line and use environment = jedi.create_environment(executable, safe=False) instead to get it working.

I'm using python3.7 with pyenv.

@Fenkiou
Thanks for trying this.
Did you get an error before? ("'executable=%s is not supported: %s.'")
If so, what was the message?

Nope the message was this one:

Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/alex/.vim/repos/github.com/davidhalter/jedi-vim/pythonx/jedi_vim.py", line 177, in set_environment environment = jedi.api.environment.Environment(executable=executable) TypeError: __init__() missing 1 required positional argument: 'path'

But in fact, I didn't run git submodule update when checking out your branch, everything is working correctly without any change! Sorry for the noise.

petobens · 2018-12-02T18:40:19Z

Hi @blueyed. I've recently started using pipenv to manage my python applications and was wondering if there is an easy way (either with this PR or with the current jedi/vim-jedi version) to have jedi automatically use a pipenv environment if one is available in the current directory. I was thinking of something along the lines:

let pipenv_venv_path = system('pipenv --venv')
if v:shell_error == 0
    let jedi_venv = trim(pipenv_venv_path)
else
   let jedi_venv = use_system_python
endif

that would triggered every time python buffer is read.

Do you think this is possible? If so, can you give a few pointers on how to implement it? Thanks!!

blueyed · 2018-12-02T19:06:20Z

system('pipenv --venv')

pipenv is slow in general, and system() should be avoided in general.
I suggest having your $PATH setup properly (as with pipenv --shell (IIRC) or through some other method).

blueyed · 2018-12-08T18:24:51Z

Rebased, without the updates to jedi/parso which I assume have landed already.

petobens · 2018-12-12T19:53:30Z

Anyway, have you tried this PR after all, which is supposed to make this simpler? (but also needs updating / more work)

I gave it a shot and it seems to work. My main problem (with your changes and with vim and venvs in general) is how to disable a virtual env:
i) Start vim/nvim
ii) Use JediUseEnvironment or something else to activate a virtual env.
iii) Run JediDebugInfo correctly shows paths to the activate venv
iv) Use unlet $VIRTUAL_ENV or vim-virtualev vim plugin to deactivate the virtualenv
v) Run JediDebugInfo again. -> Still reports path to the activated venv.

Can you reproduce that? Is there a workaround? Thanks! And sorry for the late reply.

aliceh75 · 2019-08-14T08:51:08Z

Just tested this a little bit, seems to work fine (the patch didn't apply right away, but simple enough to fix). Thanks 😄 🎉 🥇

The autocompletion for JediUseEnvironment only looks in the path (which is fair enough :) ). One suggestion would be to allow this to be configured - ideally with some sort of wildcard system. I'm current using mkvirtualenv so my virtual environments are under ~/.virtualenvs/<project name>. If I used Pipenv they would be in ~/.local/share/virtualenvs/<project name> (I think). It's also not uncommon to have virtualenv within project folder (I've encountered both venv and .ve folder).

For anyone reading this - it's only the auto-completion I'm talking about. You can provide the correct path to JediUseEnvironment and it works as expected.

It's great as it is though, I could easily enough provide my own wrapper around JediUseEnvironment so I wouldn't let this suggestion delay a possible merge of this feature 😄

blueyed · 2019-09-23T15:16:21Z

@aliceh75
Thanks for review. I've rebased it now.

As for completing from certain paths there is g:jedi#environment_paths already, which (IIRC) should do what you want.
E.g. :let g:jedi#environment_paths = [expand('~/.pyenv/versions')] works for me to complete installed pyenv versions.

blueyed · 2019-09-23T15:22:21Z

@petobens

My main problem (with your changes and with vim and venvs in general) is how to disable a virtual env:
i) Start vim/nvim
ii) Use JediUseEnvironment or something else to activate a virtual env.
iii) Run JediDebugInfo correctly shows paths to the activate venv
iv) Use unlet $VIRTUAL_ENV or vim-virtualev vim plugin to deactivate the virtualenv
v) Run JediDebugInfo again. -> Still reports path to the activated venv.

Unletting $VIRTUAL_ENV will not have an impact on JediDebugInfo / the
env being used.

Mixing it with vim-virtualenv requires you to know what you're doing, but
in general should not have an impact on jedi-vim in this regard, apart from
when it would define $VIRTUAL_ENV that Jedi might pick up by default.

Can you reproduce that? Is there a workaround?

It should work OK when only sticking to :JediUseEnvironment, and using it
also after / instead of unlet $VIRTUAL_ENV/vim-virtualenv.

Maybe jedi-vim could be made smarter about handling $VIRTUAL_ENV (defined in Vim).
Currently only Jedi handles $VIRTUAL_ENV, and you might run into cache issues
there If I remember correctly - check the code at https://github.com/davidhalter/jedi/blob/master/jedi/api/environment.py.

davidhalter

I feel like the API here is definitely going into the right direction.

I've always had a feeling that we should add something like a quickfix window where you could select the "new" environment. get_known_environments will make that possible.

I feel like completing the environments might work, but it's a bit risky since it takes a lot of time. Not that I'm saying it's a bad idea to use completions, it's just slow (as I noted in a comment).

Otherwise good work. I definitely like where this is going!

davidhalter · 2019-10-01T20:26:38Z

pythonx/jedi_vim.py

    return environment


 def get_known_environments():
    """Get known Jedi environments."""
-    envs = list(jedi.api.environment.find_virtualenvs())
+    paths = vim_eval("g:jedi#environment_paths")


If paths is empty, it should probably be the current directory. Or does that not make sense?

davidhalter · 2019-10-01T20:31:58Z

@petobens Just FYI we won't support a combination of Jedi/vim-virtualenv. That will break in a lot of ways for sure. There's so many potential issues there. Also it will really not be needed once this stuff is finished.

syphar · 2019-12-03T06:22:09Z

@blueyed having problems with jedi and virtualenvs too, just discovered this PR here.
Is this testable? I don't know jedi-code good enough to review that, but I could test it under my conditions :)

(in my case, completion with packages virtualenv works perfectly, but goto-definition not, $VIRTUAL_ENV is set correctly)

blueyed · 2019-12-03T06:38:06Z

@syphar sure, give it a try. (I've just rebased it on master)

petobens · 2020-01-17T20:44:35Z

I'm been using this with my poetry/pipenv plugin petobens/poet-v#2 and it seems to work great so far. Thank you!!

blueyed · 2020-01-17T21:07:35Z

@petobens
Thanks for the feedback. I also think this should be ready, but IIRC needs a last round of polishing I guess.

petobens · 2020-01-18T12:05:18Z

One minor question: is there a way to disable enviroment usage and point jedi to use the global python executable? The current way I'm doing it is :execute 'JediUseEnvironment ' . py3eval('sys.executable') but I don't know if a new more direct way could be added (something like JediDisableEnvironment)? Thanks

davidhalter · 2020-01-19T11:36:00Z

@petobens This is already the default, you don't have to do this.

Default environment selection changed quite a bit over the last year. At the moment it's pretty much what you are already doing.

petobens · 2020-01-19T14:05:33Z

Ok. Thanks

petobens · 2020-03-08T12:13:51Z

@blueyed if this is not moving forward yet can you rebase on latest master? Thanks!!

petobens · 2020-03-08T13:04:49Z

Thank you!

petobens · 2020-07-20T16:24:36Z

@blueyed Sorry for asking this again but can this branch be rebased on latest master? Also any idea/plans for this to be actually merged (i've been using it for a while now without any issues AFAICT)?

petobens · 2020-07-20T20:24:03Z

Thanks!

davidhalter · 2021-01-02T11:46:06Z

@blueyed I feel like this change is obsolete with my latest changes. I was heavily inspired by your changes here, but ultimately decided no to reuse the code for various reasons. I would therefore probably close it or do you think that there is still stuff in here that needs to be kept?

blueyed mentioned this pull request Jun 23, 2018

API: Environment: create_environment is not deterministic davidhalter/jedi#1155

Closed

blueyed mentioned this pull request Jun 23, 2018

Environment: improve/fix getting sys.prefix davidhalter/jedi#1108

Closed

blueyed mentioned this pull request Jun 23, 2018

s:init_python: handle g:jedi#force_py_version again #834

Merged

blueyed commented Jun 23, 2018

View reviewed changes

blueyed mentioned this pull request Jun 27, 2018

"Error when loading source jedi: Failed to start server" when using virtualenv at Windows deoplete-plugins/deoplete-jedi#170

Closed

blueyed force-pushed the envs branch from 6d17ee9 to 4668c37 Compare July 4, 2018 20:30

blueyed mentioned this pull request Jul 6, 2018

[RFC] Upgrade Jedi deoplete-plugins/deoplete-jedi#179

Merged

blueyed force-pushed the envs branch from 4668c37 to 487b8cc Compare July 10, 2018 10:53

blueyed mentioned this pull request Jul 15, 2018

[WIP/RFC] Only use Jedi, without worker/client/server/cache deoplete-plugins/deoplete-jedi#184

Closed

4 tasks

blueyed force-pushed the envs branch from 487b8cc to 33dbfca Compare July 15, 2018 15:40

blueyed changed the title ~~[WIP] Improve selection of environments~~ [WIP/RFC] Improve selection of environments Jul 15, 2018

davidhalter reviewed Jul 19, 2018

View reviewed changes

davidhalter mentioned this pull request Jul 23, 2018

Auto Completion not working with pipenv on msys2 #853

Closed

blueyed force-pushed the envs branch from cda13a5 to 9f6e9de Compare July 24, 2018 15:43

blueyed added a commit to blueyed/jedi-vim that referenced this pull request Jul 27, 2018

Improve JediDebugInfo for envs

aebbafe

This is taken out of davidhalter#836.

This was referenced Jul 27, 2018

Improve JediDebugInfo for envs #858

Merged

my jedi-vim can't show package from python3 #857

Closed

davidhalter pushed a commit that referenced this pull request Jul 27, 2018

Improve JediDebugInfo for envs (#858)

4c430ed

Improve JediDebugInfo for envs This is taken out of #836.

blueyed force-pushed the envs branch from 9f6e9de to 38fc343 Compare July 28, 2018 03:43

blueyed mentioned this pull request Aug 6, 2018

jedi-vim is not aware of pyenv path #869

Open

blueyed force-pushed the envs branch from 38fc343 to c06c8ef Compare August 6, 2018 18:28

Fenkiou reviewed Oct 6, 2018

View reviewed changes

blueyed mentioned this pull request Apr 10, 2019

add g:jedi#virtualenv_path to specify virtualenv #926

Closed

shuntaka9576 mentioned this pull request Apr 21, 2019

Python環境周り shuntaka9576/dotfiles#22

Closed

4 tasks

blueyed mentioned this pull request Aug 11, 2019

jedi-vim does not actually use virtualenv #685

Closed

blueyed force-pushed the envs branch from 584dfd9 to aa68d0e Compare September 23, 2019 15:09

davidhalter reviewed Oct 1, 2019

View reviewed changes

blueyed force-pushed the envs branch from aa68d0e to ca24593 Compare October 16, 2019 20:55

blueyed force-pushed the envs branch from ca24593 to 945b595 Compare December 3, 2019 06:37

blueyed force-pushed the envs branch from 945b595 to 54f30f6 Compare January 17, 2020 09:19

blueyed force-pushed the envs branch from 54f30f6 to 3402bd8 Compare March 8, 2020 12:47

WIP: improve selection of environments

868e922

blueyed force-pushed the envs branch from 3402bd8 to 868e922 Compare July 20, 2020 18:30

davidhalter mentioned this pull request Aug 1, 2020

Use the new Jedi API #1025

Merged

		@@ -158,42 +158,84 @@ def wrapper(args, *kwargs):
		return func_receiver


		# XXX: might use lru cache, given that you can configure it per buffer?!

		current_environment = (None, None)


		def set_environment(executable=None, bufnr=None):

[WIP/RFC] Improve selection of environments #836

Are you sure you want to change the base?

[WIP/RFC] Improve selection of environments #836

Conversation

blueyed commented Jun 23, 2018 • edited Loading

blueyed commented Jun 23, 2018

davidhalter commented Jun 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueyed commented Jul 15, 2018

blueyed commented Jul 15, 2018

Choose a reason for hiding this comment

davidhalter commented Jul 19, 2018

codecov-io commented Aug 6, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petobens commented Dec 2, 2018

blueyed commented Dec 2, 2018

blueyed commented Dec 8, 2018

petobens commented Dec 12, 2018 • edited Loading

aliceh75 commented Aug 14, 2019

blueyed commented Sep 23, 2019

blueyed commented Sep 23, 2019

davidhalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidhalter commented Oct 1, 2019

syphar commented Dec 3, 2019

blueyed commented Dec 3, 2019

petobens commented Jan 17, 2020

blueyed commented Jan 17, 2020

petobens commented Jan 18, 2020

davidhalter commented Jan 19, 2020

petobens commented Jan 19, 2020

petobens commented Mar 8, 2020

petobens commented Mar 8, 2020

petobens commented Jul 20, 2020

petobens commented Jul 20, 2020

davidhalter commented Jan 2, 2021

blueyed commented Jun 23, 2018 •

edited

Loading

codecov-io commented Aug 6, 2018 •

edited

Loading

petobens commented Dec 12, 2018 •

edited

Loading