virtualenvs.prefer-active-python uses poetry's Python venv, not the active pyenv virtualenv

[paultiplady]

• Poetry version: 1.5.1
• Python version: 3.10.10
• OS version and name: MacOS 11.7 (20G817)
• pyproject.toml: https://gist.github.com/paultiplady/bceab0f836e38724d0a4366fe36b7033

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The (experimental) virtualenvs.prefer-active-python flag doesn't seem to be working for me.

I set up a local pyenv venv with:

pyenv virtualenv 3.10.10 pet
pyenv local pet

And this venv is activated automatically in my shell. However Poetry isn't picking it up; if I say

> poetry env info -vvv
Loading configuration file /Users/paul/Library/Application Support/pypoetry/config.toml

Virtualenv
Python:         3.10.10
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   darwin
OS:         posix
Python:     3.10.10
Path:       /Users/paul/.pyenv/versions/3.10.10
Executable: /Users/paul/.pyenv/versions/3.10.10/bin/python3.10

Reading the code, I don't get how this flag is supposed to work; tracing the local code execution we correctly run:

poetry/src/poetry/utils/env/env_manager.py

Lines 237 to 242 in fedff6d

	prefer_active_python = self._poetry.config.get(
	"virtualenvs.prefer-active-python"
	)
	python_minor = self.get_python_version(
	precision=2, prefer_active_python=prefer_active_python, io=self._io
	).to_string()

And determine the minor version of the local python to be 3.10. Then we run

poetry/src/poetry/utils/env/env_manager.py

Lines 274 to 275 in fedff6d

	if not create_venv:
	return self.get_system_env()

And fetch the system python (which is /Users/paul/.pyenv/versions/3.10.10/bin/python3.10 in this case, the pyenv system version that I installed poetry with).

I can't see how this code is supposed to work; it's just extracting python_minor from the pyenv interpreter, and maybe looking that up in the poetry virtualenv repo? But of course that won't work for venvs managed by pyenv. It seems to me that EnvManager.get() should just say:

        if prefer_active_python:
            executable = self._detect_active_python()
            bin_dir = executable.parent.parent
            path = bin_dir.parent
            return VirtualEnv(path, Path(executable.name))

Hacking that change in gives me this output:

Virtualenv
Python:         3.10.10
Implementation: CPython
Path:           /Users/paul/.pyenv/versions/pet
Executable:     /Users/paul/.pyenv/versions/pet/bin/python
Valid:          True

System
Platform:   darwin
OS:         posix
Python:     3.10.10
Path:       /Users/paul/.pyenv/versions/3.10.10
Executable: /Users/paul/.pyenv/versions/3.10.10/bin/python3.10

This seems to be the right interpreter. Does this seem like the right path?

[finswimmer]

Hey @paultiplady,

I'm not sure I can follow you. Maybe you misunderstand what virtualenvs.prefer-active-python is doing? It's saying: "If Poetry creates a venv it should pick up the current activated python interpreter to create this venv".

So can you please point out what you've expected to see and what happens instead? This makes it easier to understand whether there is an issue or not.

fin swimmer

[paultiplady]

Thanks, it looks like I misunderstood what's going on with this flag then. I read #4852 and perhaps didn't read too much into "the pyenv way", assuming that meant how I was familiar with folks using Pyenv.

The workflow that I used with Poetry from when it first came out until when it broke some time in the last year was to use pyenv with the pyenv-virtualenv plugin to manage the venv. (The pyenv incantations I shared at the top of my original post). This means when I cd into the project directory, my venv is automatically activated, and I don't need to say poetry run <python-entrypoint> or poetry shell to use in the venv, I just have my venv activated automatically. (I'd previously just install poetry inside the per-project venv, and that worked fine up until some point between 2022 and now.)

In my Dockerfiles I was using virtualenvs.create=False but I don't think I was explicitly setting that in the dev env config. (I do have that flag set now on my new "official installer" install.)

It feels like the code that currently exists is pretty close to working as I expect it with virtualenvs.prefer-active-python=True & virtualenvs.create=False, and regardless of whether that's intended to be supported, I suspect there is still a bug; with prefer-active-python=True shouldn't poetry env info show the currently-active python (in this case a virtualenv), not the system python that was used to install poetry?

To be clear, in the diags I provided above, this command:

> poetry env info -vvv
Loading configuration file /Users/paul/Library/Application Support/pypoetry/config.toml

Virtualenv
Python:         3.10.10
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   darwin
OS:         posix
Python:     3.10.10
Path:       /Users/paul/.pyenv/versions/3.10.10
Executable: /Users/paul/.pyenv/versions/3.10.10/bin/python3.10

Is being run inside the venv /Users/paul/.pyenv/versions/pet/bin/python. I want poetry to use the active python (in this case a venv), not infer the system python.

For what it's worth, I retested this using a non-venv pyenv version and the instructions in https://python-poetry.org/docs/managing-environments/, and that's broken in the same way; I think that is the expected usecase from the docs:

❯ pyenv local 3.9.9

❯ poetry env info

Virtualenv
Python:         3.10.10
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   darwin
OS:         posix
Python:     3.10.10
Path:       /Users/paul/.pyenv/versions/3.10.10
Executable: /Users/paul/.pyenv/versions/3.10.10/bin/python3.10

In this case we should be pulling the 3.9 pyenv version as the active python version, and instead it's pulling 3.10.10 (the pyenv system version, used to install poetry).

My fix resolves this case too (though I see now that we can't hardcode the executable to a VirtualEnv, since it might be a non-venv interpreter. Maybe it can be a GenericEnv?)

[jonvet]

I also misunderstood how virtualenvs.prefer-active-python = true should work. however I just wasn't able to use pyenv & poetry together: I had the same problem that poetry install would create a new venv, even though I had virtualenvs.prefer-active-python = true and my pyenv venv was active.

with virtualenvs.create = false it would install everything into the system python (a pyenv version in my case). using @paultiplady's fix it works for me

[bogdanpetrea]

Here's the workaround for any sad soul coming here and seeing no new activity:

0. Pre-requisites:

   • You have your poetry project ready.
   • You have your pyenv virtualenv ready and set to be used in your project. (pyenv local <venv-name-here>)
   • Make sure you are in a freshly opened terminal tab, otherwise the pyenv venv might actually be active if you've just created it. (poetry env info should confirm the pyenv venv is not in use)

1. Set these on:
   poetry config virtualenvs.prefer-active-python true (not sure if this actually matters, but I can't be bothered)
   poetry config virtualenvs.create true

2. Run any command that will trigger poetry to create and use it's own venv, if you haven't already.
   poetry install

3. From the previous command or from running poetry env info, find out the poetry's venv name. It should be something like <project>-cpDZ0dC7-py3.12.

4. Delete the ************:
   poetry env remove <project>-cpDZ0dC7-py3.12

5. Confirm your pyenv venv is now in use, by running poetry env info again.

6. Set:
   poetry config virtualenvs.create false

7. Open a new terminal tab and check again to make sure you are not being bamboozled.

[finswimmer]

Seems to work as expected. I also tried to reproduce the output of poetry env info and it shows the correct venv created and activated by pyenv.