install virtualenv in project directory

[marcosfede]

Hi, I wonder if its possible to install the virtualenv on the projects directory instead of installing it in .pyenv/versions .
there are many tools that assume you have your venvs on the same directory like vscode.

Is there any flag I can add to pyenv virtualenv to do this?

[yyuu]

You can just create virtualenv in any location. To use it from pyenv, create symlink to the environment in ~/.pyenv/versions

[vc5]

In VS Code,you can just use ctrl +p then type python:select wordspace interpreter

[cirosantilli]

I wish there would be a more "managed" way of doing this, rather than forcing users to make symlinks.

Does it work if you just install pip install virtualenv from inside pyenv?

[grayaii]

Ugh. We have a jenkins machine that runs multiple jobs at the same time on the same machine, each creating/destroying a same-named virtualenv. This stomps on each other, and thus causes build failures. It would be nice if pyenv created the virtualenv local to the jenkins workspace, so that they are self contained within the workspace of the job. Looking at this post, I THINK you can use "virtualenv" (not pyenv virtualenv)... https://stackoverflow.com/questions/30407446/pyenv-choose-virtualenv-directory

[russellvt]

Yes @grayaii, you can use just straight virtualenv, but then you're going to get in to another "chicken and egg" problem, and potentially induce confusion with "which virtualenv" - not to mention, you'd need to increase the ability to reasonably control the Python versions (assuming you're in-need of testing multiple versions of Python).

For me, I just use scripts within Jenkins that add-in unique variables, such as the build name or number to name the pyenv, when it's created. It works "well enough," particularly if you're routinely destroying the environments following the build.

[vedal]

I would also really like to specify install directory when creating a new venv with pyenv.

[grayaii]

@russellvt your suggestion of adding build name and build number made it work "well enough". It still sporadically fails when creating unique pyenv virtualenvs at the same time. We get these weird errors when running pip installs of the same package at the same time, but in different virtualenvs.

IOError: [Errno 2] No such file or directory: '/opt/pyenv/install_dir/versions/2.7.14/lib/python2.7/site-packages/configparser-4.0.2.dist-info/RECORD.pip'

But at least it's not failing every single time.

[tvalentyn]

Thanks for sharing your experience, @grayaii

Ugh. We have a jenkins machine that runs multiple jobs at the same time on the same machine, each creating/destroying a same-named virtualenv.

I was also thinking about using pyenv on Jenkins, and I'd like temporary environments be created under workspace/, which can be cleaned up between the runs. Having the pyenv environment created outside the workspace means that botched runs could leave created environments behind, potentially filling up the space on the Jenkins worker, so we'd have find a way to clean them up periodically.

It still sporadically fails when creating unique pyenv virtualenvs at the same time.

What is the explanation for these failures?

[tvalentyn]

Does anyone see any concern with a following solution to the question raised by OP:

pyenv install -s 3.8.3 # install preferred version if needed
PYENV_VERSION=3.8.3 python -m venv /tmp/your/prefered/dir ; source /tmp/your/prefered/dir/activate

Instead of creating an environment with pyenv virtualenv, we are using the interpreter provided by pyenv, but create an environment in a desired location ourselves via python -m venv.

[native-api]

2all:

What are the use cases where you need this?

---

The only one use case given so far is in CI where you need to clean up envs between bulids -- this can be achieved with pyenv virtualenvs --bare --skip-aliases | xargs -n 1 pyenv virtualenv-delete -f

---

There has also been a request to use a different base directory for Pyenv-Virtualenv than $(pyenv root) if Pyenv is installed globally -- it may be useful for this case, too.

[tvalentyn]

What are the use cases where you need this?

Somewhat orthogonal, but in one use case, I need a developer environment to have one Python interpreter per Python minor version, and I would prefer to make as little assumptions as possible about how interpreters are provided: via apt, compiled from sources, or via pyenv. I would personally prefer pyenv, since it allows a convenient fine-grained control over python patch versions, and does not interfere with system's installation of Python. I install Pythons, make several interpreters globally available via pyenv global 3.5.6 3.6.8 3.7.10 .... Then, I can provide universal commands to create an environment via python3.x -m venv /tmp/your/prefered/dir that would work for me and other developers on the project who decided to choose a different tooling for installing pythons and/or managing environments.

The only one use case given so far is in CI where you need to clean up envs between bulids -- this can be achieved with pyenv virtualenvs --bare --skip-aliases | xargs -n 1 pyenv virtualenv-delete -f

The 'between builds' part may be tricky to time if multiple builds run continuously on the same worker.

There has also been a request to use a different base directory for Pyenv-Virtualenv than $(pyenv root) if Pyenv is installed globally -- it may be useful for this case, too.

Where is this tracked?

edit: s/does interfere/does not interfere

[native-api]

@tvalentyn, thanks for the info!

If a virtualenv is located in some arbitrary place -- and can have a duplicate name, too (in case of multiple concurrent CI builds) -- how do you expect Pyenv to find/distinguish/manage it (e.g. for commands like pyenv virtualenvs)?

I guess the most obvious solution would be to allow to specify a path instead of env's name! (a path in the current dir will have to be prepended with ./, I guess, to distinguish it from a plain name).
pyenv virtualenvs won't be able to find it -- but that doesn't sound like something that concerns you too much.

(Another solution would be to use a different optional envvar for Pyenv-Virtualenv's base path -- but since the base path is supposed to support multiple envs, Pyenv-Virtualenv will have to create the same kind of dir hierarchy there as it does now -- which sounds like not what you want.)

[tvalentyn]

It's possible to avoid duplicate names, my concern was that "clean all envs" in one build could accidentally clean envs created by another build-still-in-progress, and "clean env-X after build-X is done" might not get executed if build somehow failed/got interrupted.

It is possible to figure a way to clean-all-envs-older-than-YYYYMMDD via some naming pattern + modification of the command you provided, and run such script periodically. You could also consider to add functionality in pyenv virtualenv-delete command to clean envs older than a TTL.

Re managing environments created in various directories, you could consider still adding metadata in PYENV_ROOT even if environments were created elsewhere.

[vedal]

@yyuu Any chance we could re-open this? I was thinking of a use-case where one could drop pyenv-virtualenv in favor of poetry (as pyenv-virtualenv updates are sporadic) and have poetry manage venvs instead. Then, one could just point pyenv to the poetry venv with pyenv local path/to/poetry-venv. Poetry already has support for same-folder venv through
poetry config virtualenvs.in-project true
:)

[native-api]

Just use bare virtualenv. If you want to switch to it via Pyenv, symlink to it in $PYENV_ROOT/versions. There's even a 3rd-party plugin to do that.

Or conversely, create the environment with Pyenv-Virtualenv and symlink to it in your project.

---

The entire purpose of Pyenv-Virtualenv is to manage created virtual environments the same as installed Python versions. This implies that the repository of them has to be global (for a specific PYENV_ROOT) and not spread around project directories -- otherwise, how would we compose a full list of them to be able to select from?

[dbersan]

The best solution imo is to use pyenv just to manage the base python versions, and then manage packages locally using virtualenv, to keep everything contained inside the repo.

# Get the python base version you need, activate it, etc.. is usually a good idea to specify the python version to use
pyenv install 3.8
pyenv shell 3.8

# Create environment in local folder
virtualenv .venv -p $(which python)

# Activate environment
source .venv/bin/activate

# Install libraries
pip install numpy boto3 # ...etc

# Export current environment
pip freeze > requirements.txt

vscode will automatically recognize this local environment, or you can press Ctrl+Shift+P and type Python > Select Interpreter, it should be there.

This launch.json configuration allows you to use this python version to start your python script/debug using this environment:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: main.py", # write whatever
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/main.py", # Change start script here
            "console": "integratedTerminal",
            "python": "${workspaceFolder}/.venv/bin/python",
            "justMyCode": true,
            "args": [ ">", "output.txt" ]
        }
    ]
}

Other solutions are to use either Anaconda (easiest imo, but can have some licensing issues), or Poetry.

[native-api]

otherwise, how would we compose a full list of them to be able to select from?

The best solution imo is to use pyenv just to manage the base python versions, and then manage packages locally using virtualenv, to keep everything contained inside the repo.

It should probably be possible to add an ability to Pyenv-Virtualenv to use a locally created environment.

But then you won't be able to select it from an arbitrary place (or will have to specify its full path instead of just name to do so).

[russellvt]

The best solution imo is to use pyenv just to manage the base python versions, and then manage packages locally using virtualenv, to keep everything contained inside the repo.

This is, essentially, what I have done, myself (despite still being a command-line based UN*X junkie); even to the point of "using VIM for an IDE" (ie. no VS here). It also/essentially means you need to maintain a development environment (ie. libraries and headers) for your operating/development environment ... and then use pyenv to maintain the specific Python modules (which may also require changes to your development environment).