Option to have the virtual environment's directory name without hash (id) in it

[jamilraichouni]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Problem

People may have projects that are maintained and used on multiple computers where an according virtual env has been created for the project on every development computer.

Example:

My project is named myproject

The project is maintained/ developed on multiple machines and on each machine poetry install command is executed but the developer doesn't want to store the env in the project at .venv.

Computer 1:

Project is at /Users/user/repos/myproject, virtual env is at /Users/user/repos/.venvs/myproject-ABC-py3.8

Computer 2:

Project is at /Users/user/repos/myproject, virtual env is at /Users/user/repos/.venvs/myproject-DEF-py3.8

One can see, the hashs in the env names differ!

This might be a one man show project on different computers.
In this case project settings controlling paths to dev tools are shared across the different machines.

An example is given via myproject/.vscode/settings.json (Visual Studio Code settings for the project)

A cutout on computer 1 may look like this:

{
    "python.analysis.extraPaths": [
        "/Users/user/.venvs/myproject-ABC-py3.8/lib/python3.8/site-packages"
    ],
    "python.formatting.blackPath": "/Users/user/.venvs/myproject-ABC-py3.8/bin/black",
    "python.linting.flake8Path": "/Users/user/.venvs/myproject-ABC-py3.8/bin/flake8",
    "python.linting.pydocstylePath": "/Users/user/.venvs/myproject-ABC-py3.8/bin/pydocstyle",
    "python.pythonPath": "/Users/user/.venvs/myproject-ABC-py3.8/bin/python",
    "python.sortImports.path": "/Users/user/.venvs/myproject-ABC-py3.8/bin/isort",
    "python.testing.pytestPath": "/Users/user/.venvs/myproject-ABC-py3.8/bin/pytest",
    "restructuredtext.linter.executablePath": "/Users/user/.venvs/myproject-ABC-py3.8/bin/doc8"
}

Now every time I switch to computer 2 and back I always have to manually adapt the synchronised project settings just because of the hash within the env name. That is not feasible and I never had that issue with conda, virtualenvwrapper, venv, pyenv ...

Solution (feature request)

Make possible that the poetry user can customise/ control if there will be a hash in the environment name without being forced to have the env as .venv in the project.

Every reader of this feature request

Click on the thumbs up of this initial post if you like that feature request so that it hopefully gets higher attention.

Thank you!
Jamil

Keywords for search hits:
Environment name, env name, hash, identifier, unique, customise environment name, customise env name, set environment name, set env name

[sinoroc]

Personal point of view:
I would not recommend doing this. If you want to install Python tools in some kind of global fashion (but local to the user), while still taking advantage of the isolation of virtual environments, then you should rather use things like:

• https://pypi.org/project/pipx/
• https://pypi.org/project/dephell/ and its jail feature
• https://pypi.org/project/pickley/
• and others

[jamilraichouni]

Thanks!

But I hope that I'm not the only developer who wants to have control and also the freedom to customise the directory name for the virtual environment (e. g. flag --prefix passed to conda create when using Miniconda/ Anaconda).

If that is not wanted then there should be an option like --no-hash that can be passed to poetry install and all other poetry commands that trigger the creation of an env outside of the project.

This would make it possible that the env name for the example in my initial post here would be
myproject-py3.8 on all computers that are in use to work on the same project.

Having

• myproject-ABC-py3.8 on computer 1,
• myproject-DEF-py3.8 on computer 2,
• myproject-GHI-py3.8 on computer 3

etc. for one and the same project that is just developed on different computers (e. g. at home and also at work) is really not helpful but producing efforts every time the machine is being switched.

I mean the envs on the different computers that all belong to the same project are (except for the dir name) equal.

It is not cool to be forced to have that hash in the middle of an env's dir name. :-)

[sinoroc]

@jamilraichouni

I do not see any reason why you would want to control the name of the virtual environment. The use case you showed, seems like a wrong use case for poetry, there are other tools better suited for this kind of use case, as I already mentioned.

I am sure that hash serves a purpose, is not just here to annoy the users.

If you insist on wanting to control the name of the virtual environment, you probably can create the environment beforehand and then let poetry use it. If I were in your situation I would try to experiment with something like the following (untested, some tweaking will be needed):

path/to/python3.8 -m venv path/to/myproject-py3.8
poetry env use path/to/myproject-py3.8/bin/python
poetry install ...

[jamilraichouni]

Thanks!

I like that proposal but I get:

So it forces me to use the existing env with the hash in its name.

What I tried then is to remove the env before I do the same again.

Hence, first:

and then again:

So I like the idea but it seems to be not possible to work around like that.

[sinoroc]

Ah... I guess you need to activate the virtual environment and use poetry inside it, instead of telling poetry to use that environment then. So maybe try a process like this:

$ path/to/python3.8 -m venv path/to/myproject-py3.8
$ path/to/myproject-py3.8/bin/activate
(myproject-py3.8) $ poetry install
(myproject-py3.8) $ exit

But if it's integration with vscode that you are after, I guess there are better ways to do this.

[jamilraichouni]

Tried that but it doesn't work:

In the end I'd like to have the same and not very special freedom to control the env name completely as it is possible with other well-known environment/ dependency managers.

What poetry offers in general is very helpful and cool. This is why I want to continue using it.

Depending on additional tools just to be able to do something that the others (e. g. conda) offer is no option for me for now. This introduces more "complexity" and dependency/ risk where it should not be needed.

[sinoroc]

I am not following why you say it does not work...

Once you ran poetry install in the activated virtual environment that you created yourself, it should be possible to deactivate the environment and simply call path/to/myproject-py3.8/bin/black from anywhere for example. So that is what you could set in your vscode configuration. Am I missing something?

Also, please use copy paste of your console output, not screenshots.

[jamilraichouni]

Yes, but poetry env info --path and all other poetry commands shall work.

source $(poetry env info --path)/bin/activate is needed for tooling that doesn't know where it is respectively what env must be activated for instance.

[sinoroc]

I am not following. On one hand you tell me you want to have total control over the name of the directory containing the virtual environment and on the other hand you tell me you need $(poetry env info --path) because you don't know where the virtual environment is... Which is it then? It feels like you are overcomplicating things unnecessarily. Is there something I am missing?

Again: nothing forces you to let poetry create the virtual environments. You can create them yourself with whatever names you want, so that the names are known in advance and the names are repeatable from machine to machine.

all other poetry commands shall work

Which ones don't work?

[jamilraichouni]

Sorry, but I expect that people assume positive intentions. I do serious software dev since more than 15 years and know what I do.

To quickly answer the question respectively describe a quite common use case.

There is a project with an according venv. The project has scripts (e. g. .sh, .py etc.) for tooling/ building project artifacts (e. g. a script to build Sphinx docs in a sub dir bin). Such scripts are executed anywhere on any machine where the venv is given at some unknown location (absolute path, not just dir name for venv!).

Now any of the project scripts can run poetry env info for instance to get a pointer to the venv for the project on the current machine to run stuff like sphinx-build, black and god knows what else.

Simple like that. Not overcomplicated and very common in toolsets. Think about variable substitutions in dev tools to keep dev toolsets runnable in a machine-independent manner.

Example outlining that for the case of VS Code users: https://code.visualstudio.com/docs/editor/variables-reference

Yes, the venv names are repeatable from machine to machine with your suggestion. But the needed absolute path to the venv is not.

POETRY_VIRTUALENVS_PATH can differ on different machines. Hence, source $(poetry env info --path)/bin/activate is not just for fun but useful :-)

In any case a work around that stops poetry commands to work properly is not acceptable.
Nevertheless, I thank you for your idea(s)!

[sinoroc]

Another thing you could try is to create the virtual environment at path/to/project/.venv. As far as I know if poetry can find a .venv virtual environment right next to the pyproject.toml file it will use that environment. And if I remember right, there is no need to explicitly call poetry use or activate that environment, and poetry env info --path should deliver the absolute path to that .venv directory. But then you can only use a single version of the Python interpreter (major.minor) at once, i.e. you could not run both Python 3.8 and 3.9 on the same project. Maybe that could work for your use cases.

My bad, you already mentioned, you don't want to use the .venv trick.

[jamilraichouni]

Yes, and there a couple of reasons:

• No way of testing against multiple py versions w/o always having to temporay rename .venv (as you mentioned)
• no clean way of potentially sharing one env with multiple projects (no, this is not always bad, but a use case)
• when using the bash commands find and/ or grep (recursively), linters and potentially other tooling in the project dir, .venv must always be excluded via according flags etc. to avoid running through the .venv dir itself.
• own .gitignore entry needed for .venv/

[sinoroc]

I would need to look the exact code, and figure out why the hash is there exactly. I assume it is to differentiate multiple projects of the same name on the same machine. The hash might be a hash of the full path or something like that (the code seems to be here). In any case I doubt it would be possible to remove that hash, because otherwise I do not think poetry would be able to associate a pyproject.toml to its virtual environments. I do not think poetry is keeping any kind of DB containing the relations project to virtual environments (although I see there is a ~/.cache/pypoetry/virtualenvs/envs.toml, that could maybe used to track those relations).

Somewhat related:

• Manually specifying the venv path #1579
• Create virtual env hash based on lock file instead of parent directory #2927

Maybe this could be solved with an environment variable containing the base path for your virtual environments (value specific to each machine) and some light scripting to figure out the full path to the virtual environment based on 1. the environment variable, 2. the python interpreter version (major, minor), and 3. the name of the project. You could copy/paste some of the poetry code. You would call that script instead of poetry env info --path.

[jamilraichouni]

Yes, thanks, that’s what I also thought of.
But an option (if possible and like feature requested here) would be a better solution.

Cheers,
Jamil

[trianxy]

@jamilraichouni A dirty solution may be to 'just' replace the method generate_env_name in

poetry/poetry/utils/env.py

Line 845 in 8312e3f

def generate_env_name(cls, name, cwd): # type: (str, str) -> str

with

@classmethod
def generate_env_name(self, name, cwd):
    return name

After that change, the virtual environment directory should be /Users/user/repos/.venvs/myproject-py3.8

I initially felt such a feature would be a) messy, b) a source of future problems (due to the loss of a unique mapping from projects to their virtual environment directories) and c) only serve use cases which have better solutions. However, maybe the problems would be fixed if we just prohibit the creation of 2 projects with the same name and no_hash=True (that tag would need to be in the pyproject.toml).

[jamilraichouni]

Thanks, I found a better workaround by using symbolic links.
Hence, currently ls -l in my .venvs dir gives me:

lrwxr-xr-x  1 owner  group    18B 18 Dec 16:24 jar -> jar-AKvBzFsq-py3.8
drwxr-xr-x  8 owner  group   256B 19 Dec 03:21 jar-AKvBzFsq-py3.8
lrwxr-xr-x  1 owner  group    24B 19 Dec 16:55 myproject -> myproject-rDfqSH_K-py3.8
drwxr-xr-x  8 owner  group   256B 18 Dec 16:21 myproject-rDfqSH_K-py3.8
lrwxr-xr-x  1 owner  group    22B 19 Dec 17:36 project -> project-JpSjzBEw-py3.8
drwxr-xr-x  8 owner  group   256B 19 Dec 17:35 project-JpSjzBEw-py3.8

Doing so, all configured paths to tools in virtal envs stay the same when switching the development computer.

[finswimmer]

Hello,

as @trianxy the hash is needed to find all venv's that belong to one project. (There could be more than one). So we cannot implement an option to deactivate the hash in the folder name, when using a central place for all venv managed by poetry.

The symlink trick looks valid for your use case. Also you can try to find out, if it is really necessary the paths to the tools must be absolute paths in vscode config file.

fin swimmer

[jamilraichouni]

Sorry, but the hash is not a must-have but a design decision.

Tell me why it works with system-wide central venvs and other well-known and used virtual env managers?

[finswimmer]

Tell me why it works with system-wide central venvs and other well-known and used virtual env managers?

If you tell me which one you have in mind I can have a look at it. So I can just guess :)

• There is no direct connection between the venv and the project. So you have no chance to ask via a cli "Which venvs belongs to this project?".
• The connection is made via a config file. Then you have no chance to create two projects with the same name in your system, because the name of the venv is not unique.

[sinoroc]

@finswimmer
What is the purpose of the ~/.cache/pypoetry/virtualenvs/envs.toml file? Could it be used to track the mappings between a pyproject.toml and its "virtual environments"?

But honestly, all in all, I am not so sure this feature is required. There are ways to work around it (as already shown).

it works with system-wide central venvs and other well-known and used virtual env managers

I am also curious to see what is meant here, and have a look, see if there is something to learn from those projects.

[jamilraichouni]

Tell me why it works with system-wide central venvs and other well-known and used virtual env managers?

If you tell me which one you have in mind I can have a look at it. So I can just guess :)

• There is no direct connection between the venv and the project. So you have no chance to ask via a cli "Which venvs belongs to this project?".
• The connection is made via a config file. Then you have no chance to create two projects with the same name in your system, because the name of the venv is not unique.

A very famous example is Anaconda (here to see how environments are managed) with its comprehensive command line tool conda.

conda environments are stored in one central directory <ANACONDA_ROOT>/envs.

conda create -h gives (cutout):

Target Environment Specification:
  -n ENVIRONMENT, --name ENVIRONMENT
                        Name of environment.
  -p PATH, --prefix PATH
                        Full path to environment location (i.e. prefix).

Hence, one can

1. set the environment name (--name) that is stored in one and the same central directory envsor
2. completely specifiy the absolute path (--prefix) for an environment

I think that covers every use case and lets the developer control everything.

[sinoroc]

@jamilraichouni I just skimmed quickly, so I might have missed it, but... How would that help for poetry? There is no lesson here that I can. These seem to be free-standing virtual environment that are not attached to anything. There is no mapping between a conda virtual environment and a specific project. You always have to call a conda virtual environment by its name chosen by the user: conda activate project-a1b2c3-py3.8. The mapping is only in the user's memory as far as I can tell. Conda does not know that a specific environment relates to a specific project, only the user does.

On the other hand, poetry uses a different approach where there is no need to remember names of virtual environments. With poetry you can cd to a directory containing a poetry-based pyproject.toml and call poetry shell (or basically any other command) and the correct virtual environment is picked up. So poetry needs a mapping logic. In that case it seems to be based on 1. the name of the project (read from pyproject.toml I suppose: project) 2. the full path to the pyproject.toml (which seems to be then hashed: a1b2c3) 3. the currently activated Python major.minor (which seems to be taken from the envs.toml file: py3.8).

The design choices in the UX are very meaningful. I am not sure what kind of UX you are expecting? How to reconcile both approaches? How to modify poetry's UX so that it is possible to select the name of the virtual environment? How commands such as poetry shell or poetry env use python3.9 would have to behave?

[julie777]

I'm not sure how relevant my experience is, but...

I am just starting to try and use Poetry and I will have the same problem. I currently have all my virtualenvs in a single directory under HOME and I share my home directory across multiple environments (Windows using bash and Linux running with WSL on Windows, and even VMs) Using virtualenvwrapper it creates a .project file in each virtualenv to provide the project directory for that virtualenv. Thus virtualenvs are linked to their projects. Each virtual env is given a name during creation that can be anything because it isn't used to connect it to the project.

As I have just started with Poetry, I haven't yet configured it to use VIRTUALENV_HOME yet. I wonder how much trouble that is going to create based on this issue?

As an aside virtualenvwrapper uses WORKON_HOME for the venv directory. I would really like it if all tools would standardize on VIRTUALENV_HOME. Currently every tool seems to hard code the location, put it in the current project directory, or provide its own environment variable name. This means that in bashrc I have to keep setting more and more variables to point to my virtualenv dir.

[ZelphirKaltstahl]

For anyone else stumbling upon this issue:

One thing you can do is to not install Poetry globally, but instead install the python3-venv system package. Then you can create a virtual environment and install Poetry inside it. That will make Poetry automatically use that virtual environment for installing package. Then really only use Poetry to install packages and for all other things just use . myvenv/bin/activate:

python3 -m venv venv
. venv/bin/activate
python3 -m pip install poetry==some-version-here
poetry init  # or copy a pyproject.toml file from somewhere
# edit your pyproject.toml file
poetry install  # or poetry lock for only making the lock file

This way you can have a named virtual environment, without having to deal with the unpredictability of Poetry's hash in the name. This is for example useful, when you want to automate things using make and such.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.