How to save disk-space for duplicated libraries.

[yingbo]

If I have multiple projects, using a same python version, and they all use some same libraries. Is there a way to not duplicate the libraries in each project?

For example, if I have five projects, all using python 3.9.5, tensorflow 2.8.0. What I'm trying to avoid is have five duplicate copies of tensorflow.

Thanks!

[sathoune]

Intreseting idea that can create multiple problems. I did not hear about this before or heard about any project that would need that. After quick googling I found this: https://github.com/jsvine/envplus project is old and not maintained but you might want to try this out.

Have you checked if Conda (Anaconda) or other environment managers maybe could help?

If you are worried about "environment duplication" I'd like to discourage you from thinking about it.

If you are running into problem with disk space on you machine you can try:

• Buying 16/32 GB pendrive to store all your projects/environments there.
• Removing environments for projects you did not touch for longer than a month - this is not bad because you have a recipe (poetry.toml) so you can recreate it anytime you want and it won't take more than 5 minutes. (I mainly use docker but still I routinely clear the images because sometimes it can takes up to even 100 GB)

I think you either want to have isolated environments or you don't.

[abn]

An option is to use the system-site packages for your projects. You would do something like this.

poetry config virtualenvs.options.system-site-packages true
python3.9 -m pip install tensorflow
poetry env use python3.9

This will only work if the tensorflow versions across your projects are always the same. If you really want to go down the rabbit hole look at fiel system solutions like overlay etc. Or even get base containers built for your venvs and use solutions like toolbox.

[yingbo]

I saw people have different thoughts, and maybe confusion of the request. Thinking this way, if I have two projects, both using python 3.9.5 and tensorflow 2.8.0. We usually have only one copy of the Python 3.9.5, either from pyenv, brew, system, condo or whatever installation method. Question is why we need to duplicate tensorflow 2.8.0? We do NOT need to the duplicate the tensorflow 2.8.0 for python 3.9.5, same as we don't need to duplicate the python 3.9.5. (I mentioned exact versions all the time to help think)

Here is a python word example, https://pdm.fming.dev/configuration/. The PDM supports the catch in two ways. I think it's great if the poetry can do the similar thing. You can are argue all details on how to implement it to make a good isolation, but we can see that we don't need the duplications in some cases.

[abn]

@yingbo using the system-site-package for your project environment (note this is not re-using the same environment, just the installations), This is similar to caching unpacked wheels. I am not sure sure what you mean by referring to PDM. Since PDM uses PEP 582, the way it works is different.

Also, virtualenvs do not duplicate the python installations themselves. You are reusing your single installation already there.

$ virtualenv --no-pip --no-setuptools --no-wheel --system-site-packages --symlinks venv
created virtual environment CPython3.10.4.final.0-64 in 63ms
  creator CPython3Posix(dest=/home/abn/venv, clear=False, no_vcs_ignore=False, global=True)
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ tree venv/
venv/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.nu
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── deactivate.nu
│   ├── python -> /usr/bin/python3
│   ├── python3 -> python
│   └── python3.10 -> python
├── lib
│   └── python3.10
│       └── site-packages
│           ├── _virtualenv.pth
│           └── _virtualenv.py
├── lib64
│   └── python3.10
│       └── site-packages
└── pyvenv.cfg

7 directories, 13 files

You will also see that the site packages are available here including pip from global environment.

$ ./venv/bin/python -m pip show zipp
Name: zipp
Version: 3.7.0
...
Required-by: importlib-metadata
$ ./venv/bin/python -m pip show httpx
WARNING: Package(s) not found: httpx
$ ./venv/bin/python -m pip install httpx
Collecting httpx
...
$ ./venv/bin/python -m pip show httpx
Name: httpx
...
$ python -m pip show httpx
WARNING: Package(s) not found: httpx

[yingbo]

Hi @abn ,

First, "using the system-site-package for your project environment" doesn't match the benefit of using cache. Given an example, what if a person has six projects, three of them using python 3.9.5 + tensorflow 2.8.0, while the other three using python 3.9.5 + tensorflow 2.7.0?

Second, you are right that PDM is different than poetry. I'm using the PDM as an EXAMPLE showing it's possible to use cache and hope it triggers some thoughts how to do it in poetry

Last, yes, virtualenvs do not duplicate the python installations themselves. This is exactly what I'm trying to point out: using cache correctly won't affect isolation, same as using same python won't affect isolation either.

[abn]

As I mentioned in my initial comment, this only works if the same version applies.

If you can provide an example of how this might work outside of PEP 582, or even better, a poc pull request, happy to review it.

In theory, adding additional paths or symlinking "installed wheel" from a dummy environment might do the trick. Which is what I am assuming you are trying to get to. However, this adds a layer of complexity that might not be desired. Nonetheless, it can be explored. Either as a core feature or as a plugin.