pip install --target does not work properly for packages with shared namespace

[monamaki]

Description

Installing packages in a target directory that share a common folder (shared namespace) does not work properly.
For example, consider zope.index and zope.interface packages. If --target is not provided, after installation, three directories are created including zope.index-5.1.0.dist-info, zope.interface-5.4.0.dist-info, and zope (shared folder between index and interface). The zope folder then contains two folders index and interface (each contained their source codes). This is a correct behavior.

However, if --target is provided to install these packages in a target directory, then the second package is not installed properly as pip does not put the source files in the common/shared directory. For example, if zope.index is installed first, then zope.interface cannot be installed properly since --target skips the installation step for moving the source files into the existing directory (the common zope folder).

Trying to work around this by adding --upgrade will result in removing the common directory (that was copied for the first package), and thus corrupting the first installed package.

Expected behavior

Following the same flow as when --target is not provided in pip install command. That is if the shared/common directory already exists, just appends to it by adding new directories for the packages that shared this directory. Indeed, if --upgrade is provided, the installation should overwrite the existing files (probably not removing?), if --upgrade is not provided, we shouldn't skip appending files to the common directory similar to when --target is not provided in pip install command.

Investigating the pip source code, the issue resides in _handle_target_dir in \_internal\commands\install.py.
Currently, it's like the following. When it wants to copy the unzipped wheel data, it says if the item (file/dir) exists:

1. If --upgrade param is not given, then skip copying the unzipped item.
2. If --upgrade is provided, and the copying item is a directory, rmtree the existing directory in the target.
3. Finally, move the unzipped directory/file as-is to the target directory/file (and thus complains if something exists there already)

My suggestion is to use something like shutil.copytree(src, dst, dirs_exist_ok=True) or distutils.dir_util.copy_tree() instead of move(), and thus don't rmtree if a directory exists there, as well as don't skip copying that item if --upgrade is not provided. In here, the item is the common directory.

pip version

21.3.1

Python version

3.7.1

OS

Windows

How to Reproduce

1. Get the packages from:
   https://pypi.org/project/zope.index/#files
   https://pypi.org/project/zope.interface/#files

2. Create a tmp folder in a drive, let's say D:

3. Run the following two commands:
   pip install zope.index-5.1.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir --target D:/tmp/
pip install zope.interface-5.4.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir --target D:/tmp/

4. Look into D:\tmp\zope, which doesn't have the directory interface with its sources.

Output

Processing zope.index-5.1.0-cp37-cp37m-win_amd64.whl
Installing collected packages: zope.index
Successfully installed zope.index-5.1.0

Processing zope.interface-5.4.0-cp37-cp37m-win_amd64.whl
Installing collected packages: zope.interface
Successfully installed zope.interface-5.4.0
WARNING: Target directory D:\tmp2\zope already exists. Specify --upgrade to force replacement.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[SudoNova]

I see similar behavior with 21.3.1

[driskell]

The target directory already existing is because there is two zope directories to install. One from pure lib dir and another from platform lib dir. zope.interface appears to install to, for example, lib64, whereas zope.event installs to lib - thus when copying the directories to the target directory there is now a problem with two zone folders attempting to copy.

Seems on some operating systems it's a non-issue since the platform lib dir is the same as the pure lib dir.

So I'm thinking the issue here is that pip install --target does not fully work anywhere that the platform lib dir is different to the pure lib dir as it installs packages separately to two locations but has no mechanism to merge them back. Perhaps when target is specified pip install --target needs to somehow ignore the platform lib dir during installation and install to a single lib directory.

[driskell]

Just as a point - Python 3.9 through homebrew on macOS has platform lib dir same as pure lib dir.
Red Hat 7 Python 3.6 has it different (it's lib64 for platform lib dir).

[Mylleranton]

I'm having the exact same issue with pip 21.3.1 and installing google-analytics-data (that pulls protobuf as a dependency) into a --target. Either the target will contain google-analytics-data and all dependencies except protobuf, or it will contain protobuf only. The wierd thing is that google-analytics-data pulls other dependencies with shared namespaces (for instance google-auth that installs into google/oauth2/) that seem to work.

To work with e.g. generating external packages for AWS Lambda, --target is an absolutely critical piece as it ultimately decides whether you can use external packages or not...

[fengliplatform]

I'm having same issue installing snowflake-snowpark-python

pip 22.2.2

Step 1 /tmp/sp doesn't exist before "pip install snowflake-snowpark-python -t /tmp/sp";

Step 2 /tmp/sp/snowflake/snowpark is created by "pip install snowflake-snowpark-python -t /tmp/sp" with WARNING. But /tmp/sp/snowflake/connector is NOT created. so only snowpark subdir in /tmp/sp/snowflake.

Step 3 when doing "pip install snowflake-snowpark-python -t /tmp/sp --upgrade", /tmp/sp/snowflake/connector is created but /tmp/sp/snowflake/snowpark is gone. so only connector subdir in /tmp/sp/snowflake.

In comparison, "pip install snowflake-snowpark-python" installed the two subdirs in default package location for example myenv/lib/python3.8/site-package/snowflake.

[duaneking]

This does not work for Django in my tests; doing a pip install --target ./current_deployment -r requirements.txt leaves me with empty Django and other folders. This makes deploying impossible in many scenarios.

[dwoz]

Is this a duplicate of #10110?

[driskell]

Is this a duplicate of #10110?

It is similar and related but not quite the same - in that instance the OP installed the packages separately in two install commands - and the second install command reports an error. I think the subsequent discussion potentially did end up reaching this issue though.

This issue is regarding a single install command using a requirements file of multiple packages, which arguably probably should return the error in #10110 if it were to be consistent - but actually completes successfully but you end up with only one of the packages installed.

[driskell]

Is this a duplicate of #10110?

I do however think the solution may be the same. So although different issues - related in that it feels like they might share a solution as the core problem for both is the handling of the shared directories in the destination folder

[jeremydvoss]

I am having this problem install multiple opentelemetry packages with "--target". Whichever comes first gets the correct "opentelemetry/..." folder structure and the later is broken. Is there a temporary workaround?

[jeremydvoss]

I am having this issue with OpenTelemetry. For instance, if I run:

pip install --target=/foobar opentelemetry-instrumentation-requests
pip install --target=/foobar opentelemetry-exporter-otlp-proto-grpc

... Then /foobar/openetelemetry/instrumentation/... exists but not /foobar/openetelemetry/exporter/... (The reverse is true if install order is reversed). However, a narrow workaround for me seems to be installing with the same command:

pip install --target=/foobar opentelemetry-instrumentation-requests opentelemetry-exporter-otlp-proto-grpc

Then, both folder structures exist for some reason.