pip install --target --upgrade removes existing packages scripts

[changsijay]

Environment

• pip version:20.0.2
• Python version:3.8.2
• OS:MacOS

Description
pip with -t removed other file in target bin folder

Expected behavior

when install package ipython should not remove package pylint bin files

How to Reproduce

mkdir test
pip install -t test pylint
ls test/bin/
epylint  isort  pylint  pyreverse  symilar

pip install -U -t test ipython
ls test/bin/
easy_install  easy_install-3.8  iptest  iptest3  ipython  ipython3  pygmentize

Output

After install ipython, pylint is deleted.

[uranusjr]

Title edited to improve readability. I hope it is alright.

[deveshks]

As per the docs at https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t , providing -U flag while installing will replace existing files/folders in the target directory.

This can also be observed by not passing the -U flag in the second pip install, due to which the contents of test/bin don't change, and warnings related to not being able to install because of files already being present are raised.

(.env) DeveshSinghMac:Desktop devesh$ pip --version
pip 20.0.2 from /Users/devesh/Desktop/.env/lib/python3.8/site-packages/pip (python 3.8)
(.env) DeveshSinghMac:Desktop devesh$ python --version
Python 3.8.2

(.env) DeveshSinghMac:Desktop devesh$ pip install -t test ipython
....
WARNING: Target directory /Users/devesh/Desktop/test/decorator-4.4.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/appnope already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython-7.13.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/wcwidth already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/backcall already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/jedi-0.17.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/easy_install.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pygments already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pickleshare-0.7.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/IPython already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/Pygments-2.6.1.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/decorator.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/jedi already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython_genutils already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pickleshare.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/setuptools-46.1.3.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/__pycache__ already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/prompt_toolkit-3.0.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/six-1.14.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/prompt_toolkit already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython_genutils-0.2.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pexpect already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ptyprocess-0.6.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/backcall-0.1.0-py3.8.egg-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ptyprocess already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/six.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pexpect-4.8.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/traitlets-4.3.3.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/appnope-0.1.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/setuptools already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pkg_resources already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/parso-0.7.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/parso already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/traitlets already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/wcwidth-0.1.9.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/bin already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/share already exists. Specify --upgrade to force replacement.

(.env) DeveshSinghMac:Desktop devesh$ ls test/bin/
epylint		isort		pylint		pyreverse	symilar

So I would say that providing -U, which ends up cleaning test/bin is the expected behaviour IMO

[gutsytechster]

@deveshks, the documentation you referred, says

Use --upgrade to replace existing packages in <dir> with new versions.

What I can infer is that it would replace existing packages with new versions. Should it remove the existing packages if not specified?

[deveshks]

Should it remove the existing packages if not specified?

It seems so from the output. All the files present in test/bin/ have been replaced after using --target --upgrade

[leos]

With pip 20.1:

~/ ❯❯❯ pip install -t test pylint --no-deps                                                                                                                                                 
Collecting pylint
  Using cached pylint-2.5.0-py3-none-any.whl (324 kB)
Installing collected packages: pylint
Successfully installed pylint-2.5.0
~/❯❯❯ pip install -t test ipython --no-deps                                                                                                                                                     
Collecting ipython
  Using cached ipython-7.14.0-py3-none-any.whl (782 kB)
Installing collected packages: ipython
Successfully installed ipython-7.14.0
WARNING: Target directory /home/vagrant/beeswaxio/test/bin already exists. Specify --upgrade to force replacement.
~/❯❯❯ ls test/bin                                                                                                                                                                               
epylint  pylint  pyreverse  symilar
~/❯❯❯ rm test
~/❯❯❯ pip install -t test pylint ipython --no-deps
Collecting pylint
  Using cached pylint-2.5.0-py3-none-any.whl (324 kB)
Collecting ipython
  Using cached ipython-7.14.0-py3-none-any.whl (782 kB)
Installing collected packages: pylint, ipython
Successfully installed pylint-2.5.0 ipython-7.14.0
~/❯❯❯ ls test/bin                                                                                                                                                                               
epylint  iptest  iptest3  ipython  ipython3  pylint  pyreverse  symilar

I think the bug here is that in --target mode across separate runs pip doesn't treat the bin/ directory as a shared directory.

Without target mode it has no problem adding script symlinks to/from a virtualenv's bin directory without either removing the whole thing or failing to add symlinks.

I'd like to be able use --target to install multiple packages into one directory across multiple pip operations but it looks like that doesn't work right now.

[dulfox]

I have the same issue with python 3.6 and pip 20.1.1.
It's a real drag to make optimized AWS lambda layers.
Is there a workaround for this issue ?

[leos]

@dulfox my workaround for this has been to pip install --no-deps each package into its own separate directory and then manage the symlinks myself. It's not as bad as it sounds, ~20 lines of python.

[dulfox]

I'm looking for an answer for the broadest possible case : I want to bring as many dependencies on board as I can.
I use a shell script via Jenkins to generate my AWS layers with virtualenv and pip install -t /layer_dest.
The output directory will be then zipped and loaded with terraform.
But @leos your python code may be useful for others, maybe you could publish it on a github gist...

[MumblingFumbler]

On a windows system, I define
target in a pip.ini file
Then I run
'pip install ipython'
which installs binaries in ...\site-packages\bin
then I run
'pip install cython'
I get complaints from pip about 'target directories already exist', and a recommendation to run
'pip install cython --upgrade'
when I do that, pip deletes the ....\site-packages\bin directory (including ipython binaries),
creates a new ....\site-packages\bin directory and puts cython binaries there.
This may be the expected behavior for pip, but it is not reasonable behavior.
It is not reasonable that the installation of binaries from one package would clobber
installation of binaries from another. Neither package should assume ownership of
....\site-packages\bin directory, as binaries from many packages should be able to coexist in this
directory. Why wouldn't the installation of one just add files to ....\site-packages\bin directory? Or
at least pip should have an option to do that...

[sbraz]

Hi, is there any chance to see a fix for this issue in the near future? This behaviour is rather counter-intuitive and I don't really see any easy workaround apart from installing to different targets and merging directories afterwards.

[arekgoral]

Same issue here. Is there an option to ramp up the
pip install -t --upgrade
with another switch that lest you to append files rather than to overwrite existing ones?

[pawamoy]

And one more data point here.

I'm installing my project dependencies first with pdm install --prod --no-lock, which contain namespaces packages under, say, the hello namespace. Then I install the project itself, which is also a hello namespaced package, with pip install --use-feature=in-tree-build . --no-deps -t __pypackages__/3.8/lib. It fails with the warning mentioned above.

If I add the --upgrade option, the previously installed namespace packages are removed from the hello namespace. Is this intended? Maybe namespace packages were not taken into account for the --target option?

[pfmoore]

To be perfectly honest, pip install --upgrade --target simply isn't a supported option, and it's unlikely that it can be "fixed up" to work correctly. The tools that read metadata look on sys.path, and so don't work with an arbitrary --target. And we're not going to write our own copies of those tools just for this one uncommon usage. We may at some point implement a more general "install scheme" mechanism to replace --target with something that's more in line with other target locations. But that's a big change and we don't have the resources to commit to any timescale for something like that.

So for now, I'd say:

1. Don't use --target and --upgrade together. If the docs say it's supported, point out where and we'll accept PRs updating the docs to remove that statement.
2. If you need to "upgrade" a --target directory, delete it and re-create it from scratch.

If that's not sufficient for you, you need to write something customised for your particular situation.

Sorry if this isn't what the people commenting on this issue want to hear, but I'd rather set expectations clearly than leave people thinking that this will be "fixed" at some point and trying to struggle on in the meantime.

[pawamoy]

Thank you for your answer @pfmoore.

Don't use --target and --upgrade together. If the docs say it's supported, point out where and we'll accept PRs updating the docs to remove that statement.

Not sure about the docs, but the warning is definitely suggesting to use the upgrade option with the target one. Maybe this can be rephrased?

-WARNING: Target directory /path/to/target/namespace already exists. Specify --upgrade to force replacement.
+WARNING: Target directory /path/to/target/namespace already exists. Specify --upgrade to force replacement (existing 'namespace' will be replaced completely, not merged)

If you need to "upgrade" a --target directory, delete it and re-create it from scratch.

It won't do for my use-case ^^ but for others maybe 🙂

If that's not sufficient for you, you need to write something customised for your particular situation.
Sorry if this isn't what the people commenting on this issue want to hear, but I'd rather set expectations clearly than leave people thinking that this will be "fixed" at some point and trying to struggle on in the meantime.

Of course, that's perfectly understandable, thank you for making it clear.
The best solution for me anyway would be to change the behavior of the project manager I use, PDM, so I don't have to rely on pip install -t at all 🙂

[PhracturedBlue]

The thread says not to use '--upgrade' with '--target', and that is ok. But without it, if 2 different modules install into /bin, only the 1st will put its executables there. As far as I can tell there is no way withy '--target' to install 2 different modules that both expect to put their files in the bin dir. That makes --target have a very limited use

[pspot2]

Sorry, but... is this for real? How is one supposed to install packages whose dependencies also need to be installed into the same namespace? Try to install google-cloud-storage with the -t option. You'll end up with the aforementioned warning and without protobuf. Using the --upgrade option will erase everything and leave only protobuf in the namespace dir.

The -t option is absolutely vital for packaging stuff for AWS Lambda and Co. Installing to shared directories using this mode is absolutely vital. Besides, this is the only option to package cross-platform wheels (for architecture X using architecture Y).

[monamaki]

We also need it for our use case where we install packages that have the same shared/common directory (i.e. namespace). For example, zope.index and zope.interface share zope directory after installation. So, if --upgrade is specified, the content of zope will be overwritten not appended by second pip installation; Otherwise, if --upgrade is not specified, then the content of the second installation doesn't get written in the zope directory at all and a warning is shown.

Repro steps on a Windows machine:
pip install zope.index-5.1.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir --target D:/tmp/
pip install zope.interface-5.4.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir --target D:/tmp/

Then, look into the zope directory here: D:/tmp/zope.

If --target is not specified at all, then installing these packages from the wheel is fine:

pip install zope.index-5.1.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir
pip install zope.interface-5.4.0-cp37-cp37m-win_amd64.whl --no-dependencies --ignore-installed --no-cache-dir

Do you suggest any workaround for us?
Thanks

[acgzr]

I'm currently testing if there are any issues with my solution, but in case anyone else is on Windows and willing to try, you could use a directory junction from the default pip installation folder to a custom folder (instead of setting this custom folder via "--target").

[wilcox-liam]

To be perfectly honest, pip install --upgrade --target simply isn't a supported option, and it's unlikely that it can be "fixed up" to work correctly.

Essentially there is no support at all for a non-standard pip installation directory? All I want is for 'site-packages' and bin to be stored in another location, without overriding each other and without having to sym link it?

Another link on the same issue with a proposed solution.

#10629

[driskell]

I found that with regards to #10629 this message about target directory already existing appears to be related to the fact that pip usually installs to two different folders. A platform library directory (e.g. lib64) and a standard library directory (e.g. lib). The issue is around that some packages go into one and others into the other, and it can create a conflict when trying to copy them to a single directory, since the installation process has already "split" them across two locations.

Not sure if this is the same process happening for bin (since the above is for lib) but maybe it is.

Seems pip install -t is not going to work on any operating system where the python installation has a separate platform library directory. Looks like it might be workaroundable since Python 3.9 since you can override the platform lib dir through environment variable but for older it might need pip to somehow install to only a single directory and ignore platform lib, so that the copying to the target directory is safe and only copies from a single place. But I can see in code there is a "data_dir" too and that's getting pumped into the same place too so there's still a possibility of conflict - but perhaps that's a less problematic area?

[driskell]

I managed to workaround it by using virtualenv and just copying out from the lib folder. It appears that virtualenv just symlinks lib64 to the lib folder. Maybe pip could do something similar when target_dir is specific (I did a quick test and it does seem it works fine... just I have no idea how to properly handle Windows :) )

[pfmoore]

Essentially there is no support at all for a non-standard pip installation directory?

The only really supported use case for --target is to install a set of packages into a directory, for vendoring or bundling into an application. The expected way of "upgrading", or indeed any modification of the set of installed packages is to delete the target directory, and rebuild it from scratch.

The structure of the directory created by --target is not intended to match the exact structure of a Python installation (most significantly because a Python installation can be spread across multiple independent directories) and so doesn't match the structure that upgrades need.

If you want a directory structure that matches a Python installation, and therefore should support upgrades, etc, then --prefix may work better for you (disclaimer: I've never used --prefix myself, so I don't know for sure).

[driskell]

@pfmoore I think there's two issues in this issue, and maybe I've polluted it a bit, sorry.

1. You cannot run pip install -t twice to the same target with different requirements. The second run is unable to "append" to the target and will instead skip anything that already exists there. This is where --upgrade is getting used in due to the warning message but the intention is that pip installs to the same target should have the same requirements as the original call. It is not intended to use --upgrade as a method to add additional files to a target. In the OP issue it should be pip install -t test -U pylint ipython to overwrite target with new set of requirements.
2. You cannot pip install a single set of requirements that have mixed pure lib and platform lib installations with the same name, on a platform where the pure lib and platform lib folders are different, as pip will not merge these together. The resulting copy to the target will be broken (zone.interface + zone.event is prime example here) as the copy from pure lib will succeed, but the copy from platform lib will see the target already exists (zone) and refuse to copy. Specifying -U will not help here as it will just mean the platform lib copy of zone will overwrite the pure lib version and thus you'd lose the pure lib version.

[pfmoore]

@driskell Correct, as I say, the expected use case (or at least my expectation) is that you don't do (1), you delete the directory and reinstall everything. As for (2), that's #10629, as you said, and I don't really know the right answer for that one, as I don't do much on systems that have purelib != platlib, so I don't have a good intuition on what that should do (or even why having purelib != platlib is a reasonable thing to have at all...). So I'll leave #10629 for someone else to investigate.

[pspot2]

@pfmoore the following command fits to what you describe to be a valid/supported use-case:

pip install google-cloud-storage -t mydir

E.g. run just once and no upgrades. Yet, this package (naturally) pulls its dependencies, one of them being protobuf. This dependency is the reason for the following message:

WARNING: Target directory /path/to/mydir/google already exists. Specify --upgrade to force replacement.

mydir ends up not containing google.protobuf and the software the package is embedded into (e.g. for example AWS Lambda) doesn't work:

Unable to import module 'xyz': No module named 'google.protobuf'

The installation to a standard location (or virtualenv), e.g. without -t works just fine, which suggests this is less a problem of the package itself, but rather a problem of pip install -t.

[pfmoore]

Hmm, OK so I suspect there's a bug with --target and implicit namespace packages. Feel free to raise it as a separate issue (it's not the same as the problem originally posted in this issue). And even better, feel free to provide a PR fixing it! (I can't honestly offer any idea of whan it will be fixed unless a community member offers a fix - --target is an odd beast, and frankly fixing bugs with it is a pretty low prority, sorry.)