-t flag seems to cause "--upgrade-strategy only-if-needed" to be ignored

[paciorek]

• Pip version: 9.0.3
• Python version: 3.6.3
• Operating system: Ubuntu 16.04

Description:

I'm trying to avoid upgrading dependencies unnecessarily when using -t to install a package in a specific location. However, when I use -t, dependencies are all upgraded even when I specify
--upgrade-strategy only-if-needed.

In contrast, --upgrade-strategy only-if-needed works as expected when used with --user.

What I've run:

Here's an example of installing pandas where I don't want to upgrade from the existing numpy (1.13.1)

## --user + --upgrade-strategy only-if-needed: works as expected 
$ pip install  --upgrade-strategy only-if-needed --user pandas
Collecting pandas
  Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: numpy>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from pandas)
Collecting python-dateutil>=2 (from pandas)
  Using cached python_dateutil-2.7.2-py2.py3-none-any.whl
Collecting pytz>=2011k (from pandas)
  Using cached pytz-2018.3-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
  Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, pandas
Successfully installed pandas-0.22.0 python-dateutil-2.7.2 pytz-2018.3 six-1.11.0

## -t + --upgrade-strategy only-if-needed: does not work as expected 
 $ pip install  --upgrade-strategy only-if-needed -t /tmp/pandas pandas
Collecting pandas
  Downloading pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl (26.2MB)
    100% |################################| 26.3MB 64kB/s 
Collecting python-dateutil>=2 (from pandas)
  Downloading python_dateutil-2.7.2-py2.py3-none-any.whl (212kB)
    100% |################################| 215kB 7.2MB/s 
Collecting pytz>=2011k (from pandas)
  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
    100% |################################| 512kB 3.0MB/s 
Collecting numpy>=1.9.0 (from pandas)
  Using cached numpy-1.14.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
  Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Successfully installed numpy-1.14.2 pandas-0.22.0 python-dateutil-2.7.2 pytz-2018.3 six-1.11.0

[benoit-pierre]

Unfortunately using -t enable --ignore-installed.

[pradyunsg]

I'm not sure what should be the correct behaviour here.

[pfmoore]

I think it would be reasonable if -t didn't force --ignore-installed, but from what I recall, the fact that it does is a technical limitation of pkg_resources' method of locating installed versions (which relies on them being on sys.path). I don't remember the details, though, and I don't know if it's easy to fix (faking pkg_resources into thinking that the target directory is on sys.path may be enough).

I should note that -t should ignore anything that's installed outside of the target directory - even if the user has requests installed somewhere on sys.path, pip install -t foo requests must install a copy of requests into foo.

[davidedelvento]

I do not know the implementation details, but this is a horrible bug, which triggers lots of problems. Here is my situation.

I am using a NVIDIA-provided container which does have numpy but not venv:

> python --version
Python 3.5.2

> pip show numpy
Name: numpy
Version: 1.14.5
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /usr/local/lib/python3.5/dist-packages
Requires: 
Required-by: tensorflow-gpu, tensorboard, scipy, scikit-learn, resampy, pandas, numba, matplotlib, librosa

> python -m venv my_env
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  

I am not root on the machine (not even inside the container) and I cannot apt-install stuff.

So as a workaround, I am using -t and PYTHONPATH, and that works to avoid the problem above, but creates two new ones. The first is that it ignores the installed one (as this issue describes). But I could live with that, if pip did not break the installation as it does:

> pip install --upgrade --upgrade-strategy=only-if-needed --target ./my_env patsy 
Collecting patsy
  Using cached https://files.pythonhosted.org/packages/ea/0c/5f61f1a3d4385d6bf83b83ea495068857ff8dfb89e74824c6e9eb63286d8/patsy-0.5.1-py2.py3-none-any.whl
Collecting six (from patsy)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting numpy>=1.4 (from patsy)
  Using cached https://files.pythonhosted.org/packages/f6/f3/cc6c6745347c1e997cc3e58390584a250b8e22b6dfc45414a7d69a3df016/numpy-1.16.3-cp35-cp35m-manylinux1_x86_64.whl
ERROR: tensorflow-gpu 1.10.0+nv has requirement numpy<=1.14.5,>=1.13.3, but you'll have numpy 1.16.3 which is incompatible.
Installing collected packages: six, numpy, patsy
Successfully installed numpy-1.16.3 patsy-0.5.1 six-1.12.0

The second problem, is that while installing the latest version of numpy, pip ignores a dependency of another package (tensorflow in this example) and breaks is. That deserves opening a different issue, which I will do (if I could not find it), but that problem is triggered for me by this bug.

And good thing that this is just a directory I can just delete, because pip is unable to fix the mess it created itself!!

> pip install --upgrade numpy==1.14.5 -t ./my_env
Collecting numpy==1.14.5
  Using cached https://files.pythonhosted.org/packages/43/17/cd9fa14492dbef2aaf22622db79dba087c10f125473e730cda2f2019c40b/numpy-1.14.5-cp35-cp35m-manylinux1_x86_64.whl
ERROR: tensorflow-gpu 1.10.0+nv has requirement setuptools<=39.1.0, but you'll have setuptools 41.0.1 which is incompatible.
ERROR: tensorflow-gpu 1.10.0+nv has requirement tensorboard<1.11.0,>=1.10.0, but you'll have tensorboard 1.13.1 which is incompatible.
Installing collected packages: numpy
Successfully installed numpy-1.14.5

Do you think that's good? Think again:

> pip show numpy
Name: numpy
Version: 1.16.3
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: ./my_env
Requires: 
Required-by: torch, Theano, tensorflow, tensorflow-estimator, tensorboard, tables, seaborn, scipy, PyWavelets, patsy, pandas, odo, numexpr, matplotlib, Keras, Keras-Preprocessing, Keras-Applications, imageio, h5py, datashape, Bottleneck, bokeh, bkcharts, astropy, tensorflow-gpu, scikit-learn, resampy, numba, librosa

As the original poster noted, using just --user instead of -t works:

> pip install patsy --user
Collecting patsy
  Using cached https://files.pythonhosted.org/packages/ea/0c/5f61f1a3d4385d6bf83b83ea495068857ff8dfb89e74824c6e9eb63286d8/patsy-0.5.1-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.4 in /usr/local/lib/python3.5/dist-packages (from patsy) (1.14.5)
Requirement already satisfied: six in /usr/local/lib/python3.5/dist-packages (from patsy) (1.11.0)
Installing collected packages: patsy
Successfully installed patsy-0.5.1

Now, I understand that technically --user and -t are treated differently, but really they should not. They are just different directories!

[pfmoore]

Now, I understand that technically --user and -t are treated differently, but really they should not. They are just different directories!

They are not. The user directory is on sys.path and is a site-directory (which means that, for example, pkg_resources can locate and introspect packages installed in that location). The directory in --target is neither of those things.

It would probably be nice if pip could install identically to arbitrary locations. But at the moment it can't, and there are fundamental and difficult-to-solve reasons why this is true.

[davidedelvento]

I got that they are not the same and therefore are treated differently. I wrote they should be treated the same, and I am convinced of that statement.

Now, to stop commiserating and possibly help others stumbling in this horrible mess, I'm going to just manually (or perhaps with a script) move my ~/.local/ somewhere, do a --user install, then move the installed crap somewhere else (to avoid conflicting with other environments which I need on that machine), restore the ~/.local/ and pray that there are not absolute paths in any of the 200+ packages that I will be relocating, so that the installed crap can work when moved around. That sucks, and I suspect that will not work, but that's the only thing I can do at this point.

[pradyunsg]

The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

This is a consequence of Debian deciding that a part of the Python Standard library should not be shipped as part of Python by default. Please feel free to take this up with Debian's Python packagers.

[qhaas]

In contrast, --upgrade-strategy only-if-needed works as expected when used with --user.

While the default of only-if-needed is used with --user by the version of pip3 (9.0.3) deployed by the python36-pip package in EL7.8, this isn't the case with the version of pip3 (9.0.1) deployed by the python3-pip package in Ubuntu 18.04.5 when done by a non-root user. See attached output:
ubuntupip3.txt

Update: Per uranusjr's suggestion, created this ubuntu question

[uranusjr]

@qhaas pip developers do not control the pip packages distributed by Debian (and Ubuntu). You need to report this to Debian instead.

[perillo]

I'm also having problem with the --target option. When using the --prefix option, pip uses the only-if-needed strategy by default and it works correctly.

Note that, in my case, prefix is not on sys.path, I'm not using the --user option and my OS is ArchLinux.

Are there any news about a possible solution?

I also checked using the legacy-resolver and the behavior is the same.

Thanks.

[davidedelvento]

Are there any news about a possible solution?

I may be wrong, but I think the only solution is to always and only use virtualenvs.

[perillo]

@davidedelvento unfortunately virtual environments have some UX issues.
As an example, pipx enable setting the --system-site-packages, but not other options like --symlinks.
hatch allow setting system-site-packages in pyproject, but IMHO this is a choice that must be done by the user, not by the package author.