Build isolation fails with system-wide pip<10 and user-site pip==10.0.0

[pv]

• Pip version: 10.0.0
• Python version: 3.6.5
• Operating system: Linux/Fedora

Description:

Apparently, the combination of system-wide installation of pip<10 and user-site installation of pip>=10 makes projects with pyproject.toml fail to build.

I would expect user-site installation of pip to work.

What I've run:

$ python3 -mpip install --user --upgrade pip
Requirement already up-to-date: pip in ./.local/lib/python3.6/site-packages (10.0.0)
$ mkdir ~/foo
$ cd ~/foo
$ echo "assert False" > setup.py
$ cat <<EOF > pyproject.toml
> [build-system]
> requires = [
>     "wheel",
>     "setuptools",
> ]
> EOF
$ ls
pyproject.toml  setup.py
$ python3 -mpip --version
pip 10.0.0 from /home/pauli/.local/lib/python3.6/site-packages/pip (python 3.6)
$ python3 -mpip wheel .
Processing /home/pauli/foo
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tlxjmy36 https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc:
  
  Usage:
    /usr/bin/python3 -m pip install [options] <requirement specifier> [package-index-options] ...
    /usr/bin/python3 -m pip install [options] -r <requirements file> [package-index-options] ...
    /usr/bin/python3 -m pip install [options] [-e] <vcs project url> ...
    /usr/bin/python3 -m pip install [options] [-e] <local project path> ...
    /usr/bin/python3 -m pip install [options] <archive url/path> ...
  
  no such option: --no-user
  
  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tlxjmy36 https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc" failed with error code 2 in None

Note that the error occurs before executing setup.py

[pfmoore]

When you say "system-wide pip<10 and user-site pip==10.0.0" you have (different versions of) pip installed in the 2 places? That sounds unlikely to work. I'm not even sure it's something that Python supports...

[pv]

The system-wide python install essentially always has some version of pip installed.
This is the default configuration pretty much on all Linux systems.

Installing newer versions of software using "pip install --user" has always worked for me so far.
Clearly, this is something that python does support. I don't know why exactly it breaks for pip
this time (it has also worked for previous pip versions), maybe it has to do with how pip sets up
the temporary environment for build dependencies.

Using the "--user" flag is even recommended here:
https://packaging.python.org/tutorials/installing-packages/

[pv]

The problem seems to be that something inside pip adds PYTHONNOUSERSITE=1 to the environment which then causes a failure at https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/prepare.py#L67

[pfmoore]

Installing packages in user-site is indeed recommended. But that should be instead of the system site-packages, not as well. I don't think anyone has ever recommended having 2 different versions of a package installed in 2 separate locations onsys.path.

The failure is because when pip 10 tries to run python -m pip, it's somehow picking up an older version of pip which doesn't support the --no-user flag. I'm inclined to classify that as a broken environment, but if you can provide a reproducible test case, we can review whether or not it's a scenario that we should handle better.

[pv]

I believe it's not correct to state that different versions in sys.path are not supported --- it's always been the case that the package that comes first wins.

This is also something that's assumed by several scenarios that deal with setting PYTHONPATH environment variable.

I already provided a reproducible test case above.

[pv]

If pip does not support having different versions installed in system site-packages and user site-packages, then "pip install --upgrade --user" should simply refuse to install a newer version of the package to the user-site directory if it's already present system-wide.

However, this does not seem like it was the intent with the --user flag, hence I think what's broken is how the build depends setup works in pip 10, not the fact that you can do pip install --user --upgrade pip.

[benoit-pierre]

Switching to a more recent version of a system package by installing a newer version in the user site is common use case.

Anyway, I don't think the command to install the build requirements should be isolated: it's already using --prefix and --ignore-installed, anything more is going to be problematic if pip itself is installed in the user site (whether or not another version of pip is available system wide), or if one of pip's dependency is (when pip has been un-vendored).

[pfmoore]

The reason the build requirements are installed using --no-user is #5085. The problem is that --user and --prefix cannot be used together.

But the issue here is that for some reason, when pip runs [sys.executable, '-m', 'pip', ...] it's picking up a different version of pip (one that doesn't support the --no-user flag introduced in pip 10). As far as I know, that should be impossible without the user's system being broken. I think the key here will be isolating how that's happening.

[benoit-pierre]

The problem is not --no-user, but using the isolated environment (with PYTHONNOUSERSITE=1), no?

[pfmoore]

Oh! I see what you're saying - https://github.com/pypa/pip/blob/master/src/pip/_internal/build_env.py#L56 is where PYTHONNOUSERSITE is set.

I'm not the best person to comment on that code - it's dealing with situations I never really understood in detail. @pradyunsg are you able to comment?

[benoit-pierre]

I'm looking into it, but as stated above, I don't think we want build isolation for the command used to install the build dependencies.

[pfmoore]

Cool thanks

[benoit-pierre]

@pv: could you confirm #5227 fix your issue?

[pradyunsg]

(ignore that last comment)

[blaiseli]

Just in case this may help others in the same situation as me, who ended up here searching for a solution: I have a local python 3.6 install, with user=1 in my ~/.pydistutils.cfg file (on and Ubuntu 16.04 system, if that matters), and I just got no such option: --no-user issues. I solved this by deleting my ~/.local/lib/python3.6/site-packages/pip and performing a python3.6 -m pip install --upgrade pip (where python3.6 is my local python, i.e. ~/bin/python3.6). pip3.6 (which is ~/.local/bin/pip3.6) is working again.

[Preston-Landers]

Here's another way I ran into this problem and how I resolved it in case it helps anyone. I have some build scripts for my software which are also used to upgrade a developer's environment in-place after we update pip and other package versions.

What happened is that, when the old version of pip is still installed, my script untargz'ed the new pip and ran setup.py install on it. This seemed to happily install the new version of pip while also leaving the old one in place, causing various problems.

The fix for my upgrade script was to run this first:
python -m pip install --upgrade pip
Then run the rest of the script as normal.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.