pip/_internal/locations.py not respecting site.ENABLE_USER_SITE

[bjmuld]

Environment

• pip version: 19.2.2
• Python version: 3.7
• OS: Ubuntu

Description
user site is disabled in site.py (and command-line [-s], and PYTHONNOUSERSITE env var)
but pip is still looking in ~/.local/lib/python*/site-packages/ during (un)installation of packages

Expected behavior
pip should respect site.ENABLE_USER_SITE=False

How to Reproduce

1. install an editable "user" package in ~/.local : pip install --user -e ~/xanity-src/
2. create a new python installation at /opt/my-python
3. install a package there: PYTHONNOUSERSITE=1 /opt/my-python/bin/python -m pip install -e ~/xanity-src/
4. try to uninstall that package: PYTHONNOUSERSITE=1 /opt/my-python/bin/python -m pip uninstall xanity
5. error "Egg-link * does not match installed location of xanity (*)"

Output

bjmuld@disco:~$ pip install -e ~/work/xanity/src/xanity/xanity_self_replication/
Obtaining file:///home/bjmuld/work/xanity/src/xanity/xanity_self_replication
Requirement already satisfied: dill in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from xanity==0.2b9) (0.3.0)
Requirement already satisfied: pandas in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from xanity==0.2b9) (0.25.1)
Requirement already satisfied: pyparsing in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from xanity==0.2b9) (2.4.2)
Requirement already satisfied: cloudpickle in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from xanity==0.2b9) (1.2.2)
Requirement already satisfied: numpy>=1.13.3 in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from pandas->xanity==0.2b9) (1.16.5)
Requirement already satisfied: pytz>=2017.2 in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from pandas->xanity==0.2b9) (2019.2)
Requirement already satisfied: python-dateutil>=2.6.1 in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from pandas->xanity==0.2b9) (2.8.0)
Requirement already satisfied: six>=1.5 in ./work/chonets/.xanity/conda_env/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas->xanity==0.2b9) (1.12.0)
Installing collected packages: xanity
  Found existing installation: xanity 0.2b9
ERROR: Exception:
Traceback (most recent call last):
  File "/home/bjmuld/work/chonets/.xanity/conda_env/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/home/bjmuld/work/chonets/.xanity/conda_env/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/home/bjmuld/work/chonets/.xanity/conda_env/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 51, in install_given_reqs
    auto_confirm=True
  File "/home/bjmuld/work/chonets/.xanity/conda_env/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 830, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/home/bjmuld/work/chonets/.xanity/conda_env/lib/python3.7/site-packages/pip/_internal/req/req_uninstall.py", line 538, in from_dist
    '(at %s)' % (link_pointer, dist.project_name, dist.location)
AssertionError: Egg-link /home/bjmuld/work/xanity/src does not match installed location of xanity (at /home/bjmuld/work/xanity/src/xanity/xanity_self_replication)

[chrahunt]

#4575 is tracking the implementation of the ability to specify which of the system, user, or virtual environment (among other schemes) you want pip to manage at a given time.

In the meantime if you try e.g. PYTHONUSERBASE=/dev/null /opt/my-python/bin/python -m pip uninstall xanity it should disregard the xanity installed in your home directory.

IMO the environment pip is managing packages for is independent of whether the packages for that environment are in sys.path. For that reason I wouldn't want to start explicitly behaving differently when PYTHONNOUSERSITE is set.

[bjmuld]

In the meantime if you try e.g. PYTHONUSERBASE=/dev/null /opt/my-python/bin/python -m pip uninstall xanity it should disregard the xanity installed in your home directory.

Adopting this for now.

...For that reason I wouldn't want to start explicitly behaving differently when PYTHONNOUSERSITE > is set.

I don't follow you here. Setting PYTHONNOUSERSITE just sets ENABLE_USER_SITE=False in site.py. I don't see what that has to do with sys.path at all...

IMO that single boolean (ENABLE_USER_SITE=False) should make anything in the user site (~/.local/lib/pythonx.y/site-packages/) invisible to everybody, pip included. Is this understanding incorrect? It almost seems like a problem in site.py: pip's calls to getusersitepackages() shouldn't return an actual path if ENABLE_USER_SITE = False...

[chrahunt]

The documented behavior of PYTHONNOUSERSITE is (from here):

If this is set, Python won’t add the user site-packages directory to sys.path.

I think that managing packages in the user site (or any package-holding directory) is unrelated to this. Pip being influenced by the default Python behavior causes a lot of confusion in general (see #6703, #6558, #5040). If you could explicitly say --system to manage the system packages, then we wouldn't even consider setting this environment variable to get the desired behavior, right? Please correct me if I'm misunderstanding.

[bjmuld]

I don't think you're misunderstanding anything other than what I'm "proposing".
I do agree that explicit pip scheme=user/global/local or pip --local/--global/--user would bring clarity and make use of env variables unnecessary.

I'm only trying to report a perceived bug in which pip sees user packages in spite of PYTHONNOUSERSITE being set.

I'm going to close because I think my problem arises from use of a sitecustomize.py which made modifications to site.USER_BASE and site.USER_SITE but didn't remove paths which has already been erroneously added to sys.path.