conda doesn't correctly uninstalls packages installed by pip

[stas00]

Current Behavior

conda does not remove all files listed in the RECORD metadata file when asked to remove a package. This leaves a corrupt distribution present, which other tools such as pip are unable to process.

Steps to Reproduce

Let's pick a package pynvml as an example:

# cleanup
conda uninstall -y pynvml
pip uninstall -y pynvml

# must install/uninstall things in this exact order
conda install -y -c conda-forge pynvml
pip uninstall -y pynvml
pip install pynvml
conda uninstall -y pynvml

# check: not good:
ls -lt /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/
total 0
-rw-rw-r-- 1 stas stas 0 Nov 12 22:03 REQUESTED

conda didn't delete the file listed in RECORD

This file wasn't in conda-forge build, but was added by pip - it seems conda removes only files that it installed,

So now the site-packages is broken and we can't use pip anymore:

pip install pynvml
Requirement already satisfied: pynvml in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (8.0.4)
WARNING: No metadata found in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/METADATA'

"Requirement already satisfied" is bogus since there is just one REQUESTED file that wasn't cleaned up in the dist dir.

this fails too:

pip uninstall -y pynvml
Found existing installation: pynvml 8.0.4
ERROR: Exception:
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/commands/uninstall.py", line 89, in run
    uninstall_pathset = req.uninstall(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 685, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 535, in from_dist
    for path in uninstallation_paths(dist):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 67, in unique
    for item in fn(*args, **kw):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 85, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1432, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/RECORD'

The only fix at the moment is to manually remove the dir by hand.

I have a package whose build process validates that the conda/pip installs worked correctly and so it happens to hit a similar sequence. But this can easily happen in real life, except install / uninstall commands can be called days apart...

Expected Behavior

conda and pip need to find a way to work better together ;)

I initially thought that it was a bug in pip, and filed a bug report there
pypa/pip#9129
but pip developers indicated that it's a bug in conda instead.

Thank you!

Environment Information

`conda info`

     active environment : main-38
    active env location : /home/stas/anaconda3/envs/main-38
            shell level : 5
       user config file : /home/stas/.condarc
 populated config files :
          conda version : 4.9.2
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages : __cuda=11.1=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/stas/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/stas/anaconda3/pkgs
                          /home/stas/.conda/pkgs
       envs directories : /home/stas/anaconda3/envs
                          /home/stas/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.7.4 Linux/5.4.0-52-generic ubuntu/20.04.1 glibc/2.31
                UID:GID : 1000:1000
             netrc file : /home/stas/.netrc
           offline mode : False

• pip version: 20.2.4

┆Issue is synchronized with this Jira Task

[pradyunsg]

FYI -- we've been getting reports about failures like this since support for the REQUESTED file (from PEP 376) was added into pip.

It was only when OP filed an issue detailing their entire sequence of reproduction steps (cross-linked by GitHub above) that we got to know what was likely the root cause of that issue. :)

[stas00]

It took quite some trial-n-error to figure out that sequence. Thank you for the appreciation note, @pradyunsg.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!