Undeleted temporary directories in site-packages affect subsequent installs

[ExplodingCabbage]

Environment

• pip version: 19.3.1
• Python version: 3.7
• OS: macOS 10.14.6

Description
The presence of ~-prefixed temporary directories left in site-packages affects the output (and, I suspect, the behaviour?) of subsequent pip install invocations.

Expected behavior
~-prefixed temporary directories should not have any effect on how subsequent installations are handled.

How to Reproduce

1. Get yourself into a state where you've got a ~-prefixed temporary directory in your site-packages folder. (I'm not entirely sure how to deliberately cause this to happen.)
2. Run a pip installation that will cause you to install a version

Output

Here's output from a recent attempt of mine to install an earlier version of flake8 to that which I'd had installed previously:

$ sudo pip3 install flake8==3.6
WARNING: The directory '/Users/markamery/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/Users/markamery/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting flake8==3.6
  Downloading https://files.pythonhosted.org/packages/34/a6/49e2849a0e5464e1b5d621f63bc8453066f0f367bb3b744a33fca0bc1ddd/flake8-3.6.0-py2.py3-none-any.whl (68kB)
     |████████████████████████████████| 71kB 4.4MB/s 
Collecting pycodestyle<2.5.0,>=2.4.0
  Downloading https://files.pythonhosted.org/packages/e5/c6/ce130213489969aa58610042dff1d908c25c731c9575af6935c2dfad03aa/pycodestyle-2.4.0-py2.py3-none-any.whl (62kB)
     |████████████████████████████████| 71kB 16.6MB/s 
Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/site-packages (from flake8==3.6) (0.6.1)
Collecting pyflakes<2.1.0,>=2.0.0
  Downloading https://files.pythonhosted.org/packages/44/98/af7a72c9a543b1487d92813c648cb9b9adfbc96faef5455d60f4439aa99b/pyflakes-2.0.0-py2.py3-none-any.whl (53kB)
     |████████████████████████████████| 61kB 32.6MB/s 
Requirement already satisfied: setuptools>=30 in /usr/local/lib/python3.7/site-packages (from flake8==3.6) (41.0.1)
ERROR: -lake8 3.7.7 has requirement pycodestyle<2.6.0,>=2.5.0, but you'll have pycodestyle 2.4.0 which is incompatible.
ERROR: -lake8 3.7.7 has requirement pyflakes<2.2.0,>=2.1.0, but you'll have pyflakes 2.0.0 which is incompatible.
Installing collected packages: pycodestyle, pyflakes, flake8
  Found existing installation: pycodestyle 2.5.0
    Uninstalling pycodestyle-2.5.0:
      Successfully uninstalled pycodestyle-2.5.0
  Found existing installation: pyflakes 2.1.1
    Uninstalling pyflakes-2.1.1:
      Successfully uninstalled pyflakes-2.1.1
  Found existing installation: flake8 3.7.9
    Uninstalling flake8-3.7.9:
      Successfully uninstalled flake8-3.7.9

Successfully installed flake8-3.6.0 pycodestyle-2.4.0 pyflakes-2.0.0

Note in particular the lines:

ERROR: -lake8 3.7.7 has requirement pycodestyle<2.6.0,>=2.5.0, but you'll have pycodestyle 2.4.0 which is incompatible.
ERROR: -lake8 3.7.7 has requirement pyflakes<2.2.0,>=2.1.0, but you'll have pyflakes 2.0.0 which is incompatible.

These should not be here.

Rerunning the install after doing rm -rf /usr/local/lib/python3.7/site-packages/~* fixed the error.

I think (but didn't test rigorously enough to be certain, and don't know enough about how Pip handles version constraints across multiple package's dependencies to immediately guess) that these temporary folders are not merely affecting output but the actual selection of versions of dependencies, since I figure Pip is probably also attempting to honour the dependency version constraints of the "-lake8" package above.

The underlying issue here is the same as in #7269; Pip should not regard ~-prefixed temp folders in site-packages as packages, but seems to. I don't remember ever encountering this issue until a few weeks ago, so wonder if there's been a bug in the parsing of site-packages contents introduced in the past few months. But I know basically nothing about Pip's internals, so I'll defer to those who do.

[chrahunt]

These temporary directories should only appear when a pip uninstall is interrupted, that could have caused the sudden appearance.

We could take two approaches:

1. as mentioned in pip3 freeze output still confusing when stray AdjacentTempDirectorys are present #7269, pip could avoid putting these files in the site-packages directory in the first place
2. in general, provide better information on the errors and possible remediations available to fix a corrupted site-packages directory