pip install --upgrade stalls at the "uninstalling" phase

[wildlyinaccurate]

Environment

• pip version: 19.0
• Python version: 2.7.15rc1
• OS: Ubuntu 18.04.1 LTS (Kernel: 4.15.0-1031-aws)

Description

Running pip install --upgrade for any package that needs upgrading will cause pip to stall at the "Uninstalling" stage.

Expected behavior

Packages should be upgraded.

How to Reproduce

1. On a system with cryptography-2.4.2 installed...
2. Run pip install --upgrade cryptography
3. Pip gets as far as Uninstalling cryptography-2.4.2 but doesn't go any further. -vvv does not provide any extra output.

Output

  Using version 2.5 (newest of versions: 0.1, 0.2, 0.2.1, 0.2.2, 0.3, 0.4, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.9, 0.9.1, 0.9.2, 0.9.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.6, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1, 1.8.2, 1.9, 2.0, 2.0.1, 2.0.2, 2.0.3, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2, 2.2.1, 2.2.2, 2.3, 2.3.1, 2.4, 2.4.1, 2.4.2, 2.5)
  Created temporary directory: /tmp/pip-unpack-RPH06y
  Looking up "https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl" in the cache
  Current age based on date: 5543
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 5543
  Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl
  Downloading from URL https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*)
  Added cryptography from https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 to build tracker '/tmp/pip-req-tracker-gljwal'
  Removed cryptography from https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 from build tracker '/tmp/pip-req-tracker-gljwal'
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in /usr/lib/python2.7/dist-packages (from cryptography) (0.24.0)
Requirement already satisfied, skipping upgrade: enum34; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography) (1.1.6)
Requirement already satisfied, skipping upgrade: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography) (1.0.22)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/local/lib/python2.7/dist-packages (from cryptography) (1.11.5)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/lib/python2.7/dist-packages (from cryptography) (1.11.0)
Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi!=1.11.3,>=1.8->cryptography) (2.19)
Installing collected packages: cryptography
  Found existing installation: cryptography 2.4.2
    Uninstalling cryptography-2.4.2:

There is more output but it's mostly "Found link" and "Skipping link". Let me know if you need the full output.

[denis-bz]

@wildlyinaccurate, when you interrupt the upgrade with ctl-C, what do you see in your site directory ?
I see e.g. /Library/Python/2.7/site-packages/-onfigparser-3.5.1.dist-info/, config -> -onfig ?!
However uninstall works fine.
(pip catches / turns off the stacktrace on so I can't tell where it's looping.)
My versions: Python 2.7, pip 19.0.0, macos 10.10

[YannickJadoul]

We're having the same problem over at the cibuildwheel project when running this on Travis: pypa/cibuildwheel#122

However, I think I've figured out what causes this:

It seems c7ae06c added a new way of generating temporary directories to copy uninstalled packages so that they can be rolled back if the new package's installation failed.

Notice the _generate_names method, generating potentially len(LEADING_CHARS)**len(name) (i.e., almost always a ridiculously enormous number) of temporary folder names. When the parent folder cannot be written to (because of permission issues, for example), the creation of these temporary folders will fail and the loop will go on checking all possibilities and never managing to find one.

See https://travis-ci.org/YannickJadoul/cibuildwheel/jobs/485146671 and https://api.travis-ci.org/v3/job/485146671/log.txt for a log of the installation and the permissions: user travis is a member of the admin group, so the group permissions are what you're looking for.

Not sure what's the best way to fix this, but I'm quite confident this is the problem.

I see e.g. /Library/Python/2.7/site-packages/-onfigparser-3.5.1.dist-info/, config -> -onfig ?!

This new method of generating temporary folder names also explains the weird - names. Probably leftovers from a failed/interrupted installations/updates? Cfr. #6194, as well.

[cjerdonek]

@zooba This issue seems to be related to a PR you worked on before (see the previous comment).

[wildlyinaccurate]

It's possible that my original issue was actually a permissions mismatch -- the original modules were installed as root, and running pip install with sudo proceeds as expected with no issues.

[cjerdonek]

Without knowing much, it seems like the right fix might be to distinguish between permissions issues and the path already existing when handling OSError in the code you linked to. Errors other than the file existing should bubble up.

[YannickJadoul]

@wildlyinaccurate I think the confusion also stems from the fact that different permissions are involved. On the one hand, a package's folder can be unremovable (e.g., .../my-package), but this issue is linked to a new folder not being able to be created next to the installed package, in the parent folder (e.g., -y-package inside site-packages). That's why I never managed to reproduce the error locally.

EDIT: Seems it was too late yesterday to think clearly, and I think got the wrong folders names. site-packages is actually writable in my case, but its parent folder (python3.4 in my logs) is not. So when -ite-packages, ~ite-packages, .ite-packages, +ite-packages, ..., --te-packages, ... are tried to be created, this fails an exponential number of times.

[zooba]

@cjerdonek That sounds like a reasonable fix. I'll take a quick look