Directory not empty: 'urllib3' when upgrading pip

[K-Yo]

Description

When trying to upgrade pip from 22.0.4 to 22.2.2 on debian, I get the following error during the upgrade process

ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'

afterwards pip binary (/usr/local/bin/pip) does not exist anymore

Another user online seems to have faced the same issue on alpine (for the same pip versions): https://stackoverflow.com/questions/73084494/pip-upgrading-within-the-python-alpine-image-error-could-not-install-packages

Expected behavior

Upgrade does not fail and I can use pip afterwards

pip version

22.0.4

Python version

3.10.5

OS

Debian

How to Reproduce

Using docker (for example):

FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pip

Output

root@c93e6f957460:/# pip install --no-cache-dir --upgrade pip
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.0.4)
Collecting pip
  Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 185.6 kB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'

root@c93e6f957460:/# pip
bash: /usr/local/bin/pip: No such file or directory

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

The Dockerfile works for me

$ cat Dockerfile 
FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pip

$ docker build .     
[+] Building 42.7s (6/6) FINISHED                                                                                                   
 => [internal] load build definition from Dockerfile                                                                           0.0s
 => => transferring dockerfile: 110B                                                                                           0.0s
 => [internal] load .dockerignore                                                                                              0.0s
 => => transferring context: 2B                                                                                                0.0s
 => [internal] load metadata for docker.io/demisto/python3-deb:3.10.5.31797                                                    3.1s
 => [1/2] FROM docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff56344  24.5s
 => => resolve docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445  0.0s
 => => sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b 11.77MB / 11.77MB                               7.2s
 => => sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445e062 2.84kB / 2.84kB                                 0.0s
 => => sha256:218a5ed3ee2aabb99331c3455f5d2352823aa0ca40a1b3f7537bd20bb3f91761 10.64kB / 10.64kB                               0.0s
 => => sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb 1.08MB / 1.08MB                                 1.4s
 => => sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7 31.38MB / 31.38MB                              22.4s
 => => sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922 233B / 233B                                     1.7s
 => => sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4 3.17MB / 3.17MB                                 6.1s
 => => sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204 283B / 283B                                     6.6s
 => => sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963 684.26kB / 684.26kB                             9.6s
 => => sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81 2.75MB / 2.75MB                                 9.9s
 => => sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad 1.51MB / 1.51MB                                12.8s
 => => sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0 329B / 329B                                    10.4s
 => => sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332 7.54MB / 7.54MB                                16.6s
 => => sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c 5.24kB / 5.24kB                                13.3s
 => => extracting sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7                                      0.8s
 => => extracting sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb                                      0.1s
 => => extracting sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b                                      0.3s
 => => extracting sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922                                      0.0s
 => => extracting sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4                                      0.1s
 => => extracting sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204                                      0.0s
 => => extracting sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963                                      0.0s
 => => extracting sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81                                      0.1s
 => => extracting sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad                                      0.1s
 => => extracting sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0                                      0.0s
 => => extracting sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332                                      0.3s
 => => extracting sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c                                      0.0s
 => [2/2] RUN pip install --upgrade pip                                                                                       14.9s
 => exporting to image                                                                                                         0.1s
 => => exporting layers                                                                                                        0.1s
 => => writing image sha256:1fb2a3e312b19e2750f81a0abba77f735dbb227a586918efc662372275b2ec83                                   0.0s 

$ docker run $(docker images -q | head -n1) pip --version
pip 22.2.2 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)

[abn]

@uranusjr came here because I have a bunch of GHA builds failing with similar issues. I have created a reproducer at https://github.com/abn/reproducer-pip-11368

See https://github.com/abn/reproducer-pip-11368/runs/8042452418?check_suite_focus=true#step:3:71 for the specific failure. This has been happening on multiple projects.

The weird part is that the builds work fine locally. Additional point of note here is that in my case the builds are done using buildah not docker.

[abn]

Verbose Log

Created temporary directory: /tmp/pip-unpack-suo34b4g
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Created temporary directory: /tmp/pip-uninstall-nqm3v44c
      Removing file or directory /usr/local/bin/pip
      Removing file or directory /usr/local/bin/pip3
      Removing file or directory /usr/local/bin/pip3.10
      Created temporary directory: /usr/local/lib/python3.10/site-packages/~ip-22.0.4.dist-info
      Removing file or directory /usr/local/lib/python3.10/site-packages/pip-22.0.4.dist-info/
      Created temporary directory: /usr/local/lib/python3.10/site-packages/~ip
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/shutil.py", line 815, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.10/site-packages/pip/' -> '/usr/local/lib/python3.10/site-packages/~ip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 405, in run
    installed = install_given_reqs(
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 68, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 638, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_uninstall.py", line 369, in remove
    moved.stash(path)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_uninstall.py", line 267, in stash
    renames(path, new_path)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 305, in renames
    shutil.move(old, new)
  File "/usr/local/lib/python3.10/shutil.py", line 833, in move
    rmtree(src)
  File "/usr/local/lib/python3.10/shutil.py", line 724, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/lib/python3.10/shutil.py", line 657, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.10/shutil.py", line 663, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/local/lib/python3.10/shutil.py", line 661, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'urllib3'
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
There was an error checking the latest version of pip
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/self_outdated_check.py", line 148, in pip_self_version_check
    best_candidate = finder.find_best_candidate("pip").best_candidate
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 868, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 809, in find_all_candidates
    page_candidates = list(page_candidates_it)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
    yield from self._candidates_from_page(self._link)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 769, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 566, in fetch_page
    return _get_html_page(location, session=self.session)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 471, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 123, in _get_html_response
    resp = session.get(
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 542, in get
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 454, in request
    return super().request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 529, in request
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 645, in send
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/cachecontrol/adapter.py", line 57, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/adapters.py", line 417, in send
  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/adapters.py", line 228, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem

[abn]

Looks like issue might have to do with pip running inside rootless containers using fuse-overlays. See containers/fuse-overlayfs#332 (comment).

That issue also references #103.

[abn]

I can confirm that the workaround specified in the fuse-overlayfs issue resolves this for me. In my case I added the following to the buildah action.

        with:
          ...
          extra-args: |
            --storage-opt=overlay.mount_program=

[uranusjr]

pip just uses standard Python functions to interact with the filesystem. So does it imply this is a CPython bug?

[abn]

Welp, I missed the question.

Just checked the build without the mitigation, and it seems it just "works" now (see https://github.com/abn/reproducer-pip-11368/actions/runs/4957985233/jobs/8870312079). No idea if the root cause somehow got fixed somewhere else or if GH just updated their environments.

But for now, I cannot seem to reproduce the issue anymore.

[uranusjr]

Combining with how the Dockerfile seems to reproduce only on certain environments (the OP’s but not mine), I wonder whether it may actually be a bug in the container layer (GHA also uses containers for their jobs) that is since fixed. In any case I’ll close this for now since there’s nothing further we can do. A new issue can always be opened if a similar issue pops up again.