pip install -U pip in venv suggests adding --user flag on Windows

[daytonb]

Environment

• pip version: 19.2.3 and 20.1
• Python version: Python 3.8.2
• OS: Windows 10

Python 3.8.2 is a user-only install from the Python.org 64-bit Windows .exe installer, installed to %LOCALAPPDATA%\Programs\Python\Python38.

Description
When I run pip install -U pip in a virtual environment (created builtin venv module), I get an error message saying

ERROR: Could not install packages due to an EnvironmentError.
Consider using the `--user` option or check the permissions.

It actually does succeed in upgrading pip in the virtual environment despite the error message. It appears that what it fails to do is clean up the %LOCALAPPDATA%\Temp\pip-uninstall-xxxxxxxx\ temporary folder.

I do not get this error message when I want to upgrade the site-wide pip (py -m pip install -U pip) or the user pip (py -m pip install --user -U pip).

I also do not get an error message like this in virtual environments on Linux.

Expected behavior
I expect the virtual environment's pip to be able to upgrade itself without producing any error messages or leaving behind any temporary files.

How to Reproduce

1. Install Python 3.8.2 using the python-3.8.2-amd64.exe from Python.org for just the current user
   • I doubt the exact version of Python or that it's installed for just the current user is important
2. Create a virtual environment by running py -3.8 -m venv testenv
3. Run .\testenv\Scripts\pip.exe --version. Reports pip 19.2.3 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
4. Check that there are no pip-* folders in %LOCALAPPDATA%\Temp\
5. Run .\testenv\Scripts\pip.exe install -U pip. Error message appears
6. Run .\testenv\Scripts\pip.exe --version. Reports pip 20.1 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
7. Find that there is now a pip-uninstall-* folder in %LOCALAPPDATA%\Temp\

Output

Here is the output of running the above commands (omitting the installation of Python):

PS C:\Users\myusername\Desktop> py -3.8 -m venv testenv
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe --version
pip 19.2.3 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
PS C:\Users\myusername\Desktop> Get-ChildItem C:\Users\myusername\AppData\Local\Temp\pip-*
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-uninstall-bnuzi1af\\pip.exe'
Consider using the `--user` option or check the permissions.

PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe --version
pip 20.1 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
PS C:\Users\myusername\Desktop> Get-ChildItem C:\Users\myusername\AppData\Local\Temp\pip-*


    Directory: C:\Users\myusername\AppData\Local\Temp


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        5/15/2020  12:04 PM                pip-uninstall-bnuzi1af

For good measure I'll add output here of re-installing pip 19.2.3 into the virtualenv , then reproducer steps just adding the verbose flag to pip install -U pip this time.
I'll also add ellipses to cut out some of the "Found link https://files.pythonhosted.org..." lines

PS C:\Users\myusername\Desktop> Get-ChildItem C:\Users\myusername\AppData\Local\Temp\pip-* | Remove-Item -Recurse
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe install pip==19.2.3
Collecting pip==19.2.3
  Using cached pip-19.2.3-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1
    Uninstalling pip-20.1:
      Successfully uninstalled pip-20.1
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-uninstall-kb_5x7j4\\pip.exe'
Consider using the `--user` option or check the permissions.

PS C:\Users\myusername\Desktop> Get-ChildItem C:\Users\myusername\AppData\Local\Temp\pip-* | Remove-Item -Recurse
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe --version
pip 19.2.3 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe install -U -v pip
Created temporary directory: C:\Users\myusername\AppData\Local\Temp\pip-ephem-wheel-cache-4muv97xc
Created temporary directory: C:\Users\myusername\AppData\Local\Temp\pip-req-tracker-1f9cv031
Created requirements tracker 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-req-tracker-1f9cv031'
Created temporary directory: C:\Users\myusername\AppData\Local\Temp\pip-install-qm960hgk
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
Analyzing links from page https://pypi.org/simple/pip/
  Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2
  .
  .
  .
  Found link https://files.pythonhosted.org/packages/4f/7d/e53bc80667378125a9e07d4929a61b0bd7128a1129dbe6f07bb3228652a3/pip-1.5.tar.gz#sha256=25f81d1a0e55d3b1709818dd57fdfb954b028f229f09bd69cb0bc80a8e03e048 (from https://pypi.org/simple/pip/), version: 1.5
  Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  Found link https://files.pythonhosted.org/packages/44/5d/1dca53b5de6d287e7eb99bd174bb022eb6cb0d6ca6e19ca6b16655dde8c2/pip-1.5.1-py2.py3-none-any.whl#sha256=00960db3b0b8724dd37fe37cfb9c72ecb8f59fab9db7d17c5c1e89a1adab49ce (from https://pypi.org/simple/pip/), version: 1.5.1
  .
  .
  .
  Found link https://files.pythonhosted.org/packages/d1/05/059c78cd5d740d2299266ffa15514dad6692d4694df571bf168e2cdd98fb/pip-20.1.tar.gz#sha256=572c0f25eca7c87217b21f6945b7192744103b18f4e4b16b8a83b227a811e192 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1
Given no hashes to check 139 links for project 'pip': discarding no candidates
Using version 20.1 (newest of versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1)
Collecting pip
  1 location(s) to search for versions of pip:
  * https://pypi.org/simple/pip/
  Getting page https://pypi.org/simple/pip/
  Found index url https://pypi.org/simple
  Looking up "https://pypi.org/simple/pip/" in the cache
  Request header has "max_age" as 0, cache bypassed
  https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/pip/
    Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2
    .
    .
    .
    Found link https://files.pythonhosted.org/packages/d1/05/059c78cd5d740d2299266ffa15514dad6692d4694df571bf168e2cdd98fb/pip-20.1.tar.gz#sha256=572c0f25eca7c87217b21f6945b7192744103b18f4e4b16b8a83b227a811e192 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1
  Given no hashes to check 139 links for project 'pip': discarding no candidates
  Using version 20.1 (newest of versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1)
  Created temporary directory: C:\Users\myusername\AppData\Local\Temp\pip-unpack-1wrw6reh
  Looking up "https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl" in the cache
  Current age based on date: 687913
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 687913
  Using cached https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl
  Downloading from URL https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl#sha256=4fdc7fd2db7636777d28d2e1432e2876e30c2b790d461f135716577f73104369 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*)
  Added pip from https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl#sha256=4fdc7fd2db7636777d28d2e1432e2876e30c2b790d461f135716577f73104369 to build tracker 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-req-tracker-1f9cv031'
  Removed pip from https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl#sha256=4fdc7fd2db7636777d28d2e1432e2876e30c2b790d461f135716577f73104369 from build tracker 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-req-tracker-1f9cv031'
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Created temporary directory: C:\Users\myusername\Desktop\testenv\Lib\site-packages\~ip-19.2.3.dist-info
      Removing file or directory c:\users\myusername\desktop\testenv\lib\site-packages\pip-19.2.3.dist-info\
      Created temporary directory: C:\Users\myusername\Desktop\testenv\Lib\site-packages\~ip
      Removing file or directory c:\users\myusername\desktop\testenv\lib\site-packages\pip\
      Created temporary directory: C:\Users\myusername\AppData\Local\Temp\pip-uninstall-0_ezw100
      Removing file or directory c:\users\myusername\desktop\testenv\scripts\pip.exe
      Removing file or directory c:\users\myusername\desktop\testenv\scripts\pip3.8.exe
      Removing file or directory c:\users\myusername\desktop\testenv\scripts\pip3.exe
      Successfully uninstalled pip-19.2.3

ERROR: Could not install packages due to an EnvironmentError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
  File "C:\Users\myusername\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-uninstall-0_ezw100\\pip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\commands\install.py", line 398, in run
    installed = install_given_reqs(
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\req\__init__.py", line 75, in install_given_reqs
    requirement.should_reinstall and
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 444, in commit
    for pth in self.pth.values():
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 284, in commit
    return new_path
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 83, in cleanup
    try:
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_vendor\six.py", line 693, in reraise
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\utils\misc.py", line 165, in rmtree
    This function should never error out. Also, this function is mainly needed
  File "C:\Users\myusername\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\myusername\AppData\Local\Programs\Python\Python38\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\myusername\desktop\testenv\lib\site-packages\pip\_internal\utils\misc.py", line 178, in rmtree_errorhandler
    if PY2:
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-uninstall-0_ezw100\\pip.exe'
Cleaning up...
Removed build tracker 'C:\\Users\\myusername\\AppData\\Local\\Temp\\pip-req-tracker-1f9cv031'
PS C:\Users\myusername\Desktop> .\testenv\Scripts\pip.exe --version
pip 20.1 from c:\users\myusername\desktop\testenv\lib\site-packages\pip (python 3.8)
PS C:\Users\myusername\Desktop> Get-ChildItem C:\Users\myusername\AppData\Local\Temp\pip-*


    Directory: C:\Users\myusername\AppData\Local\Temp


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        5/15/2020  12:08 PM                pip-uninstall-0_ezw100

[djhoese]

I'm running in to this too on an Azure Windows job:

$ pip install -U numpy Cython pip setuptools
Collecting numpy
  Downloading numpy-1.18.4-cp36-cp36m-win32.whl (10.8 MB)
Collecting Cython
  Downloading Cython-0.29.18-cp36-cp36m-win32.whl (1.5 MB)
Collecting pip
  Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
  Downloading setuptools-46.4.0-py3-none-any.whl (583 kB)
Installing collected packages: numpy, Cython, pip, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.1
    Uninstalling pip-20.1:
      Successfully uninstalled pip-20.1
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\pip-uninstall-o_mcmui8\\pip.exe'
Consider using the `--user` option or check the permissions.

Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\hostedtoolcache\windows\Python\3.8.2\x64\Scripts\cibuildwheel.exe\__main__.py", line 7, in <module>
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\site-packages\cibuildwheel\__main__.py", line 226, in main
    cibuildwheel.windows.build(build_options)
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\site-packages\cibuildwheel\windows.py", line 167, in build
    shell([before_build_prepared], env=env)
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\site-packages\cibuildwheel\windows.py", line 25, in shell
    return subprocess.check_call(' '.join(args), env=env, cwd=cwd, shell=True)
  File "c:\hostedtoolcache\windows\python\3.8.2\x64\lib\subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'pip install -U numpy Cython pip setuptools' returned non-zero exit status 1.

I'll try playing with different ways of calling this with different versions and see what happens.

[djhoese]

Setting my code to install pip != 20.1.1 result in no update of pip (already had 20.1.0) and it doesn't fail. Based on the release notes and the relevant PRs this is probably related to #7555 and #8221. @pradyunsg I see you made #8221, any ideas what is going on here? Specify --user? Based on the nature of the PR/issue I'm a little scared of what the options are. 😨

[daytonb]

Specifying --user will result in updating the user-wide pip install, not the pip in the virtual environment. I did check that. I also remember seeing a github ticket about what behavior --user should have when using pip inside a virtual environment. I believe the consensus was that it should install a package to the user site-packages not to the environment.

I could be wrong, but given that pip is using a new wheel file for the new pip version, I don't believe this issue is related to out of tree builds (#7555 and #8221).

[pradyunsg]

@djhoese it does seem like what you're hitting is a bug, albeit a different one.

On Windows, an executable cannot be deleted while it is running. However, there is supposed to be code in pip, that is supposed to protect against hitting the issue of "try to delete pip.exe while running on pip.exe" on Windows, and suggest using python -m pip ... when it detects the user is trying to do this.

I suggest changing (or getting the project you're using to make a change) to use python.exe -m pip instead of pip, in case, it might try to modify pip.

Additionally, could you please file a new issue describing how to reproduce this issue?

[djhoese]

@pradyunsg Thanks. You're right the python -m pip install fixed it it seems. As for filing an issue, I have no idea how to reproduce this without a complex build system. I ran in to this in the wheel building process for my Cython-based project that uses the cibuildwheel project to build the wheels. Cibuildwheel is being run on Azure environments where it then starts up docker containers (the official PyPA ones for wheel building I think). If I run into this again in a different environment I'll make sure to file an issue.