7.0.3 Windows - problem with directory cleanup on upgrade

[techtonik]

>py -m pip install -U cffi
Collecting cffi
  Using cached cffi-1.1.0-cp27-none-win32.whl
Requirement already up-to-date: pycparser in c:\python27\lib\site-packages (from cffi)
Installing collected packages: cffi
  Found existing installation: cffi 1.0.2
    Uninstalling cffi-1.0.2:
      Successfully uninstalled cffi-1.0.2
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 297, in run
    root=options.root_path,
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 633, in install
    requirement.commit_uninstall()
  File "C:\Python27\lib\site-packages\pip\req\req_install.py", line 734, in commit_uninstall
    self.uninstalled.commit()
  File "C:\Python27\lib\site-packages\pip\req\req_uninstall.py", line 153, in commit
    rmtree(self.save_dir)
  File "C:\Python27\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\Python27\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\Python27\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\Python27\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 89, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 101, in rmtree_errorhandler
    func(path)
WindowsError: [Error 5] Access is denied: 'c:\\users\\user\\appdata\\local\\temp\\pip-zzlhal-uninstall\\python27\\lib\\site-packages\\_cffi_backend.pyd'

[johnmlhll]

I'm on Mac OSX and looking into this also. I'm stuck on 7.0.3 and to get things like scikit, numpy and scipy, I need to uprade to 7.1.0 apparently. Im on Python 2.7.6 and after following the error messages when I tried to download mysql-python, I was brought to an error message like shown

So, as I understand it, this error is driven by an incompatability bundled with Mac's OS X on Python 2.7.6. Maybe there is something similar going on with Windows as your errors are referring back to Python 2.7. As brew on Mac was what I used to upgade python in the past, I used "brew update" and when brew updated, I went and upgraded python with "brew install --upgrade python" and when complete I checked my versions which are as follows:

Learnt lesson for me to solve this error was to upgrade python to 2.7.9 (I read about a bug in prior versions affecting Pip) or above. Pip will bundle with Python 2.7.10 as Pip 7.1.0.

[jayvdb]

Similar problem when upgrading pip on Appveyor.

https://ci.appveyor.com/project/ogrisel/python-appveyor-demo/build/1.0.127/job/1y2sr88ib7b043i3

pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.0.3-py2.py3-none-any.whl#md5=6950e1d775fea7ea50af690f72589dbd
Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122, in main
    logging_dictConfig({
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 283, in run
    wb = WheelBuilder(
  File "C:\Python27\lib\site-packages\pip\req.py", line 1443, in install
  File "C:\Python27\lib\site-packages\pip\req.py", line 610, in commit_uninstall
  File "C:\Python27\lib\site-packages\pip\req.py", line 1860, in commit
  File "C:\Python27\lib\site-packages\pip\util.py", line 43, in rmtree
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\Python27\lib\site-packages\pip\util.py", line 62, in rmtree_errorhandler
WindowsError: [Error 5] Access is denied: 'c:\\users\\appveyor\\appdata\\local\\temp\\pip-zgpfai-uninstall\\python27\\scripts\\pip.exe'

Storing debug log for failure in C:\Users\appveyor\pip\pip.log
Command exited with code 2

I tried --no-clean , but that doesnt help. --user does workaround this problem.

[darkmorpher]

RC @dstufft

Was patch f355623 supposed to remedy this?

Related:
pyca/cryptography#1771 (comment)
#2859 #2698 & #1299 (comment)
pyca/pyopenssl#33 (comment)

4/23

Updating cffi, I got this:
http://paste.is/15624/61431940/raw/

Updating cryptography :
http://paste.is/14993/14586595/raw

[pfmoore]

Lots of different issues here.

• Access is denied: '... pip.exe' is because you're upgrading pip using the pip.exe wrapper which gets replaced in the upgrade. On Windows you can't replace in-use executables/DLLs. You should use python -m pip install --upgrade pip. IIRC this is noted in the documentation.
• Access is denied: '... _cffi_backend.pyd' is because you're trying to upgrade an in-use DLL (possibly being used in another Python session). Don't do that, Windows doesn't allow it.
• The OSX one is unrelated, and I'm afraid I don't know anything about OSX to comment.

[techtonik]

you're trying to upgrade an in-use DLL (possibly being used in another Python session). Don't do that, Windows doesn't allow it.

Windows allows to rename it and remove later when it is not used. pip could do this.

[pfmoore]

But pip doesn't. And is unlikely to in the foreseeable future.

[techtonik]

@pfmoore you said that.

[pfmoore]

I'm confused. What I said was that Windows doesn't allow you to upgrade an in-use DLL. This is correct. It's possible to work around this by renaming the DLL (so that the process using it can continue doing so) and mark it for deletion later, and in the meantime put a new version of the DLL in place. This gives you a new version, but doesn't upgrade the one the existing process is using.

And there are a large number of technical issues with doing this robustly in pip. So pip does not do this (that's simply a fact). The reason it's unlikely that pip will do so in the foreseeable future is that none of the pip developers feels that the cost of developing a solution is worth the benefits, and nobody outside of the pip development team has come up with a mergeable PR that does this (even if someone did, that's no guarantee it would be applied, as there are other considerations such as maintainability, but without the first step of a usable PR, nothing will happen).

@darkmorpher The commit you refer to is unrelated - it's about dealing with VCSs that mark their control directories as read only. The cryptography issues you refer to seems to be similar, and so the same comments apply (don't upgrade something that's in use). The others are indeed all variations on this debate, which keeps coming up (even though the docs cover not using pip.exe to install itself - we could add a further comment "don't upgrade packages that are in use" but I doubt it would have any impact...)

[techtonik]

And there are a large number of technical issues with doing this robustly in pip. So pip does not do this (that's simply a fact). The reason it's unlikely that pip will do so in the foreseeable future is that none of the pip developers feels that the cost of developing a solution is worth the benefits, and nobody outside of the pip development team has come up with a mergeable PR that does this (even if someone did, that's no guarantee it would be applied, as there are other considerations such as maintainability, but without the first step of a usable PR, nothing will happen).

Can you be specific about there is the major problem in pip from technical point of view?

[dstufft]

Closing this, we've fixed this by not importing any C extensions in the install/upgrade path of pip on Windows.