windows in-place pip upgrades #1299

Open
qwcode opened this Issue Nov 5, 2013 · 70 comments

Projects

None yet
@qwcode
Contributor
qwcode commented Nov 5, 2013

not sure of the status of pip install --ugrade pip on windows.

need this for PEP453

@pfmoore ?

see #188 for an old discussion on this.

@pfmoore
Member
pfmoore commented Nov 5, 2013

Use python -m pip install --upgrade pip. I think @vsajip was considering some form of magic in the exe wrappers to mean that self-replacement of the exe could be made to work. But I don't know if he did anything in the version of distlib we're using. I'm not actually sure it's something that could be done without help from the application (pip). And I'm pretty sure no code has been added to pip for this.

Personally, I consider this to be a minor issue - telling people to use `python -m pip`` seems to me to be a perfectly acceptable answer, given that the issue is fundamental to how Windows handles deletion of open files.

Ultimately it's going to be a question of how much complexity (with the consequent maintenance cost) and risk we want to take on in order to solve this.

@dstufft
Member
dstufft commented Nov 5, 2013

Personally i'm ok with pip install --upgrade pip printing a message to use python -m pip install --upgrade pip. I just don't think we should throw an error about open file handles or whatever the exception is.

@pfmoore
Member
pfmoore commented Nov 5, 2013

I was just about to say the same thing.

Here's a question - would you want that message to only appear on Windows, or would it be reasonable to say the same on Unix (for a consistent user experience). Personally, I'd like to be consistent across platforms (because it's easier to document that way).

Also, we should trap any command that attempts to (re-)install pip - pip install -I pip is another one, for example.

@dstufft
Member
dstufft commented Nov 5, 2013

Personally I think it should be windows only. I view it as a work around for the way the Windows platform works, and something that's a target to eventually remove for something nicer down the road.

@pfmoore
Member
pfmoore commented Nov 5, 2013

I knew you were going to say that :-(

Frankly, I don't believe we will ever get "something nicer", basically because we don't have enough Windows expertise to develop and maintain it. I know of no Windows software with a satisfactory self-replacement of executables solution. I certainly don't have an answer, and I wouldn't expect any of the Unix-based developers to be able to help. Even if we did get something, it's likely to be complex and so a probable source of bit rot and obscure bugs.

Making a solution that works on all platforms the "official answer" seems to me to avoid sending confusing messages to the user. Back it up with a warning as described and we have a robust solution, even if it's not quite as perfect as getting pip install -U pip to work.

Please excuse a certain level of mild grumpiness in this reply, caused by a feeling that it's always the Windows platform that has to be the one to compromise - I don't mean to make an issue out of this. In practice, I'll go with whatever the consensus is. I don't think it's a big deal.

@dstufft
Member
dstufft commented Nov 5, 2013

FWIW my goal after PEP453 is done is to get our own testing infrastructure stood up which includes Windows, so hopefully that'll solve some of the windows bitrot problems.

I can understand the grumpyness though :) If we were designing this system brand new I'd be a lot more amendable to that but I don't think it's user friendly to break pip install -U pip for everything non Windows in order to maintain consistency. If this was a brand new thing then there's no (or less at least) user epectation :]

@pfmoore
Member
pfmoore commented Nov 5, 2013

OK, that sounds fair. You're right that historical usage is more important here.

I actually have no feel at all for what the user experience on Unix is, so I'm probably just as guilty of one-sided thinking as anyone else. I must try to see how using Python on Linux feels - is there a good "Python on Linux for beginners" resource that shows how to set up a Python (3) environment, get the various tools set up, etc? [One thing I always find doesn't get covered, is how to make "python" run Python 3 by default, how to make "virtualenv" build Python 3 virtualenvs by default, all that sort of thing - it seems to me that Python 3 is much more second-class on Unix than on Windows, oddly enough.]

@dstufft
Member
dstufft commented Nov 5, 2013

Typically you don't want to make python point to python3 on *nix because the system uses Python and not all of their libraries are going to be ported to Python3. However there are some Linux distros that have Python3 as a default now... I think Arch? And Maybe the next Fedora?

@pfmoore
Member
pfmoore commented Nov 5, 2013

Hmm, I'm not talking at a system level, but really you can't choose what to use for Python in your personal account? That definitely makes Python 3 second-class, to my way of thinking... But it does explain why Unix users don't seem to find using things like "pip3" or "pip3.3" as ugly as sin :-)

(Anyway, we're way off-topic here, so never mind).

@dstufft
Member
dstufft commented Nov 5, 2013

You can setup your shell so it has ~/bin/ before the system bin and add a symlink there from python to python3, but yea offtopic. I'm happy to help though if you're actually interested in getting things set up :] IRC or Email.

@vsajip
Contributor
vsajip commented Nov 5, 2013

My work which I believe Paul was referring to was just to implement the 'deleteme' dance for replacing files in use (i.e. exe wrappers) - that's in distlib Python code, not in the .exe wrappers. This is in the distlib.scripts module and should be exercised whenever that module is used to create a script.

@dstufft
Member
dstufft commented Nov 5, 2013

So perhaps this is already solved for Wheels? And it's a setuptools problem elsewise?

@pfmoore
Member
pfmoore commented Nov 6, 2013

Sounds possible. I'll do some tests today, if I can.

@pfmoore
Member
pfmoore commented Nov 6, 2013

Sadly not, at least with "pip install -I pip". (It's hard to test an actual upgrade, as only the absolute latest version of pip has script wrapper support):

>pip install -f $x -I pip --use-wheel --no-index --pre
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking pip
Installing collected packages: pip
  Found existing installation: pip 1.5.dev1
    Uninstalling pip:
      Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 365, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\basecommand.py", line 121, in main
    status = self.run(options, args)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\commands\install.py", line 258, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1330, in install
    requirement.commit_uninstall()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 557, in commit_uninstall
    self.uninstalled.commit()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1743, in commit
    rmtree(self.save_dir)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 42, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 460, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 367, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 61, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

Storing complete log in C:\Users\uk03306\pip\pip.log
@vsajip
Contributor
vsajip commented Nov 6, 2013

So pip's not using distlib.scripts to install scripts ... I guess it doesn't fit in the pip workflow.

@pfmoore
Member
pfmoore commented Nov 6, 2013

I suspect it's because we do an uninstall then an install, rather than an install overwriting what's there (for obvious reasons).

@agronholm
Contributor

Windows doesn't allow overwriting or deleting open exe files, but it does allow renaming/moving them.

@rgbkrk
rgbkrk commented Nov 26, 2013

Well, with some tomfoolery with the Win32 API you can close the handle to pip.exe much like you can when using Sysinternal's process explorer.

@pfmoore
Member
pfmoore commented Nov 26, 2013

I don't think you can close all handles to the currently running exe. But if you have working code as an example, I'd love to see it, as this has long been a problem that people have debated and the consensus has always been that it's not possible (except by tricks like "rename and delete later"). Working code that does this would be extremely useful to the community.

@techtonik
Contributor

Run into the same issue.

As a quick hack, can pip.exe detect that it is being upgraded prior to execution and restart itself with python -m pip {{ sys.argv }}?

@dstufft
Member
dstufft commented Mar 25, 2014

I don't think that actually works.

But maybe a stupid thought, afaik the .exe doesn't need to be updated, can we just special case pip.exe and leave it alone? Would possibly leave a pip.exe laying around after an install though.

@pfmoore
Member
pfmoore commented Mar 25, 2014

@dstufft correct, "restarting yourself" isn't as simple as @techtonik makes it sound on Windows, where there's no equivalent of the Unix exec call.

Technically, pip.exe may need updating if the entry point changes or if the distlib wrapper code is changed. That's not common, but it can happen (early on, the distlib code changed a fair bit till things settled down). What it may be possible to do would be to write the new pip.exe to a temporary location, then if it's identical to the current one, skip it. You wouldn't avoid errors when it did need updating, but in the 99% of cases when it didn't, you'd be fine.

BTW, we're going round in circles here. According to Vinay's earlier comments, this is solved in distlib (which is where I think it should be solved) but pip doesn't use the API in a way that allows it to work. If we could use the distlib solution, then great. Otherwise, we need someone to write a pip-specific solution (and I remain skeptical that this can be done in a robust, maintainable manner that isn't far more complex than the issue warrants).

Suggestion:

  1. Someone add a section to the pip docs at http://www.pip-installer.org/en/latest/installing.html#install-or-upgrade-pip noting that the python -m pip workaround is needed on Windows when upgrading pip with itself. I might get a chance to do this at some point, but PRs from people who have contributed to this issue would be gratefully accepted :-).
  2. If anyone has a viable solution, and is prepared to implement it, they should raise a PR and we can review actual code rather than proposed approaches.

I think we're at the stage where actual working code is the only thing that will move this issue forward.

@techtonik techtonik added a commit to techtonik/pip that referenced this issue Mar 25, 2014
@techtonik techtonik Pseudocode to fix issue #1299
I am not sure how to detect that pip is run as pip.exe
090f9ac
@techtonik
Contributor

I am -1 on bloating the docs without trying other solutions.

@qwcode
Contributor
qwcode commented Mar 25, 2014

I'm +1 on documenting what works right now.

@techtonik
Contributor

Even with doc comment it is still workaround - not a solution, and you've got command prompt for Windows wrong.

@qwcode qwcode removed the PEP453 label Mar 26, 2014
@mikofski

Maybe this is crazy but I always just do this:

$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

of course this requires one to have libcurl, but there are windows binaries available here.

@zaducu
zaducu commented May 10, 2014

I've tried to install it on a Mac but I received this

Alexs-MacBook-Air:Pipmodules aleand$ pip install -U pip
-bash: pip: command not found
Alexs-MacBook-Air:Pipmodules aleand$ python get-pip.py
Downloading/unpacking pip
Downloading pip-1.5.5-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
Cleaning up...
Exception:
Traceback (most recent call last):
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/req.py", line 1435, in install
requirement.install(install_options, global_options, _args, *_kwargs)
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/req.py", line 671, in install
self.move_wheel_files(self.source_dir, root=root)
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/req.py", line 901, in move_wheel_files
pycompile=self.pycompile,
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/wheel.py", line 215, in move_wheel_files
clobber(source, lib_dir, True)
File "/var/folders/8g/b9q8snrd5ln_d5hqnwh2_86r0000gn/T/tmpFTGrC3/pip.zip/pip/wheel.py", line 205, in clobber
os.makedirs(destdir)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/pip'

Storing debug log for failure in /Users/aleand/Library/Logs/pip.log

@mikofski

@zaducu, I can tell from your traceback that you are trying to update the pip version that is in the version of Python that came pre-installed with your mac. Since you are updating the pre-installed version, you will need to elevate your credentials by using sudo:

$ sudo curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

To use sudo you may have to add yourself to /etc/sudousers. Google it and you will find numerous links showing how to do this.

Mac is smart, it keeps your packages in /Library/Python instead of corrupting your /System/Frameworks/Python.framework site-pacakges folder. Yay! But to access this folder you must be an administrator with root privileges, hence sudo.

An alternative to consider is to keep your personal packages and modules separated by using virtualenv and/or a .local folder. To install packages into a local folder use distutils instead of pip and follow the directions on installing python modules.

Other alternatives to using the pre-installed Python are to use official Python, homebrew, MacPorts, Fink or build it yourself.

For official Python, you will still need sudo but it will install most items in /System/Frameworks not /System/Library/Frameworks For Homebrew, you should not need sudo. Can't speak for MacPorts or Fink.

@Mikaela Mikaela referenced this issue in Mikaela/shell-things Sep 4, 2014
Closed

pip aliases #44

@Mikaela Mikaela pushed a commit to Mkaysi/limnoria-doc that referenced this issue Sep 4, 2014
Mikaela Suomalainen use/install.rst: use python -m pip instead of pip
Windows pip cannot update itself so Windows instructions need a little fixing and having both formats could cause confusion. Specifying python also can make which pip uses as more clear as pypy doesn't seem to get it's own binary for pip and no binary links there.

Links:
* http://pip.readthedocs.org/en/latest/installing.html#upgrade-pip
    * http://pip.readthedocs.org/en/latest/installing.html#id11
        * pypa/pip#1299
5d69932
@Mikaela Mikaela referenced this issue in Limnoria/Limnoria-doc Sep 4, 2014
Closed

use/install.rst: use python -m pip instead of pip #33

@aaanders aaanders added a commit to aaanders/pip that referenced this issue Dec 23, 2014
@aaanders aaanders Update installing.rst
Fixes unresolved conflict caused by footnote bump. Windows should point to 6(pypa#1299)
cdfbd7f
@bialix
bialix commented Apr 6, 2015

Still have the same problem on Windows, upgrading from 1.5.4 to 6.0.8:

E:\Distrib\Programming>pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-6.0.8-py2.py3-none-any.whl#md5=41e73fae2c86ba2270ff51c1d86
f7e09
Installing collected packages: pip
Found existing installation: pip 1.5.4
Uninstalling pip:
Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
File "C:\Python\3.3-32\lib\shutil.py", line 371, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Отказано в доступе: 'c:\users\bialix\appdata\local\temp\pip-6i_i7a-uninstall\python\3.3-32\scripts\p
ip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Python\3.3-32\lib\site-packages\pip\basecommand.py", line 122, in main
write_debug_log = check_path_owner(debug_log_path)
File "C:\Python\3.3-32\lib\site-packages\pip\commands\install.py", line 283, in run
build_delete = (not (options.no_clean or options.build_dir))
File "C:\Python\3.3-32\lib\site-packages\pip\req.py", line 1443, in install
File "C:\Python\3.3-32\lib\site-packages\pip\req.py", line 610, in commit_uninstall
File "C:\Python\3.3-32\lib\site-packages\pip\req.py", line 1860, in commit
File "C:\Python\3.3-32\lib\site-packages\pip\util.py", line 43, in rmtree
File "C:\Python\3.3-32\lib\shutil.py", line 478, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python\3.3-32\lib\shutil.py", line 368, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python\3.3-32\lib\shutil.py", line 368, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python\3.3-32\lib\shutil.py", line 368, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python\3.3-32\lib\shutil.py", line 373, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Python\3.3-32\lib\site-packages\pip\util.py", line 62, in rmtree_errorhandler
PermissionError: [WinError 5] Отказано в доступе: 'c:\users\bialix\appdata\local\temp\pip-6i_i7a-uninstall\python\3.3-32\scripts\p
ip.exe'

Any changes on this issue?

@pfmoore
Member
pfmoore commented Apr 6, 2015

It's not fixable in any practical sense (at least, someone would need to supply a complete, working patch for assessment if they thought it was, as nobody has yet figured out a way to fix it - it's essentially a Windows limitation).

The correct way to upgrade pip on Windows is python -m pip install -U pip.

@dalcinl
dalcinl commented May 19, 2015

@pfmoore What about Python 2.6?

[py26] C:\Users\IEUser>python -m pip --version
C:\Users\IEUser\Anaconda\envs\py26\python.exe: pip is a package and cannot be directly executed
@pfmoore
Member
pfmoore commented May 19, 2015

@dalcinl Sigh. I hate having to think about Python 2.6 :-)

If I remember, python -m pip.__main__ works there. I don't have a copy of Python 2.6 to hand to test it at the moment, sorry...

@pfmoore
Member
pfmoore commented May 20, 2015

I've just tested on Python 2.6, and python -m pip.__main__ does indeed work there.

@AmazingHorse

Hello!

python -m pip install -U pip does not work for me (w7, powershell, installed python via chocolatey); the install fails with permission issues:

PS C:\tools> python --version
Python 3.4.3
PS C:\tools> pip --version
pip 6.0.8 from C:\tools\python\lib\site-packages (python 3.4)
PS C:\tools> python -m pip install -U pip
You are using pip version 6.0.8, however version 7.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.0.1-py2.py3-none-any.whl#md5=9b403ba9b82d4a1e5fda5b6cc8952b57
  Using cached pip-7.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\tools\python\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      PermissionError: [WinError 5] Access is denied: 'c:\\tools\\python\\lib\\site-packages\\pip-6.0.8.dist-info\\description.rst' -> 'C:\\Users\\Administrator\\AppData\\Local\\Temp\\pip-4ghq1oa9-uninstall\\tools\\python\\lib\\site-packages\\pip-6.0.8.dist-info\\description.rst'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\tools\python\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\tools\python\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\tools\python\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\tools\python\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\tools\python\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\tools\python\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\tools\python\lib\shutil.py", line 536, in move
          os.unlink(src)
      PermissionError: [WinError 5] Access is denied: 'c:\\tools\\python\\lib\\site-packages\\pip-6.0.8.dist-info\\description.rst'

Thoughts?

@pfmoore
Member
pfmoore commented May 28, 2015

There's no way anything should have that file locked, or that you should not have permission to delete it. Could it be a chocolatey issue? Can you delete that file manually from a command prompt?

@mikofski

@AmazingHorse have you tried

curl -k https://bootstrap.pypa.io/get-pip.py | python

assuming you have cURL?

@AmazingHorse

Interesting, @pfmoore, that file is locked even on an administrator command prompt.

So, sounds like a windows permissions issue rather than something with pip.

Does it have to do with the fact that my python was installed outside of the windows directory structure in C: ?

I'll be a'googling this but I'm curious if anyone can post here what they might think this would be caused by.

Cheers guys

@pfmoore
Member
pfmoore commented May 29, 2015

@AmazingHorse Honestly I've no idea. The other possibility is that it may be that there's something particular about how Chocolatey installs Python - maybe report the issue on their tracker (I assume they have one?) and see what they say.

But yes, it's not a pip issue from the sound of it.

@eryksun
eryksun commented Aug 21, 2015

Windows allows renaming/moving memory-mapped files (mapped as image or data) to a new name on the same volume. So the rmtree error handler can try renaming the file to move it to the base temp directory. At least there it won't interfere with removing the directory. This workaround fails if the file is on a different volume than the temp directory, or for a sharing violation, or if the file's ACL prevents the operation. But none of those should apply in this case.

If the Administrators group is enabled in the process token, then the renamed file can be scheduled for deletion when the system restarts. Just use ctypes to call MoveFileExW(path, None, MOVEFILE_DELAY_UNTIL_REBOOT). This fails for regular users, however, because they aren't allowed to modify the session manager's PendingFileRenameOperations registry value.

@dszimmer

pip on windows 7/10 working fine except for update> pip install --upgrade pip
I try to load the least possible to my C:\ (SSD) and install software to tree starting at D:\NUC
Here's what I'm seeing:

D:\NUC\Python34>pip install --upgrade pip
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Downloading pip-7.1.2-py2.py3-none-any.whl (1.1MB)
    100% |################################| 1.1MB 204kB/s
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "D:\NUC\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      OSError: [WinError 17] The system cannot move the file to a different disk drive: 'd:\\nuc\\python34\\scripts\\pip.exe' -> 'C:\\Users\\David\\AppData\\Local\\Temp\\pip-247i52js-uninstall\\nuc\\python34\\scripts\\pip.exe'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "D:\NUC\Python34\lib\site-packages\pip\basecommand.py", line 232, in main
        File "D:\NUC\Python34\lib\site-packages\pip\commands\install.py", line 347, in run
        File "D:\NUC\Python34\lib\site-packages\pip\req\req_set.py", line 543, in install
        File "D:\NUC\Python34\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
        File "D:\NUC\Python34\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
        File "D:\NUC\Python34\lib\site-packages\pip\utils\__init__.py", line 316, in renames
        File "D:\NUC\Python34\lib\shutil.py", line 536, in move
          os.unlink(src)
      PermissionError: [WinError 5] Access is denied: 'd:\\nuc\\python34\\scripts\\pip.exe'

The AppData folder path is available on my D: drive, wondering how best to handle this situation for subsequent tools and usage. Thought of a junction, but it will still seem like a different drive. Maybe script the diffs of the expected location to actual location to check give the correct path? Before doing this, pip installed django just fine (but complained about the version which led to this discovery). hmmm... and now pip seems broken.

D:\NUC\Python34\Scripts>pip -h
Traceback (most recent call last):
  File "D:\NUC\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\NUC\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\NUC\Python34\Scripts\pip.exe\__main__.py", line 5, in <module>
ImportError: No module named 'pip'

D:\NUC\Python34\Scripts>dir
 Volume in drive D is TOSHIBA EXT
 Volume Serial Number is E085-D5D7

 Directory of D:\NUC\Python34\Scripts

08/25/2015  09:08 AM    <DIR>          .
08/25/2015  09:08 AM    <DIR>          ..
08/25/2015  09:08 AM            92,045 django-admin.exe
08/25/2015  09:08 AM               136 django-admin.py
08/25/2015  09:01 AM               346 easy_install-3.4-script.py
08/25/2015  09:01 AM            65,536 easy_install-3.4.exe
08/25/2015  09:01 AM               651 easy_install-3.4.exe.manifest
08/25/2015  09:01 AM               338 easy_install-script.py
08/25/2015  09:01 AM            65,536 easy_install.exe
08/25/2015  09:01 AM               647 easy_install.exe.manifest
08/25/2015  08:19 AM            91,984 pip.exe
08/25/2015  08:19 AM            91,984 pip3.4.exe
08/25/2015  08:19 AM            91,984 pip3.exe
08/25/2015  09:08 AM    <DIR>          __pycache__
              11 File(s)        501,187 bytes
               3 Dir(s)  328,895,152,128 bytes free

D:\NUC\Python34\Scripts>
@stonebig
Contributor
stonebig commented Sep 7, 2015

you have to do python.exe -m pip install --upgrade pip

@nmusatti

On Python 2.6 you need to run python -m pip.__main__ install -U pip

@pekkaklarck
Contributor

I consider the current situation really bad for new Python users as well as for old Python users like me who just typically don't use Windows. I'm perfectly fine using python -m pip install --upgrade pip, though, but I really think pip install --upgrade pip should recommend that automatically on Windows. @dstufft proposed this already in 2013. Is there a good reason not to do it?

PS: Sorry for reporting #3462 as a duplicate of this issue. I didn't find this issue by searching the tracker for WindowsError. Hopefully after this comment others in the future will.

@stonebig
Contributor
stonebig commented Feb 7, 2016

To the best of my knowledge, it is already the command that pip suggests to the user when detecting a new version of itself. The situation is not that bad.

@pfmoore
Member
pfmoore commented Feb 7, 2016

@pekkaklarck The main reason for not doing it is that it's really hard to spot. For a start, I don't know how to reliably detect that pip is being run from pip.exe rather than from python -m pip. And even if you detect that, you need to check if pip is in the things being installed (consider pip install --upgrade -r http://some.remote/requirements.txt - by the time you've read the requirements file, it's arguably too late to issue the message).

Of course, something that's not 100% perfect is an entirely reasonable thing to implement - it's making sure we don't object to entirely valid usages, rather than making sure we catch all the corner cases, that's really important.

We'd certainly consider merging a PR to address this issue, but I don't think any of the core pip developers find the issue sufficiently problematic to attempt a fix themselves (I'm probably the heaviest Windows user in the core devs, and I find remembering to use python -m pip reasonably easy).

@Ivoz
Member
Ivoz commented Feb 7, 2016

for the purposes of finding whether we're pip[.exe] or not, I imagine

"pip" in os.path.basename(sys.argv[0])

would work fine. Also just warning to seeing [-U|--upgrade] and pip in the invocation would probably be enough to fix 90% of invocations.

@pekkaklarck
Contributor

@stonebig: Could you clarify when that command is recommended? I tried pip install -U pip after a fresh Python 2.7.11 installation and that failed with this error:

WindowsError: [Error 5] Access is denied: 'c:\users\koulutus\appdata\local\temp\pip-65z4kx-uninstall\python27\scripts\pip.exe'

@pfmoore: Upgrading pip in requirements file sounds pretty rare and could probably be ignored. Like @Ivoz I thought pip.exe would be in sys.argv. Alternatively it ought to be possible to catch the WindowsError and react to that.

@pfmoore
Member
pfmoore commented Feb 7, 2016

@Ivoz does that work? I thought the pip.exe wrapper invoked python as a subprocess, so it wouldn't work. But a quick test (printing sys.argv[0] in pip.main()) shows:

>py -m pip
C:\Users\Gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\pip\__main__.py

Usage:
...

>pip
C:\Users\Gustav\AppData\Local\Programs\Python\Python35\Scripts\pip

Usage:
...

>pip.exe
C:\Users\Gustav\AppData\Local\Programs\Python\Python35\Scripts\pip

Usage:

So it works, but you need to be careful how you spot pip in sys.argv[0] (note that the first example includes the directory pip in the path to __main__.py)

Thanks for pointing this out. I learned something today :-)

@pekkaklarck The problem with catching the WindowsError (if I recall) is that it happens quite late, and the install is part-complete (and not easy to roll back).

But I don't want to discourage anyone from trying to fix this issue, so I'll leave it at that :-)

@Ivoz
Member
Ivoz commented Feb 7, 2016

but you need to be careful how you spot pip in sys.argv[0]

@pfmoore hence the call to os.path.basename.

@pfmoore
Member
pfmoore commented Feb 7, 2016

@Ivoz Sigh. Too late at night here. Yes, obviously :-)

@stonebig
Contributor
stonebig commented Feb 8, 2016

@pekkaklarck . by example when doing pip list (on windows). As far as I remember in the discussions of last year:

  • this was the only solution the pypa team imagined as reliable,
  • it is only shown on windows.

pip_example

There could be a better solution, but I assume it's already 80% of a perfect answer from previous situation.

Maybe a better solution would be to have an "upgrade_pip.bat" delivered with python default Windows installation, not tighted to "pip", and that would do that command. (or pip_update.bat to better match conda ?)

@kennethreitz
Contributor

I hit this today while diving-into the fresh Windows Python user experience. It was terrible.
Can we make -m automatic for that command? (perhaps give upgrading pip it's own command?)

No user should ever have to see this, especially after running exactly the command pip told them to run.

screen shot 2016-02-23 at 9 51 53 pm

@dstufft
Member
dstufft commented Feb 24, 2016

I think we already have the logging setup in more recent versions of pip than 7.0.1 to tell Windows users to use python -m pip install --upgrade pip instead of pip install --upgrade pip.

@kennethreitz
Contributor

🍰

@xavfernandez
Contributor

Yes, since 7.1 cf #2830

@kennethreitz
Contributor

\o/

@grasx
grasx commented Mar 8, 2016

I can't upgrade my pip on win 10, what's wrong?

default

@piotr-dobrogost

@grasx
If you really want to know what's wrong then please read this thread. If you only are interested in getting this to work then please read the last line from the output you got and compare it with what you ran.

@nmusatti
nmusatti commented Mar 8, 2016

@piotr-dobrogost
Is it? The last red line appears to point to a decoding problem, rather to the access denied error that is the usual outcome of in place update on Windows. Could @grasx 's problem be related to the use of a cyrillic codepage?

@pekkaklarck
Contributor

I doubt the problem @grasx encountered is related to this issue. I've seen similar probles on Windows, though. See #3463.

@piotr-dobrogost

@nmusatti
You are right. My bad.

@grasx
grasx commented Mar 9, 2016

@piotr-dobrogost @nmusatti @pekkaklarck However, thank you, I changed user's name, so now it contains only ASCII chars and successfully upgraded pip to 8.1

@pzz2011
pzz2011 commented Mar 23, 2016

@grasx changed user's name? I am confused in what you said... I got the same trouble, something difference is that I was in windows 7

@RonnyPfannschmidt
Contributor

random idea, how about not replacing the exe file if its not necessary

as far as i understood the exe files are very small trampolines that typically work

the more important logic is in the accompanying script files

@grasx
grasx commented Mar 23, 2016

@pzz2011 Well, actually I reinstalled my win and set new user name. It was easier in my case then to find out how to change it in all the places and routes where it is mentioned :)

@vsajip
Contributor
vsajip commented Mar 23, 2016

@RonnyPfannschmidt Depends what is used to write the .exe - setuptools used to use foo.exe + foo-script.py (not sure what it does now), whereas distlib (if so configured) creates a single foo.exe which embeds the script.

@pzz2011
pzz2011 commented Mar 25, 2016

@grasx hhh... I found an easier way to solve it. I just reinstall the python from 2.7.x to 2.7.10.
Then the success smile to me~

@godfroyt
godfroyt commented Apr 3, 2016

On Windows, use the Command Prompt windows in Administrator mode. Then type command
python -m pip install --upgrade pip

I hope it works for you. That's how I got it to work an Window 10

Good luck.

@Robotbeatshuman

@grasx Not entirely sure why, but that bug comes up when selecting "install for all users" on the Python installer on both windows 8.1 and 10. Do a reinstall, select PATH and custom install. Make sure the top box is unchecked and select install location. Using "python -m pip install --upgrade pip" in cmd works fine after that.

@emigenix
emigenix commented Aug 17, 2016 edited

Just an FYI, for Windows users. If you install Cygwin, Python2.7 is automatically installed. All you need to do then, is run:

wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py

(Yes, you need to install wget first.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment