Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

windows in-place pip upgrades #1299

Closed
qwcode opened this issue Nov 5, 2013 · 89 comments
Closed

windows in-place pip upgrades #1299

qwcode opened this issue Nov 5, 2013 · 89 comments

Comments

@qwcode
Copy link
Contributor

@qwcode qwcode commented Nov 5, 2013

not sure of the status of pip install --ugrade pip on windows.

need this for PEP453

@pfmoore ?

see #188 for an old discussion on this.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 5, 2013

Use python -m pip install --upgrade pip. I think @vsajip was considering some form of magic in the exe wrappers to mean that self-replacement of the exe could be made to work. But I don't know if he did anything in the version of distlib we're using. I'm not actually sure it's something that could be done without help from the application (pip). And I'm pretty sure no code has been added to pip for this.

Personally, I consider this to be a minor issue - telling people to use `python -m pip`` seems to me to be a perfectly acceptable answer, given that the issue is fundamental to how Windows handles deletion of open files.

Ultimately it's going to be a question of how much complexity (with the consequent maintenance cost) and risk we want to take on in order to solve this.

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

Personally i'm ok with pip install --upgrade pip printing a message to use python -m pip install --upgrade pip. I just don't think we should throw an error about open file handles or whatever the exception is.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 5, 2013

I was just about to say the same thing.

Here's a question - would you want that message to only appear on Windows, or would it be reasonable to say the same on Unix (for a consistent user experience). Personally, I'd like to be consistent across platforms (because it's easier to document that way).

Also, we should trap any command that attempts to (re-)install pip - pip install -I pip is another one, for example.

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

Personally I think it should be windows only. I view it as a work around for the way the Windows platform works, and something that's a target to eventually remove for something nicer down the road.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 5, 2013

I knew you were going to say that :-(

Frankly, I don't believe we will ever get "something nicer", basically because we don't have enough Windows expertise to develop and maintain it. I know of no Windows software with a satisfactory self-replacement of executables solution. I certainly don't have an answer, and I wouldn't expect any of the Unix-based developers to be able to help. Even if we did get something, it's likely to be complex and so a probable source of bit rot and obscure bugs.

Making a solution that works on all platforms the "official answer" seems to me to avoid sending confusing messages to the user. Back it up with a warning as described and we have a robust solution, even if it's not quite as perfect as getting pip install -U pip to work.

Please excuse a certain level of mild grumpiness in this reply, caused by a feeling that it's always the Windows platform that has to be the one to compromise - I don't mean to make an issue out of this. In practice, I'll go with whatever the consensus is. I don't think it's a big deal.

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

FWIW my goal after PEP453 is done is to get our own testing infrastructure stood up which includes Windows, so hopefully that'll solve some of the windows bitrot problems.

I can understand the grumpyness though :) If we were designing this system brand new I'd be a lot more amendable to that but I don't think it's user friendly to break pip install -U pip for everything non Windows in order to maintain consistency. If this was a brand new thing then there's no (or less at least) user epectation :]

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 5, 2013

OK, that sounds fair. You're right that historical usage is more important here.

I actually have no feel at all for what the user experience on Unix is, so I'm probably just as guilty of one-sided thinking as anyone else. I must try to see how using Python on Linux feels - is there a good "Python on Linux for beginners" resource that shows how to set up a Python (3) environment, get the various tools set up, etc? [One thing I always find doesn't get covered, is how to make "python" run Python 3 by default, how to make "virtualenv" build Python 3 virtualenvs by default, all that sort of thing - it seems to me that Python 3 is much more second-class on Unix than on Windows, oddly enough.]

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

Typically you don't want to make python point to python3 on *nix because the system uses Python and not all of their libraries are going to be ported to Python3. However there are some Linux distros that have Python3 as a default now... I think Arch? And Maybe the next Fedora?

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 5, 2013

Hmm, I'm not talking at a system level, but really you can't choose what to use for Python in your personal account? That definitely makes Python 3 second-class, to my way of thinking... But it does explain why Unix users don't seem to find using things like "pip3" or "pip3.3" as ugly as sin :-)

(Anyway, we're way off-topic here, so never mind).

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

You can setup your shell so it has ~/bin/ before the system bin and add a symlink there from python to python3, but yea offtopic. I'm happy to help though if you're actually interested in getting things set up :] IRC or Email.

@vsajip
Copy link
Contributor

@vsajip vsajip commented Nov 5, 2013

My work which I believe Paul was referring to was just to implement the 'deleteme' dance for replacing files in use (i.e. exe wrappers) - that's in distlib Python code, not in the .exe wrappers. This is in the distlib.scripts module and should be exercised whenever that module is used to create a script.

@dstufft
Copy link
Member

@dstufft dstufft commented Nov 5, 2013

So perhaps this is already solved for Wheels? And it's a setuptools problem elsewise?

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 6, 2013

Sounds possible. I'll do some tests today, if I can.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 6, 2013

Sadly not, at least with "pip install -I pip". (It's hard to test an actual upgrade, as only the absolute latest version of pip has script wrapper support):

>pip install -f $x -I pip --use-wheel --no-index --pre
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking pip
Installing collected packages: pip
  Found existing installation: pip 1.5.dev1
    Uninstalling pip:
      Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 365, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\basecommand.py", line 121, in main
    status = self.run(options, args)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\commands\install.py", line 258, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1330, in install
    requirement.commit_uninstall()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 557, in commit_uninstall
    self.uninstalled.commit()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1743, in commit
    rmtree(self.save_dir)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 42, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 460, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 367, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 61, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

Storing complete log in C:\Users\uk03306\pip\pip.log
@vsajip
Copy link
Contributor

@vsajip vsajip commented Nov 6, 2013

So pip's not using distlib.scripts to install scripts ... I guess it doesn't fit in the pip workflow.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 6, 2013

I suspect it's because we do an uninstall then an install, rather than an install overwriting what's there (for obvious reasons).

@agronholm
Copy link
Contributor

@agronholm agronholm commented Nov 25, 2013

Windows doesn't allow overwriting or deleting open exe files, but it does allow renaming/moving them.

@rgbkrk
Copy link

@rgbkrk rgbkrk commented Nov 26, 2013

Well, with some tomfoolery with the Win32 API you can close the handle to pip.exe much like you can when using Sysinternal's process explorer.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Nov 26, 2013

I don't think you can close all handles to the currently running exe. But if you have working code as an example, I'd love to see it, as this has long been a problem that people have debated and the consensus has always been that it's not possible (except by tricks like "rename and delete later"). Working code that does this would be extremely useful to the community.

@techtonik
Copy link
Contributor

@techtonik techtonik commented Mar 25, 2014

Run into the same issue.

As a quick hack, can pip.exe detect that it is being upgraded prior to execution and restart itself with python -m pip {{ sys.argv }}?

@dstufft
Copy link
Member

@dstufft dstufft commented Mar 25, 2014

I don't think that actually works.

But maybe a stupid thought, afaik the .exe doesn't need to be updated, can we just special case pip.exe and leave it alone? Would possibly leave a pip.exe laying around after an install though.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Mar 25, 2014

@dstufft correct, "restarting yourself" isn't as simple as @techtonik makes it sound on Windows, where there's no equivalent of the Unix exec call.

Technically, pip.exe may need updating if the entry point changes or if the distlib wrapper code is changed. That's not common, but it can happen (early on, the distlib code changed a fair bit till things settled down). What it may be possible to do would be to write the new pip.exe to a temporary location, then if it's identical to the current one, skip it. You wouldn't avoid errors when it did need updating, but in the 99% of cases when it didn't, you'd be fine.

BTW, we're going round in circles here. According to Vinay's earlier comments, this is solved in distlib (which is where I think it should be solved) but pip doesn't use the API in a way that allows it to work. If we could use the distlib solution, then great. Otherwise, we need someone to write a pip-specific solution (and I remain skeptical that this can be done in a robust, maintainable manner that isn't far more complex than the issue warrants).

Suggestion:

  1. Someone add a section to the pip docs at http://www.pip-installer.org/en/latest/installing.html#install-or-upgrade-pip noting that the python -m pip workaround is needed on Windows when upgrading pip with itself. I might get a chance to do this at some point, but PRs from people who have contributed to this issue would be gratefully accepted :-).
  2. If anyone has a viable solution, and is prepared to implement it, they should raise a PR and we can review actual code rather than proposed approaches.

I think we're at the stage where actual working code is the only thing that will move this issue forward.

techtonik added a commit to techtonik/pip that referenced this issue Mar 25, 2014
I am not sure how to detect that pip is run as pip.exe
@techtonik
Copy link
Contributor

@techtonik techtonik commented Mar 25, 2014

I am -1 on bloating the docs without trying other solutions.

@qwcode
Copy link
Contributor Author

@qwcode qwcode commented Mar 25, 2014

I'm +1 on documenting what works right now.

@techtonik
Copy link
Contributor

@techtonik techtonik commented Mar 25, 2014

Even with doc comment it is still workaround - not a solution, and you've got command prompt for Windows wrong.

@mikofski
Copy link

@mikofski mikofski commented Mar 26, 2014

Maybe this is crazy but I always just do this:

$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

of course this requires one to have libcurl, but there are windows binaries available here.

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Sep 8, 2017

@ofek #4490 made a change in pip, printing a better message, IIRC.

bamarni added a commit to bamarni/dcos-cli that referenced this issue Apr 3, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to bamarni/dcos-cli that referenced this issue Apr 3, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to dcos/dcos-cli that referenced this issue Apr 4, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to bamarni/dcos-cli that referenced this issue Apr 4, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to dcos/dcos-cli that referenced this issue Apr 6, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to dcos/dcos-cli that referenced this issue Apr 10, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
bamarni added a commit to dcos/dcos-cli that referenced this issue Apr 10, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
@ihormelnyk
Copy link

@ihormelnyk ihormelnyk commented Apr 17, 2018

So for now if I've installed python without pip I will not be able to install pip module in future?

c:\Python27>python.exe get-pip.py ERROR: To modify pip, please run the following command: c:\Python27\python.exe -m pip

c:\Python27>c:\Python27\python.exe -m pip install c:\Python27\python.exe: No module named pip

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Apr 17, 2018

Hey @ihormelnyk!

That's not the case. The conditional for this was too broad -- it blocks using any script with the word "pip" in it from modifying pip. You can try renaming get-pip.py to not contain the word "pip", to work around this issue.

A 10.0.1 release (which will likely happen this week) will fix this issue.

@AlexArcPy
Copy link

@AlexArcPy AlexArcPy commented Apr 18, 2018

@pradyunsg thanks for sharing the tip. I had the same issue as @ihormelnyk and renaming the get-pip.py file to a name that does not contain the word pip helped.

@BJQ98
Copy link

@BJQ98 BJQ98 commented Apr 19, 2018

I print pip in cmd inwindows,but it showed
Fatal error in launcher: Unable to create process using '""d:\program files (x86)\python3.6\python.exe" "D:\Program Files (x86)\python3.6\Scripts\pip.exe" '
I have already set environment for pip,but it doesn't work,how to figure it out?

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Apr 19, 2018

@BJQ98 You'd want to look at #5223.

whtsky added a commit to whtsky/bencoder.pyx that referenced this issue May 1, 2018
@supermansaga
Copy link

@supermansaga supermansaga commented May 7, 2018

pradyunsg said 10.0.1 would fix this, but i don't know how. Thanks.

C:\Users\clarkk\AppData\Local\Programs\Python\Python36>python -m pip install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
  Rolling back uninstall of pip
Exception:
Traceback (most recent call last):
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Users\yangg\AppData\Roaming\Python\Python36\site-packages\pip\req\req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\req\req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\wheel.py", line 462, in move_wheel_files
    generated.extend(maker.make(spec))
  File "C:\Users\yangg\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 372, in make
    self._make_script(entry, filenames, options=options)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 276, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 212, in _write_script
    launcher = self._get_launcher('t')
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 351, in _get_launcher
    result = finder(distlib_package).find(name).bytes
AttributeError: 'NoneType' object has no attribute 'bytes'
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented May 7, 2018

That'll be the same as #5343.

@supermansaga
Copy link

@supermansaga supermansaga commented May 7, 2018

Thanks pradyunsg, Someone else's solution turns out to be simpler. Just performed "python -m pip uninstall pip" first, then "python -m pip install -U pip"

Requirement already up-to-date: pip in c:\user\clarkk\appdata\local\Programs\Python\Python36\lib\site-packages (10.0.1)

bamarni added a commit to dcos/dcos-core-cli that referenced this issue Jul 11, 2018
Don't run `pip install --upgrade pip` on Windows, it started to fail in
our Jenkins node and this is not the right command.

https://jira.mesosphere.com/browse/DCOS_OSS-2332
https://pip.pypa.io/en/stable/installing/#upgrading-pip
pypa/pip#1299
@nidhi100
Copy link

@nidhi100 nidhi100 commented Nov 3, 2018

      Personally I think it should be windows only. I view it as a work around for the way the Windows platform works, and something that's a target to eventually remove for something nicer down the road.

I am getting error while running above command as syntax error to the word get what might be the solution for this

@pypa pypa deleted a comment from Brykerr78 Dec 3, 2018
@pfmoore
Copy link
Member

@pfmoore pfmoore commented Jan 12, 2019

@AProNoob You should be running the commands at a command prompt (cmd or powershell), not at a Python prompt.

@AProNoob
Copy link

@AProNoob AProNoob commented Jan 12, 2019

Thanks! That worked!

@pypa pypa deleted a comment from liyiliuxingyu Jan 23, 2019
@lock
Copy link

@lock lock bot commented May 28, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the S: auto-locked label May 28, 2019
@lock lock bot locked as resolved and limited conversation to collaborators May 28, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.