Skip to content
This repository

Add "upgrade" and "upgrade-all" commands #59

Open
vbabiy opened this Issue March 14, 2011 · 41 comments
Vitaly Babiy

new description from @qwcode


pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

pip upgrade-all would upgrade all installed packages.

In order to do this properly, pip needs a real dependency resolver first. (see #988). Without this, pip upgrade can be destructive to a previously working environment.

For history's sake, note that #571 was an attempt to change the default behavior of pip install --upgrade to non-recursive by default. It has a lot of work in there, but ultimately, it was decided not to break compatibility, and put the energy towards a new pip upgrade.

Vitaly Babiy

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.


Original Comment By: Carl Meyer
Vitaly Babiy

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.


Original Comment By: Jannis Leidel
Vitaly Babiy

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.


Original Comment By: Carl Meyer
Vitaly Babiy

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in /lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.


Original Comment By: Ian Bicking
Vitaly Babiy

Issue #167 was marked as a duplicate of this issue.


Original Comment By: Carl Meyer
Vitaly Babiy

+1 for this bug


Original Comment By: smyrman
Vitaly Babiy

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's


Original Comment By: Kelsey Hightower
Vitaly Babiy

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?


Original Comment By: Anonymous
Vitaly Babiy

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).


Original Comment By: Tomasz Elendt
Vitaly Babiy

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!


Original Comment By: Carl Meyer
Vitaly Babiy

Carl: Cool, I will get started and update this ticket with the results.


Original Comment By: Kelsey Hightower
Vitaly Babiy

While working on the upgrade command the following questions came up:

  • What methods should pip upgrade support to specify which packages to
    upgrade? Should we support a requirements file?

  • How should pip upgrade handle packages that are not already installed?
    Should we install missing packages?

pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?


Original Comment By: Kelsey Hightower
Vitaly Babiy

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.


Original Comment By: Carl Meyer
Vitaly Babiy

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.


Original Comment By: Kelsey Hightower
Vitaly Babiy

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

  • Tests

  • Filter requirements file; add non-editables to list of packages to
    upgrade.

Running pip using the upgrade command
# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower
Vitaly Babiy

Last set of questions (I hope):

  • Should pip upgrade parse the requirements file and filter out editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

  • Should pip upgrade search the indexes for a later version to install? (This is what I am doing now). If not how should the upgrade command determine if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

  • The package is not installed

  • An upgrade is available (later version from the indexes), and no explicit
    version was requested

  • The requested version is different from the installed one (Version miss
    match).

  • I am deferring the requirements file to the install command until I filter
    out the non-editable requirements.


Original Comment By: Kelsey Hightower
Vitaly Babiy

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
Vitaly Babiy

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
Vitaly Babiy

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c


Original Comment By: Kelsey Hightower
Vitaly Babiy

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.


Original Comment By: Carl Meyer
Vitaly Babiy

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade
in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all
, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command
# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower
Vitaly Babiy

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.


Original Comment By: Carl Meyer
Vitaly Babiy

Thanks Carl, I will wrap this up and start looking at #13


Original Comment By: Kelsey Hightower
Vitaly Babiy

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py


Original Comment By: Kelsey Hightower
Jens Diemer
jedie commented July 07, 2011

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

Ivan D Vasin

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

Gian Marco Gherardi

+1

Januzz

+1!

Sebastian Dahlgren

+1

Jannis Leidel
Owner

Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.

Instead I'd be thrilled to see a comment "patch done!" ;)

GoTLiuM InSPiRiT

+1

Pradyun

Knock Knock! Any Updates? Is there any plan to add this, it's 3 years old now..

Kelsey Hightower

I thought this was merged already. I can dust off my python skill and give it another go.

Adam Lindberg

Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...

jason gors

i don't have the time to hack around in the pip codebase at the moment, but just wrote a simple script to do this, which can be found here at pip_updater. hope this helps in the interim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.