New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "upgrade" and "upgrade-all" commands #59

Closed
vbabiy opened this Issue Mar 15, 2011 · 250 comments

Comments

Projects
None yet
@vbabiy
Contributor

vbabiy commented Mar 15, 2011

new description from @qwcode


pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

pip upgrade-all would upgrade all installed packages.

For history's sake, note that #571 was an attempt to change the default behavior of pip install --upgrade to non-recursive by default. It has a lot of work in there, but ultimately, it was decided not to break compatibility, and put the energy towards a new pip upgrade. the current --upgrade flag would likely be deprecated.

@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.


Original Comment By: Jannis Leidel
Contributor

vbabiy commented Mar 15, 2011

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.


Original Comment By: Jannis Leidel
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in <sys.prefix>/lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.


Original Comment By: Ian Bicking
Contributor

vbabiy commented Mar 15, 2011

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in <sys.prefix>/lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.


Original Comment By: Ian Bicking
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

+1 for this bug


Original Comment By: smyrman
Contributor

vbabiy commented Mar 15, 2011

+1 for this bug


Original Comment By: smyrman
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Issue #167 was marked as a duplicate of this issue.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

Issue #167 was marked as a duplicate of this issue.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).


Original Comment By: Tomasz Elendt
Contributor

vbabiy commented Mar 15, 2011

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).


Original Comment By: Tomasz Elendt
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?


Original Comment By: Anonymous
Contributor

vbabiy commented Mar 15, 2011

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?


Original Comment By: Anonymous
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Carl: Cool, I will get started and update this ticket with the results.


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Carl: Cool, I will get started and update this ticket with the results.


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

While working on the upgrade command the following questions came up:

  • What methods should pip upgrade support to specify which packages to
    upgrade? Should we support a requirements file?
  • How should pip upgrade handle packages that are not already installed?
    Should we install missing packages?
pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

While working on the upgrade command the following questions came up:

  • What methods should pip upgrade support to specify which packages to
    upgrade? Should we support a requirements file?
  • How should pip upgrade handle packages that are not already installed?
    Should we install missing packages?
pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

  • Tests
  • Filter requirements file; add non-editables to list of packages to
    upgrade.
Running pip using the upgrade command
# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

  • Tests
  • Filter requirements file; add non-editables to list of packages to
    upgrade.
Running pip using the upgrade command
# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Last set of questions (I hope):

  • Should pip upgrade parse the requirements file and filter out
    editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

  • Should pip upgrade search the indexes for a later version to install?
    (This is what I am doing now). If not how should the upgrade command determine
    if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

  • The package is not installed
  • An upgrade is available (later version from the indexes), and no explicit
    version was requested
  • The requested version is different from the installed one (Version miss
    match).
  • I am deferring the requirements file to the install command until I filter
    out the non-editable requirements.

Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Last set of questions (I hope):

  • Should pip upgrade parse the requirements file and filter out
    editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

  • Should pip upgrade search the indexes for a later version to install?
    (This is what I am doing now). If not how should the upgrade command determine
    if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

  • The package is not installed
  • An upgrade is available (later version from the indexes), and no explicit
    version was requested
  • The requested version is different from the installed one (Version miss
    match).
  • I am deferring the requirements file to the install command until I filter
    out the non-editable requirements.

Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade
in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all
, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command
# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade
in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all
, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command
# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.


Original Comment By: Carl Meyer
Contributor

vbabiy commented Mar 15, 2011

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.


Original Comment By: Carl Meyer
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

Thanks Carl, I will wrap this up and start looking at #13


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

Thanks Carl, I will wrap this up and start looking at #13


Original Comment By: Kelsey Hightower
@vbabiy

This comment has been minimized.

Show comment
Hide comment
@vbabiy

vbabiy Mar 15, 2011

Contributor

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py


Original Comment By: Kelsey Hightower
Contributor

vbabiy commented Mar 15, 2011

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py


Original Comment By: Kelsey Hightower
@jedie

This comment has been minimized.

Show comment
Hide comment
@jedie

jedie Jul 7, 2011

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

jedie commented Jul 7, 2011

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

@nisavid

This comment has been minimized.

Show comment
Hide comment
@nisavid

nisavid Feb 21, 2012

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

nisavid commented Feb 21, 2012

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

@gimmi

This comment has been minimized.

Show comment
Hide comment
@gimmi

gimmi commented Sep 15, 2012

+1

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost commented Sep 22, 2012

@kelseyhightower Any progress?

@fradeve

This comment has been minimized.

Show comment
Hide comment
@fradeve

fradeve commented Sep 25, 2012

+1

@januz

This comment has been minimized.

Show comment
Hide comment
@januz

januz commented Sep 28, 2012

+1!

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg Jan 14, 2017

Member

This issue is now really long and really old, a lot of things have been discussed, too many things have happened and this issue has almost attained a low signal/noise. It's hard to see what all has happened.

FWIW, the reason that upgrade was on the cards was because of the fact that install --upgrade has a broken default. Since we are now one step closer to fixing that, I guess it's best we have a new issue for that.

I suggest that this issue be closed because of the above and new issues be created for whatever here is seen as unresolved. I can see 2 things:

  • Shift the default upgrade strategy to only-if-needed.
  • Add "upgrade the world" functionality that depends on #988.
Member

pradyunsg commented Jan 14, 2017

This issue is now really long and really old, a lot of things have been discussed, too many things have happened and this issue has almost attained a low signal/noise. It's hard to see what all has happened.

FWIW, the reason that upgrade was on the cards was because of the fact that install --upgrade has a broken default. Since we are now one step closer to fixing that, I guess it's best we have a new issue for that.

I suggest that this issue be closed because of the above and new issues be created for whatever here is seen as unresolved. I can see 2 things:

  • Shift the default upgrade strategy to only-if-needed.
  • Add "upgrade the world" functionality that depends on #988.
@Liso77

This comment has been minimized.

Show comment
Hide comment
@Liso77

Liso77 Jan 14, 2017

Liso77 commented Jan 14, 2017

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg Feb 12, 2017

Member

@Liso77
Well, No. It is not.

Conda and pip are tools with different goals. This is a great read on this topic. The 3rd point is most relevant.


If the discussion for upgrade-all surfaces again (hopefully in a new issue) my vote is for spelling it as follows:

pip install --upgrade :all:
Member

pradyunsg commented Feb 12, 2017

@Liso77
Well, No. It is not.

Conda and pip are tools with different goals. This is a great read on this topic. The 3rd point is most relevant.


If the discussion for upgrade-all surfaces again (hopefully in a new issue) my vote is for spelling it as follows:

pip install --upgrade :all:
@WhyNotHugo

This comment has been minimized.

Show comment
Hide comment
@WhyNotHugo

WhyNotHugo Feb 12, 2017

pip install --upgrade :all: is extremely weird. Let's stick to POSIX semantics, which basically everyone is doing nowadays: pip install --upgrade --all

WhyNotHugo commented Feb 12, 2017

pip install --upgrade :all: is extremely weird. Let's stick to POSIX semantics, which basically everyone is doing nowadays: pip install --upgrade --all

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Feb 12, 2017

What about just pip install --upgrade without any package names or specifiers? To easy to run accidentally?

pip-tools does it like this for pip-compile -P.

FichteFoll commented Feb 12, 2017

What about just pip install --upgrade without any package names or specifiers? To easy to run accidentally?

pip-tools does it like this for pip-compile -P.

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg Feb 12, 2017

Member
Member

pradyunsg commented Feb 12, 2017

@defjaf

This comment has been minimized.

Show comment
Hide comment
@defjaf

defjaf Feb 24, 2017

How about a version of pip list --outdated that produces its list in a format that can be directly (i.e., without sed, cut, etc.) ingested by pip install --upgrade (e.g., pip list --outdated --format install | xargs pip install --upgrade or something similar with backticks)?

defjaf commented Feb 24, 2017

How about a version of pip list --outdated that produces its list in a format that can be directly (i.e., without sed, cut, etc.) ingested by pip install --upgrade (e.g., pip list --outdated --format install | xargs pip install --upgrade or something similar with backticks)?

@SMH17

This comment has been minimized.

Show comment
Hide comment
@SMH17

SMH17 Mar 10, 2017

whatever syntax will be used the most important thing is introducing this command, It's unbelievable that is still missing

in meanwhile I suggest you to try
https://github.com/jgonggrijp/pip-review
with pip-review --local --interactive ask you package by package if you want to update, not very good but better than nothing

SMH17 commented Mar 10, 2017

whatever syntax will be used the most important thing is introducing this command, It's unbelievable that is still missing

in meanwhile I suggest you to try
https://github.com/jgonggrijp/pip-review
with pip-review --local --interactive ask you package by package if you want to update, not very good but better than nothing

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Mar 11, 2017

Member

@SMH17 It's entirely believable that it's still missing, as there are exactly zero commercial Python vendors formally providing funded development time to work on Python packaging usability improvements on behalf of their customers.

So if you'd like to see the situation improve, it's likely that the most helpful thing you can personally do is to either encourage your Python support vendor to invest developer time in improving the tools you use, or if you don't have a support vendor yet, encourage your employer to pay one.

Member

ncoghlan commented Mar 11, 2017

@SMH17 It's entirely believable that it's still missing, as there are exactly zero commercial Python vendors formally providing funded development time to work on Python packaging usability improvements on behalf of their customers.

So if you'd like to see the situation improve, it's likely that the most helpful thing you can personally do is to either encourage your Python support vendor to invest developer time in improving the tools you use, or if you don't have a support vendor yet, encourage your employer to pay one.

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Mar 11, 2017

Member

As an additional piece of context regarding the lack of urgency around this issue, it's worth keeping in mind that the general recommendations are to

  • keep your working environment definitions under source control to improve reproducibility on other systems (using something like https://github.com/jazzband/pip-tools or https://github.com/kennethreitz/pipenv to keep those definitions up to date)
  • aim to routinely upgrade to new releases of dependencies in order to minimise exposure windows to unknown or undisclosed security vulnerabilities

That doesn't mean the commands proposed here aren't useful, they're just markedly less valuable if the current working environment is already being maintained through pip-compile + pip-sync, or pipenv lock + pipenv install.

Member

ncoghlan commented Mar 11, 2017

As an additional piece of context regarding the lack of urgency around this issue, it's worth keeping in mind that the general recommendations are to

  • keep your working environment definitions under source control to improve reproducibility on other systems (using something like https://github.com/jazzband/pip-tools or https://github.com/kennethreitz/pipenv to keep those definitions up to date)
  • aim to routinely upgrade to new releases of dependencies in order to minimise exposure windows to unknown or undisclosed security vulnerabilities

That doesn't mean the commands proposed here aren't useful, they're just markedly less valuable if the current working environment is already being maintained through pip-compile + pip-sync, or pipenv lock + pipenv install.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 11, 2017

It would be valuable to update the original description as I guess there have been some changes made to pip install since the update made by @qwcode.

piotr-dobrogost commented Mar 11, 2017

It would be valuable to update the original description as I guess there have been some changes made to pip install since the update made by @qwcode.

@sebma

This comment has been minimized.

Show comment
Hide comment
@sebma

sebma Mar 23, 2017

Hi everyone.

I have broken some of my python package dependencies because the following command:

pip install --upgrade packageName has upgraded packages recursively.

Why not change the default behaviour of the --upgrade option, that is to uninstall and reinstall ONLY the given package from the command line ?

How about this ?

sebma commented Mar 23, 2017

Hi everyone.

I have broken some of my python package dependencies because the following command:

pip install --upgrade packageName has upgraded packages recursively.

Why not change the default behaviour of the --upgrade option, that is to uninstall and reinstall ONLY the given package from the command line ?

How about this ?

@aaossa

This comment has been minimized.

Show comment
Hide comment
@aaossa

aaossa Apr 3, 2017

@sebma I think that the default behaviour should not be changed. Maybe you could try using the -no-dependencies flag the next time. It should work 👍

aaossa commented Apr 3, 2017

@sebma I think that the default behaviour should not be changed. Maybe you could try using the -no-dependencies flag the next time. It should work 👍

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Apr 4, 2017

@sebma, @aaossa, I'll have you both know that it has pretty much been decided already that the default upgrade strategy will change in the future (ref: #3871 (comment)). The necessary feature (i.e. the --upgrade-strategy argument) has been added in #3972.

As @pradyunsg mentioned earlier, this issue is kind of a left-over. The first part is sort of handled by now (see my first paragraph) and the second part is the only reason why this package is still open, it seems. I don't know if a separate "upgrade-all" issue has been created since.

FichteFoll commented Apr 4, 2017

@sebma, @aaossa, I'll have you both know that it has pretty much been decided already that the default upgrade strategy will change in the future (ref: #3871 (comment)). The necessary feature (i.e. the --upgrade-strategy argument) has been added in #3972.

As @pradyunsg mentioned earlier, this issue is kind of a left-over. The first part is sort of handled by now (see my first paragraph) and the second part is the only reason why this package is still open, it seems. I don't know if a separate "upgrade-all" issue has been created since.

@simion

This comment has been minimized.

Show comment
Hide comment
@simion

simion Apr 26, 2017

I've released a nice interactive upgrader for requirements file: https://github.com/simion/pip-upgrader

simion commented Apr 26, 2017

I've released a nice interactive upgrader for requirements file: https://github.com/simion/pip-upgrader

@pradyunsg

This comment has been minimized.

Show comment
Hide comment
@pradyunsg

pradyunsg Jun 16, 2017

Member

Shift the default upgrade strategy to only-if-needed.

#4500 did this.

Add "upgrade the world" functionality that depends on #988.

#4551 for discussion on this.


Addressing the points made in the current top-post:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here.

It has been decided against adding an upgrade command or making pip install upgrade already installed packages. pip now does have non-recursive upgrades by default, with the recursive behavior available behind --upgrade-strategy eager.

pip upgrade-all would upgrade all installed packages.

#4551 exists and it would be nice to have a fresh discussion on this; when #988 is done.


@dstufft @xavfernandez @pfmoore Do any of you think this issue should be closed?

Edit (05-18-2017): Punctuation + minor text added

Member

pradyunsg commented Jun 16, 2017

Shift the default upgrade strategy to only-if-needed.

#4500 did this.

Add "upgrade the world" functionality that depends on #988.

#4551 for discussion on this.


Addressing the points made in the current top-post:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here.

It has been decided against adding an upgrade command or making pip install upgrade already installed packages. pip now does have non-recursive upgrades by default, with the recursive behavior available behind --upgrade-strategy eager.

pip upgrade-all would upgrade all installed packages.

#4551 exists and it would be nice to have a fresh discussion on this; when #988 is done.


@dstufft @xavfernandez @pfmoore Do any of you think this issue should be closed?

Edit (05-18-2017): Punctuation + minor text added

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jun 17, 2017

Member

Seems reasonable.

Member

dstufft commented Jun 17, 2017

Seems reasonable.

@serafeimgr

This comment has been minimized.

Show comment
Hide comment
@serafeimgr

serafeimgr Sep 5, 2017

Hello all,
I made a simple script/gist that does the job.

https://gist.github.com/serafeimgr/b4ca5d0de63950cc5349d4802d22f3f0

serafeimgr commented Sep 5, 2017

Hello all,
I made a simple script/gist that does the job.

https://gist.github.com/serafeimgr/b4ca5d0de63950cc5349d4802d22f3f0

@qoheniac

This comment has been minimized.

Show comment
Hide comment
@qoheniac

qoheniac Nov 23, 2017

Why not simply do this?

pip install --upgrade $(pip list --outdated | awk '{print $1}' | tr '\n' ' ')

qoheniac commented Nov 23, 2017

Why not simply do this?

pip install --upgrade $(pip list --outdated | awk '{print $1}' | tr '\n' ' ')

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Nov 23, 2017

Because it's not that easy in reality since you may install versions that don't satisfy some of your other packages' dependencies.

FichteFoll commented Nov 23, 2017

Because it's not that easy in reality since you may install versions that don't satisfy some of your other packages' dependencies.

@defjaf

This comment has been minimized.

Show comment
Hide comment
@defjaf

defjaf Mar 23, 2018

Based on and with thanks to @serafeimgr's gist, I've written a possibly-useful command-line tool, pip_upgrade_outdated; source on github. Feedback welcome.

(See also this issue: Yes, the parallel execution is particularly dangerous, and, yes, this may break things. Nonetheless many people run something like this by hand all the time, so might find it useful.)

defjaf commented Mar 23, 2018

Based on and with thanks to @serafeimgr's gist, I've written a possibly-useful command-line tool, pip_upgrade_outdated; source on github. Feedback welcome.

(See also this issue: Yes, the parallel execution is particularly dangerous, and, yes, this may break things. Nonetheless many people run something like this by hand all the time, so might find it useful.)

@serafeimgr

This comment has been minimized.

Show comment
Hide comment
@serafeimgr

serafeimgr Mar 25, 2018

Thank you for taking the time to build a complete solution.
Even though my recommendation would be to find a way to push this feature to pip.

I think pipenv & pipfile is going to replace pip/requirements.txt anyway.
Maybe @kennethreitz knows more about the roadmap and the --upgrade all feature.

serafeimgr commented Mar 25, 2018

Thank you for taking the time to build a complete solution.
Even though my recommendation would be to find a way to push this feature to pip.

I think pipenv & pipfile is going to replace pip/requirements.txt anyway.
Maybe @kennethreitz knows more about the roadmap and the --upgrade all feature.

@marcelocantos

This comment has been minimized.

Show comment
Hide comment
@marcelocantos

marcelocantos Apr 2, 2018

@qoheniac | tr ... is redundant.

marcelocantos commented Apr 2, 2018

@qoheniac | tr ... is redundant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment