Skip to content

Add "upgrade" and "upgrade-all" commands #59

Open
vbabiy opened this Issue Mar 15, 2011 · 137 comments
@vbabiy
vbabiy commented Mar 15, 2011

new description from @qwcode


pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

pip upgrade-all would upgrade all installed packages.

For history's sake, note that #571 was an attempt to change the default behavior of pip install --upgrade to non-recursive by default. It has a lot of work in there, but ultimately, it was decided not to break compatibility, and put the energy towards a new pip upgrade. the current --upgrade flag would likely be deprecated.

@vbabiy
vbabiy commented Mar 15, 2011

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.


Original Comment By: Jannis Leidel
@vbabiy
vbabiy commented Mar 15, 2011

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in /lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.


Original Comment By: Ian Bicking
@vbabiy
vbabiy commented Mar 15, 2011

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

+1 for this bug


Original Comment By: smyrman
@vbabiy
vbabiy commented Mar 15, 2011

Issue #167 was marked as a duplicate of this issue.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).


Original Comment By: Tomasz Elendt
@vbabiy
vbabiy commented Mar 15, 2011

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?


Original Comment By: Anonymous
@vbabiy
vbabiy commented Mar 15, 2011

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

Carl: Cool, I will get started and update this ticket with the results.


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

While working on the upgrade command the following questions came up:

  • What methods should pip upgrade support to specify which packages to
    upgrade? Should we support a requirements file?

  • How should pip upgrade handle packages that are not already installed?
    Should we install missing packages?

pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

  • Tests

  • Filter requirements file; add non-editables to list of packages to
    upgrade.

Running pip using the upgrade command
# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Last set of questions (I hope):

  • Should pip upgrade parse the requirements file and filter out editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

  • Should pip upgrade search the indexes for a later version to install? (This is what I am doing now). If not how should the upgrade command determine if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

  • The package is not installed

  • An upgrade is available (later version from the indexes), and no explicit
    version was requested

  • The requested version is different from the installed one (Version miss
    match).

  • I am deferring the requirements file to the install command until I filter
    out the non-editable requirements.


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade
in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all
, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command
# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.


Original Comment By: Carl Meyer
@vbabiy
vbabiy commented Mar 15, 2011

Thanks Carl, I will wrap this up and start looking at #13


Original Comment By: Kelsey Hightower
@vbabiy
vbabiy commented Mar 15, 2011

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py


Original Comment By: Kelsey Hightower
@jedie
jedie commented Jul 7, 2011

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

@nisavid
nisavid commented Feb 21, 2012

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

@gimmi
gimmi commented Sep 15, 2012

+1

@fradeve
fradeve commented Sep 25, 2012

+1

@januz
januz commented Sep 28, 2012

+1!

@sebdah
sebdah commented Oct 4, 2012

+1

@nicolargo

+1

@jezdez
jezdez commented Oct 14, 2012

Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.

Instead I'd be thrilled to see a comment "patch done!" ;)

@gotlium
gotlium commented Nov 29, 2012

+1

@pradyunsg

Any Updates? Is there any plan to add this, it's 3 years old now..

@kelseyhightower

I thought this was merged already. I can dust off my python skill and give it another go.

@eproxus
eproxus commented Oct 22, 2013

Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...

@riyadparvez

+1

@chriskrycho

What is the status of this issue?

@eduardocereto

It is blocked by #988

@ariccio
ariccio commented Apr 26, 2014

If it's really important for you, there are workarounds for upgrading all packages; I threw together a script to do so in parallel(https://github.com/ariccio/update-pip-packages), and there are many other implementations elsewhere on the internet.

@rgommers

There's two parts to this issue. upgrade-all may be blocked by gh-988, but I don't see how upgrade is blocked. pip upgrade can be a simple alias for pip install -U --no-deps. This would resolve one of the main problems of using install_requires in setup.py files. Can't this be done sometime soon?

@qwcode
Python Packaging Authority member
qwcode commented Aug 17, 2014

pip upgrade can be a simple alias for pip install -U --no-deps

from the description:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

that is not pip install -U --no-deps

“non-recursive” in this context does not simply mean –no-deps. A non-recursive upgrade will upgrade dependencies, but only if needed to fulfill parent requirements.

@rgommers

@qwcode thanks for the clarification. Then that's not what I care about. Why would you call this "non-recursive", it's still recursive but just a bit smarter/different?

I was under the impression from the discussion in gh-571 that really non-recursive was the desired default. That would certainly make sense, and prevent having to always write code like scipy/scipy@8e7ee0c.

@qwcode
Python Packaging Authority member
qwcode commented Aug 17, 2014

in #571, "non-recursive" is not --no-deps it's the "smarter/different" recursive as you say.
notice the test_upgrade_with_needed_recursive_upgrades() test from that PR.

without getting stuck on terms, there's 3 things:

  1. the "smarter/different upgrade", i.e. the kind that occurs in OS packaging that can also imply upgrading dependencies (but only if they actually need upgrading).
  2. what pip does now, which is to upgrade all dependencies, regardless of need or conflict resolution.
  3. --no-deps

some possible ways to distinguish #1 and #2

  1. "non-recursive" vs "recursive"
  2. "normal" vs "recursive"
  3. "only if needed recursive" vs "do it regardless recursive"

I'm open to using the best terms... just not sure what those are.

@qwcode
Python Packaging Authority member
qwcode commented Aug 17, 2014

I think I'm liking this "only-if-needed recursive" phrase. maybe i should use that here in the docs:

@rgommers

I like it too. If you'd describe the three options all together as

a. non-recursive
b. only if needed recursive
c. recursive (or "do it regardless recursive")

that would be clear.

Then you want to pick good defaults. For both upgrade and install -U, either (a) or (b) could make sense. I strongly prefer (a), but (b) makes sense too given that that's what OS packaging does.

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U.

@rgommers

To clarify why I strongly prefer (a): an unsuspecting user wanting (b) and getting (a) will have to read a message saying "non-recursive upgrade can't satisfy all dependencies, please use only-if-needed recursive", which is not that big a deal. If the default would be (b), that unsuspecting user may end up with an upgrade or even a broken install of a package he really didn't want to be touching. That can take hours or even days to recover from.

@qwcode
Python Packaging Authority member
qwcode commented Aug 18, 2014

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U

the reason to leave install -U alone is just for compatibility reasons, and then to eventually deprecate it.

If the default would be (b), that unsuspecting user may end up with an upgrade
or even a broken install of a package he really didn't want to be touching

if a user wants necessary dependency upgrades to go unfulfilled, they should specifically have to ask for that using --no-deps. there's no way in my mind that could ever be the default. that behavior would do more damage than what you're considering (which is the outlier case). Over and over again, users would be left not having the dependency upgrades they needed.

@rgommers

Deprecating install -U sounds good.

I agree (b) is more common than (a). Even if it would be 100x more common, which is not the case I think, more damage is untrue. Reading a clear error message before install begins is so much better than for example a compile error halfway through, that imho (a) is still the better default.

Relying on --no-deps may be fine for very experienced users, but new users are only going to learn about it after things go wrong.

Anyway, looks like I won't be able to convince you. Then back to the manual

try:
   import dependency
except ImportError:
    install_requires.append('dependency')
else:
    if dependency.version < 'x.y.z':
        raise ValueError('Please upgrade dependency to x.y.z')

it is.

@rgommers

@qwcode thanks for the detailed explanation. I hope this will be moving forward in the near future - only if needed recursive would be a huge improvement over the current behavior.

@jseabold

Spending again many hours this week fielding issues on why we don't use install_requires, I'm adding a :+1: to moving away from the current behavior.

@andreabedini andreabedini added a commit to andreabedini/PyTables that referenced this issue Mar 22, 2015
@andreabedini andreabedini Remove install -U switch as they are being deprecated
See discussion in
pypa/pip#59 (comment)
9b9c10c
@bodokaiser

Are there any updates on this issue?

We have now 2015 and I still need to copy from pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U stackoverflow

@ariccio
ariccio commented Apr 16, 2015

Yeah, essentially all the progress is linked to this issue.

pip_upgrade_github

GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)

@bodokaiser
@qwcode
Python Packaging Authority member
qwcode commented Apr 16, 2015

as for pip upgrade-all, the consensus seems to be to wait for work related to #988 before releasing any shiny new commands that are still ultimately flawed and can be dangerous to a working environment. A simple version of upgrade-all that doesn't properly resolve requirements can easily break existing packages

@lomoalbert

+1

@muhasturk

What are you doing along 4 years?

+1

@yonyonson

+1

@RichieB2B RichieB2B referenced this issue in crits/crits Jun 11, 2015
Closed

Fix dependencies for CentOS 7 #511

@Csega
Csega commented Jul 1, 2015

+1

@pradyunsg

@muhasturk Currently, waiting... #59 (comment)

@petebachant

+1

@0x90shell

+1

@ryanpcmcquen

+1

@rasmusbrandt

+1

@cztchoice

+1

@romanpitak

+1

@Ir1sh
Ir1sh commented Sep 2, 2015

+1

@IgorFobia

+1

@ekevoo
ekevoo commented Sep 5, 2015

+1

@mwaskom mwaskom referenced this issue in lightning-viz/lightning-python Sep 10, 2015
Open

Improved dependency upgrade handling #31

@timmattison

+1

@ssbarnea

+1 ...sorry for the spam, but running pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U hurts!

@laike9m
laike9m commented Sep 28, 2015

+1

@njsmith
Python Packaging Authority member
njsmith commented Oct 11, 2015

In case anyone is interested in working on this, I think the comments above are somewhat confusing -- AFAICT the actual status is that pip upgrade-all is currently blocked by the need for a proper dependency resolution system, but pip upgrade could be implemented at any time. If someone wants to work on that part then it would be extremely awesome :-).

(See also the thread starting here and @dstufft's reply here and comment here agreeing with the above assessment.)

@qwcode
Python Packaging Authority member
qwcode commented Oct 11, 2015

here's another discussion from the pypa-dev list from a year ago (that agrees that pip upgrade FOO could be done now) https://groups.google.com/forum/#!searchin/pypa-dev/pip$20upgrade/pypa-dev/vVLmo1PevTg/oBkHCPBLb9YJ

@rgommers

Thanks @qwcode! I also just saw the new description at https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade, that's helpful.

@jkseppan jkseppan added a commit to jkseppan/pip that referenced this issue Oct 18, 2015
@jkseppan jkseppan Implement a "pip upgrade" command
Upgrade those requirements given directly but dependencies only as
needed to satisfy dependency relations. Inspired by Nathaniel Smith's
post at http://thread.gmane.org/gmane.comp.python.scientific.user/36377

See also: #59
39e0f27
@wodim
wodim commented Oct 24, 2015

+1

@xavfernandez

If I'm not mistaken:

If xxx is not installed:

  • pip upgrade xxx would be equivalent to pip install xxx
  • pip upgrade xxx --recursive would be the equivalent of pip install xxx --upgrade

If xxx is installed:

  • pip upgrade xxx --recursive would still be the equivalent of pip install xxx --upgrade (by design)
  • but there is currently no equivalent for pip upgrade xxx, this could be pip install xxx --upgrade-only-if-needed/--upgrade-lazy

It's not clear what the added value of a new command is over adding a new option to pip install ?

@rgommers

It's not clear what the added value of a new command is over adding a new option to pip install ?

The default pip install -U behavior is unacceptable to many projects. See my comment above (#59 (comment)). And here's a longer explanation: http://article.gmane.org/gmane.comp.python.distutils.devel/24218

@xavfernandez

If it is unacceptable, then I guess you plan to change your usage of pip install --upgrade to pip upgrade ?
Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?
What does a new command provide that a new option could not ?

@dstufft
Python Packaging Authority member
dstufft commented Oct 29, 2015

It's all paving cow paths. It's not @rgommers personal usage he's worried about, it's his users. People are going to reach for the "obvious" answer if they don't know better and right now the obvious answer is problematic for some major pieces of the Python ecosystem.

@rgommers

Indeed. People don't read docs. Indeed we'll fix all the install docs we can find, but as soon as a user sees -U or --upgrade that's what he'll use. The chance that people will use --no-deps or --only-as-needed or whatever before they've been seriously bitten by it is remote.

@rgommers

Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?

And to be clear: we avoid pip install --upgrade now and do not use install_requires because of this. That's how bad it is.

@njsmith
Python Packaging Authority member
njsmith commented Oct 29, 2015

I think there's a minor misunderstanding here -- IIUC @xavfernandez agrees with adding the non-recursive upgrade functionality, their question is just why the UI to that functionality has to be as a new top-level command instead of a new option to pip install.

@xavfernandez: Note that there's currently a discussion happening in #3194 about what would be the clearest UI for this functionality.

@njsmith
Python Packaging Authority member
njsmith commented Oct 29, 2015

(And pip install -U will be deprecated and then removed, so that answers the question about how users will know not to use it :-).)

@rgommers

I think there's a minor misunderstanding here

I'm pretty sure that there isn't. And yes, we're only debating UI. But that's important.

@xavfernandez

Yup only debating the UI and I agree it is important. But what I see coming is that once pip upgrade is out, there will be no reason to keep using pip install...
And the UI to install any package will become "pip upgrade foo"

@rgommers

there will be no reason to keep using pip install

which is a problem why? The issue (at least for @qwcode in comments above) with changing pip install --upgrade to the right behavior is breaking backwards compatibility. So pip upgrade is the way to avoid that break and get the right default.

@njsmith
Python Packaging Authority member
njsmith commented Oct 29, 2015

@rgommers: to be fair though, we will get some non-zero number of confused questions about why did you tell me to run pip upgrade foo when I don't even have foo installed, it didn't work (argue back and forth until they actually try running the command instead of assuming that it won't work and pretending that they ran it)

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

On #3194 I made the comment that maybe the right way forward is to just remove --upgrade from pip install, make pip install always do an upgrade of the explicitly named items, and default to a minimal upgrade strategy for dependencies with a --recursive flag to get the current behavior on upgrade.

I think I agree that I feel like with #3194 the default UX will become pip upgrade foo instead of pip install [-U] foo. The only problem I have with that is I think it's sort of confusing to have the two commands and given that I think pip upgrade would be the "right" one for people to use, I think it's crummy to have the obvious name be the wrong name.

I have a migraine so I could also just not be thinking entirely correct, but I'm sort of feeling like figuring out how to handle the deprecation path to that suggestion might be the right way forward.

@rgommers

Well, I won't be arguing that backwards compat is important here. I was always in favor of changing the behavior of pip install --upgrade. It's the cleanest, and the cost seems small. But it looked to me like it was vetoed by @qwcode, so pip upgrade was the next best thing (and already OK'ed by the pip devs a year ago).

If we now want to go back to changing defaults/behavior for pip install, great.

@rgommers

@dstufft that makes total sense to me - the headache can't be that bad:)

@xavfernandez

Since we agree to deprecate --upgrade, we could keep that and add two options to pip install: --upgrade-only-needed and --upgrade-eager , the second one being an alias for the deprecated --upgrade

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

@xavfernandez Forcing decisions onto users is kind of crummy I think. Unless I understand the fairly subtle issues I'm not sure that I would really know if I wanted --upgrade-only-needed and --upgrade-eager.

@rgommers

@xavfernandez so you're proposing that in the final situation the command would then be pip install --upgrade-only-needed? Seems ugly.

@chriskrycho

It would be much better to have something that maps to the equivalent of the semantic version pinning available in e.g. npm, hex, or Cargo. That would certainly need adjustment in the Python context, since (a) PEP 440 versions do not and cannot map precisely to semantic versioning and (b) the Python community at large doesn't necessarily hew to semantic-like versioning in its releases even within PEP 440.

Nonetheless, something along those lines would be very helpful, as would the notion of pinning a specified version.

Given the constraints, in the short term one viable option might be doing something analogous to homebrew's upgrade and upgrade --all commands, where the latter simply goes ahead with upgrading everything (potentially with a warning the first time).

In the long term, however, it would be enormously useful to create shared conventions about what versions mean in the Python packaging community, and building pip support around that could be a big part of it.

@njsmith
Python Packaging Authority member
njsmith commented Oct 30, 2015

The obvious migration path would be:

pip version X:

  • add options pip install --eager, pip install --no-eager, where --eager means that for requirements with no version restriction attached we attempt to install the latest version even if there is already some version installed, and --no-eager means that we're satisfied if any version is installed
  • also add option pip install --eager-recursive with the current -U semantics
  • --no-eager remains the default
  • if none of these options are specified, and pip install is passed a requirement with no attached version number, and this requirement is already satisfied, then output a warning saying that we didn't upgrade your package, but in the future we will, so you should use --no-eager if you want to keep the current behavior
  • add a deprecation warning to pip install -U referring people to pip install --eager-recursive

pip version Y:

  • switch the default for pip install to --eager

pip version Z:

  • remove pip install -U

I made Y and Z different versions because I am eager for --eager so maybe we could make that change on a more aggressive schedule, while keeping pip install -U around for a bit since a lot of documentation already refers to it, and it doesn't cause any harm. But obviously Y = Z is an option :-)

@njsmith
Python Packaging Authority member
njsmith commented Oct 30, 2015

@chriskrycho: That all sounds like great stuff to have, but this discussion is about how to handle the situation where the user is explicitly trying to request an upgrade to the latest version of a package, which is a situation that commonly happens already today and we don't have a good answer to. I don't know if there's a bug open requesting pinning support yet, but if not then maybe you should start one?

@chriskrycho
@rgommers

What's the normal deprecation path for pip? Time/version-based? How long does something need to be deprecated before removal?

@rgommers

By the way, I think "eager" is a word that users will not understand. recursive / only-as-needed recursive / always recursive is much clearer.

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

We typically do a two version deprecation cycle.

@rgommers

Major versions I assume? Minor versions are really fast (7.0 -> 7.1 was one month). Typically half a year between major versions, so 1 year of deprecation warnings?

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

Yea, sorry. Major versions. If we deprecate something in 8.x it will be a yellow warning for the rest of 8.x, a red warning for all of 9.x, and removed in 10.x.

@rgommers

Thanks.

Copying @dstufft's comment from #3194 (comment) here, because it's the best summary of the situation:

I guess the tl;dr is that I totally think that we need to fix the "default" behavior, we just need to settle on the specific implementation of fixing it:

  1. Keep the UX be pip install --upgrade and change the default.
  2. Move the UX to pip upgrade with the "right" default and add a --recursive flag.
  3. Move the UX to pip install, remove --upgrade, and add a --recursive flag.

My 2c:

(1) has a good UX, can be done now
(2) a bit worse UX (have both install and upgrade), can be done now
(3) second best UX, changing pip install can be done now, removing --upgrade will only be done in 10.0 (deprecated in 8.0 and 9.0).

I don't see why the backwards compat issue (which should be minor either way) for (1) would be worse than for (3). So (1) seems best. If bw compat is a real worry, then choose (2).

Anyway, time to call it a night.

@njsmith
Python Packaging Authority member
njsmith commented Oct 30, 2015

I think the difference between (3) and (1) comes down to whether we think that install-or-upgrade is the most common operation people want -- if so then making it the short command as in (3) is probably best. It's not a big deal either way though.

By the way, I think "eager" is a word that users will not understand. recursive / only-as-needed recursive / always recursive is much clearer.

I'm not at all attached to the "eager" spelling, but recursive just doesn't make sense as the main word for toggling upgrades on and off. I guess we could use --upgrade-non-recursive (future default), --upgrade-recursive, --upgrade-none or so, but it still foregrounds the confusing recursive behavior (I would have no idea what --upgrade-non-recursive meant if I weren't already familiar with the weird recursive behavior of the legacy -U option), and --upgrade-none misleadingly sounds like it will make sure that no packages are upgraded even if needed to preserve self-consistency.

If we go this route then I don't care that much about the spelling period, since the option that most people want would just be the default and the recursive-upgrade and no-upgrade options would both be special rarely used things that most users could ignore.

@rgommers

--upgrade-non-recursive (future default) ...

of course that's confusing. but you're ignoring the simplest option, which is not to provide this switch at all. because if it's identical to the default behavior, why would you ever need it?

@rgommers

I think the difference between (3) and (1) comes down to whether we think that install-or-upgrade is the most common operation people want

I'd expect "install". At least I know that I only upgrade packages that I heavily use on purpose, but each time I see something interesting/useful it's a simply pip install pkgname away.

(not that I'm a typical user though, so my expectation could be wrong)

@xavfernandez

I think I'd be in favor of:

  • No new pip upgrade that would be an alias for pip install
  • pip install (without --upgrade* option) behavior does not change
  • pip install --some_name would only try to upgrade the packages specified on the command line and not their dependencies
  • pip install --some_other_name to keep the old --upgrade behavior available

And then there are two options:

  • we don't need/want a deprecation path, then --some_name can be --upgrade and --some_other_name can be whatever seems best
  • we need a deprecation path, then --upgrade in pip8 will give a warning saying it is deprecated and we should use --some_other_name to keep the old behavior or rather switch to a safer --some_name to only upgrade the specified packages and their dependencies only if needed.

    In this second case, --some_name can not be --upgrade. So we have to find two new options names for --some_name and --some_other_name

@pfmoore
Python Packaging Authority member
pfmoore commented Oct 30, 2015

It seems to me that the obvious "best" solution is to keep pip install as the command, and to change the behaviour of --upgrade to not upgrade dependencies. The only real issue is backward compatibility, but have we had any users arguing that they rely on the current behaviour?

Personally, I'm inclined to the "go for it, and to heck with backward compatibility" view. I could argue that the current behaviour is actually a bug, and this change would be a bugfix. But I do think that at some point we need to be able to say that we've reached a good point, and we're going to take a much stricter view on backward compatibility. I don't think we're at that point yet, but we may need to start letting the community know what we feel still needs to be done in order to get to that point.

There is still (IMO) a need for some sort of pip install --upgrade :all: command to upgrade all installed packages. But that's new functionality, so isn't relevant here.

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

The current behavior is useful, particularly for projects like Pyramid which are not a single project but are actually a collection of projects. It'd be useful to be able to do something like pip install --upgrade-recursive Pyramid in order to upgrade Pyramid and all of it's dependencies to the latest version. Removing the recursive thing all together means I either have to manually track down all of the dependencies and do pip install --upgrade Pyramid dep1 dep2 dep3 dep4 (with the entire dependency tree) or I need to do a hypotheticalpip install --upgrade :all: and possibly upgrade more things than I actually want to upgrade. In #3194 at least one user mentioned that they would like the current behavior to be available via a flag because it's useful in some cases.

Is there any reason (other than backwards compatibility) to not make it so pip install implicitly does a "minimal" upgrade? I'm trying to figure out in what situation I would actually want it to only install and not do an upgrade (IOW, one where the latest version is fine if I don't already have it installed, but not if I do have it installed) and I'm having a problem coming up with a case where I would want the current behavior.

@qwcode
Python Packaging Authority member
qwcode commented Oct 30, 2015

I like the idea of a new pip upgrade [--recursive] command as in #3194 (and deprecating install --upgrade)

I have concerns about any choices that break compatibility or have complex deprecations or that burden an already option-heavy pip install with more changes or complexity. Also, I'm personally attracted to an "upgrade" command, that by default, just upgrades similar to how distro tools work. "install" installs, and "upgrade" upgrades...

I'm inclined to the "go for it, and to heck with backward compatibility" view

I'm not : ) at least for core logic like this.

make it so pip install implicitly does a "minimal" upgrade

for me, considering this seems like a non-starter, right? consider the # of automated build breakages that might occur. also, people get locked into a conceptual model at a certain point about what pip install is... and this would force a reload of a new mental model. people don't like that.

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

@qwcode I don't know, I don't think it'd be OK. The main problem with pip upgrade is either you remove the way for people to say "give me the latest version of this regardless of if it's installed or not" or you have two commands that do almost entirely the same thing (pip install and pip upgrade). There might be some breakage in the short term, but I'm a bit worried about the actual mental model of pip upgrade because I can see people just wide spread replacing their install instructions from pip install foo to pip upgrade foo (but why am I upgrading something that I don't have installed already?!).

@dstufft
Python Packaging Authority member
dstufft commented Oct 30, 2015

Er, I think it would be OK I mean.

@qwcode
Python Packaging Authority member
qwcode commented Oct 30, 2015

wide spread replacing their install instructions pip install foo to pip upgrade foo

roger, that's why I think upgrade should just upgrade. as for the "turd" --fill-in-missing, I'm not stuck on that, so don't take me to mean that I have to have that, but see below.

"give me the latest version of this regardless of if it's installed or not"

  • if I don't have FOO, then I pip install FOO and I get the latest version.
  • if I have FOO, then pip upgrade FOO and I get the latest version.

I think we're talking about 2 cases:

  1. I don't know whether I have FOO, and I want one command to get the latest FOO, whether I have it or not.
  2. I want one command to get the latest of FOO and BAR. I have FOO installed, but not BAR.

Do we owe users one command for these? I'd prefer simplicity and clarity in the commands over giving people shortcuts. But if users demand it, then that's where something like --fill-in-missing comes in.

@njsmith
Python Packaging Authority member
njsmith commented Oct 30, 2015

Also, I'm personally attracted to an "upgrade" command, that by default, just upgrades similar to how distro tools work. "install" installs, and "upgrade" upgrades...

Point of clarification: I just checked how three popular distro tools work, and AFAICT (partly this is based on man pages b/c I don't use all these myself):

  • apt:
    • apt install <pkg> performs an "upstall" -- either upgrades or installs depending on whether it's already installed or not.
    • apt upgrade upgrades all installed packages
    • apt upgrade <pkg> doesn't exist, the only way to upgrade a single package is apt install <pkg> (which also accepts various kinds of version specifiers)
  • yum:
    • yum install <pkg> performs an "upstall"
    • yum upgrade upgrades all installed packages
    • yum upgrade <pkg> upgrades a specified list of packages. I don't know what happens if they're not already installed.
  • homebrew:
    • brew install <pkg> performs an "upstall"
    • brew upgrade upgrades all packages
    • brew upgrade <pkg> upgrades a specified list of packages. I don't know what happens if they're not already installed.

So using install for upgrades is actually what distro tools do :-). Doesn't mean we have to as well; just a data point.

I don't know whether I have FOO, and I want one command to get the latest FOO, whether I have it or not.

I think the most obviously compelling use case for this is in documentation. Example: the django docs that I linked in the other thread that instruct people to run pip install -U <some package>. You don't want to tell people to "run pip install <pkg> except if you already have it installed run pip upgrade <pkg>" because that's way too confusing for newbies, but if install doesn't upgrade then just telling people to run pip install <pkg> is also going to create confusion where people start working through your tutorial with an old version installed and then have no idea why things don't work right and blame you for it. So you definitely need a single "upstall" command, and it needs to be simple and obvious to include in docs.

@chriskrycho

Follow-up on Homebrew's behavior for comparison: brew upgrade fails if you try to install a non-installed package:

$ brew upgrade git
Error: No such file or directory - /usr/local/Cellar/git

Also, the current brew upgrade behavior is changing (see discussion here); in a future release, it will switch to requiring --all to upgrade all installed packaged.

@qwcode
Python Packaging Authority member
qwcode commented Oct 31, 2015

Point of clarification

I knew someone would pick up on this. : )
I almost responded to myself. I was focused on upgrades.

yum upgrade [...] I don't know what happens if they're not already installed.

it doesn't do anything

yum install performs an "upstall"

true, but the default behavior is to prompt for confirmation, which pip doesn't have.

the django docs that I linked in the other thread

to be clear, they weren't the actual Django docs. the Django docs say pip install Django (https://docs.djangoproject.com/en/1.8/topics/install/)

it's actually a bad thing I think, given the way -U currently works, to get people in the habit of using install -U.

in your linked example, it was for redis which doesn't have any dependencies I think, so it's ok in this case.

going to create confusion where people start working through your tutorial
with an old version [...] So you definitely need a single "upstall" command

IDK, It's not so obvious to me that tutorials should be telling people to upstall in general. If someone is doing a tutorial for FOO, and they already have FOO installed, then maybe they shouldn't be blindly "upstall"ing.... maybe an upgrade would damage the existing environment.

@dstufft
Python Packaging Authority member
dstufft commented Oct 31, 2015

I think the tutorials should be telling people to "upstall" (great term!) because a fair number of them are written by the projects themselves and when people go to look at tutorials they rarely look first to see what the existing version of FOO they have installed is, rather they tend to go look at whatever the latest docs are.

I also agree that the way -U currently works, it's a bad habit to get people in the habit of using it indiscriminately, but the key thing I think is withe the proposed behavior I don't think it's a bad habit, but rather a good thing to do. If you need to ensure a particular version is installed than you should already be using == and we can add a flag to turn the default "upstall" back into a plain "install" for people who need to make sure that the currently installed version OR the latest version is installed (though I still can't come up with a scenario in which this is actually what anyone wants).

@qwcode
Python Packaging Authority member
qwcode commented Oct 31, 2015

I think the tutorials should be telling people to "upstall"

again, though, what is this environment that they're in, where it's obviously ok to upgrade? If it's a distro system python environment, then certainly not. If it's some other environment, they made for an application, then an upstall could be damaging. if it's an environment they just made for the tutorial, then they don't need to upgrade.

a key point here is that PyPI is not a "curated" repo, like distro repos are. you can be fairly assured that an "upstall" is safe from a default distro repository... you don't know that with PyPI. Every upgrade is a gamble somewhat. "install", as it is, is conservative.

@qwcode
Python Packaging Authority member
qwcode commented Oct 31, 2015

keep in mind that making pip install act like an "upstall" w/o fixing #988 will break things.

consider where A and B are installed and A requires B<2. Now I go run the new pip install B, which upstalls B to v3 or whatever (due to not considering the installed constraints), and now my environment is busted.

@rgommers

for people who need to make sure that the currently installed version OR the latest version is installed (though I still can't come up with a scenario in which this is actually what anyone wants).

It's trivial to come up with such scenarios. Example (not made up unfortunately): currently statsmodels is broken by the latest pandas (0.17.0). I'm a user of both. I have a bunch of Python versions installed and venvs lying around. So I know that for any Python/venv I'd like to install pandas only if it's not there yet. If it is, I will probably also have statsmodels and then "upstall" will break it.

@rgommers rgommers referenced this issue in numpy/numpy Nov 1, 2015
Open

Install and packaging improvements todo list #6599

12 of 18 tasks complete
@dstufft
Python Packaging Authority member
dstufft commented Nov 1, 2015

@rgommers Hmm, so you're absolutely sure that you don't have a pandas 0.17.0 in a virtual environment anywhere? I guess it could happen though I think I'd probably spell that by doing pip install pandas!=0.17.0 or pip install pandas<0.17 but I guess it's not entirely unreasonable to rely on what you already have installed.

@rgommers
rgommers commented Nov 1, 2015

Hmm, so you're absolutely sure that you don't have a pandas 0.17.0 in a virtual environment anywhere?

That's not what I said. I do have a venv with pandas 0.17.0, just one in which I don't have statsmodels. pip install pandas<0.17 will do the job, but it hadn't occurred to me to use that (mainly because I didn't need to, the current install command works for this purpose).

Anyway, I don't have much of a preference for or against upstall. I just wanted to point out a realistic scenario when you said you couldn't come up with one.

@Dundee
Dundee commented Nov 25, 2015

+1

@rgommers

This discussion seems to have stalled without coming to a conclusion. Any option on the table is a major improvement on the current status. It looks like all the relevant pros and cons have been considered; there's only a difference of opinion on the relative importance of backwards compat vs. the optimal API. Maybe the pip developers can attempt to finalize a decision?

@rgommers

@rgommers they probably did decision? https://pip.pypa.io/en/stable/user_guide/#only-if-needed-recursive-upgrade

@Liso77 thanks, but no - it really is the discussion on this thread that needs finalizing. That documentation you linked to just points back here.

@Liso77
Liso77 commented Dec 22, 2015

@rgommers from link I posted above I read that developers are going to make upgrade command. And don't plan to add "only if needed" option to install command because they didn't create term for this option and use only description in quotation marks. So from your (or dstuff's) comment on 30 october - it is: 2. Move the UX to pip upgrade with the "right" default... (btw I prefer this)
I am sure you were thinking more about this topic and you see more subtle thing that I do. So please write what (if you think that something) is still open. If you meant that it could be good that developers write here explicitly their decision and eventually close this issue - I agree.

Btw. situation could be much less problematic if we had something like

pip freeze > checkpoint.txt
pip checkout checkpoint.txt

which could guarantee revert installation to any "checkpoint". (pip install -r is not good in this moment) I was thinking to add this proposal in new issue but I am sure I have to study and analyze more to bring really useful proposal. I see there are many caveats but I really like to have possibility to

pip checkout git+https://github.com/somebody_trusted/pip_distro4ubuntu@bleeding_egde_branch
#or
pip checkout git+https://github.com/somebody_trusted/pip_distro4ubuntu@stable

I mean that could solve (with somebody_trusted (or community with lot of unit tests) help) something like pandas and statsmodels problem you described above.

(useful to see for this topic is also: https://pypi.python.org/pypi/peep)

@rgommers

@Liso77 it's simple: the discussion on this issue is not finished. There is a PR waiting with the upgrade command implemented (gh-3194), but the pip devs (at least qwcode and @dstufft, and probably also @xavfernandez and @pfmoore) need to decide whether that's what they want, or they want to change the behavior of pip install instead. Until that happens, nothing will be changed.

@PhillipSz

+1

@FichteFoll

So, it appears that introducing the new upgrade command was agreed on and the latest unsettled debate is whether or not there should be an "upstall" command and if yes, which main command (install or upgrade) should get it as a switch or default.

Throwing in my personal opinions:

I think having a fail-safe singular command for upstalling is "worth it", if only for cases where you want to verify programmatically that an environment is up to date for any reason. Why "programmatically"? Because it is not interactive, i.e. there is no user to react to an error, warning or other message pointing him to the brother or sister command because a package does or does not exist already.
Optionally, a prompt could be added asking the user whether he wants to upgrade or install instead of just complaining.

That said, this "upstall" command should not be the default of either command (install/upgrade) and thus require an option to be invoked. This adds clarity to the semantically clearly defined "install" and "upgrade" verbs, and in my opinion all package managers that upstall by default on one of these are doing it wrong. Prompting for action (as yum appears to do) is alright.

This leaves us with upgrade --install (shorter than --install-missing) and install --upgrade. Semantically, "upgrade to latest [and install if doesn't exist]" and "install latest [or upgrade to latest if already exists]". Not much of a difference here imo. install --upgrade is the already existing version and would be recognizable, but backwards-incompatible.
Here's another one for the mix: A new command named install-upgrade (or upstall?), where neither install nor upgrade get the option to "also do the other" but a new command is introduced with clear semantics in an important position: the command name itself. This would be my preference.

TL;DR: Introduce upgrade and upgrade-install, deprecate install --upgrade. All of the functionality with clear semantics.
Add messages or optionally prompts to install and upgrade commands if their operations fail because of a package already or not existing.

@ilevkivskyi

Did this again manually this morning. Are there any news on this feature request?

@danningchen

+1

@edouardberthe

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.