Add "upgrade" and "upgrade-all" commands #59

Open
vbabiy opened this Issue Mar 15, 2011 · 227 comments

Projects

None yet
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

new description from @qwcode


pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

pip upgrade-all would upgrade all installed packages.

For history's sake, note that #571 was an attempt to change the default behavior of pip install --upgrade to non-recursive by default. It has a lot of work in there, but ultimately, it was decided not to break compatibility, and put the energy towards a new pip upgrade. the current --upgrade flag would likely be deprecated.

@vbabiy
Contributor
vbabiy commented Mar 15, 2011

"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.

For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.


Original Comment By: Jannis Leidel
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.

upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in <sys.prefix>/lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).

If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.

I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.


Original Comment By: Ian Bicking
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Yeah, you're right that can_uninstall and can_upgrade are different.

I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

+1 for this bug


Original Comment By: smyrman
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Issue #167 was marked as a duplicate of this issue.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

1

(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip

install -U

This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.

Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).


Original Comment By: Tomasz Elendt
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

+1 for upgrade --all

Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?


Original Comment By: Anonymous
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

I wouldn't mind taking a shot at an implementation, but first a few questions.

Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?

Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Carl: Cool, I will get started and update this ticket with the results.


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

While working on the upgrade command the following questions came up:

  • What methods should pip upgrade support to specify which packages to
    upgrade? Should we support a requirements file?
  • How should pip upgrade handle packages that are not already installed?
    Should we install missing packages?
pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package

Try and locate a package that satisfies the requirement. If the

requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.

# pip upgrade --all

Locate all installed packages (non-editables) and update them to a new

version if available.

# pip upgrade some_other_package

Warning: some_other_package not installed, use pip install

some_other_package.

My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.

Thoughts?


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.

Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Got most of the code done, just a little polish left before I post the code
for review.

TODO:

  • Tests
  • Filter requirements file; add non-editables to list of packages to
    upgrade.
Running pip using the upgrade command
# pip upgrade --all

All packages up-to-date


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML

Package updates available:

  PyYAML: N/A (installed) 3.09 (latest)

Downloading/unpacking PyYAML

  Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  PyYAML: 3.09 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

All packages up-to-date


# pip upgrade PyYAML==3.08

Downloading/unpacking PyYAML==3.08

....

Successfully installed PyYAML

Cleaning up...


# pip upgrade --all -v

Packages installed at latest version:

  pip: 0.8.2 (latest)

  distribute: 0.6.14 (latest)

  Python: 2.7.1 (latest)

  wsgiref: 0.1.2 (latest)

Package updates available:

  PyYAML: 3.08 (installed) 3.09 (latest)

Downloading/unpacking PyYAML

...

Successfully installed PyYAML

Cleaning up...

  Removing temporary dir /root/upgrade_env/build...

Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Last set of questions (I hope):

  • Should pip upgrade parse the requirements file and filter out
    editables? What about URL requirements?

For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.

Any tips are welcome (currently looking at pip.req.parse_requirements)

  • Should pip upgrade search the indexes for a later version to install?
    (This is what I am doing now). If not how should the upgrade command determine
    if there is an upgrade?

Right now I am only adding packages to the upgrade list when:

  • The package is not installed
  • An upgrade is available (later version from the indexes), and no explicit
    version was requested
  • The requested version is different from the installed one (Version miss
    match).
  • I am deferring the requirements file to the install command until I filter
    out the non-editable requirements.

Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.

The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).

I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.

https://bitbucket.org/khightower/pip/changeset/2bdc202b446c


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade
in functionality.

It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all
, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)

Running pip upgrade command
# pip upgrade Mako


Downloading/unpacking Mako

  Running setup.py egg_info for package Mako


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)

  Running setup.py egg_info for package MarkupSafe


Installing collected packages: Mako, MarkupSafe

  Found existing installation: Mako 0.3.6

    Uninstalling Mako:

      Successfully uninstalled Mako

  Running setup.py install for Mako

    changing mode of build/scripts-2.7/mako-render from 644 to 755


    warning: no files found matching '*.jpg' under directory 'doc'

    warning: no files found matching '*.sty' under directory 'doc'

    warning: no files found matching 'autohandler' under directory 'doc'

    warning: no files found matching '*.xml' under directory 'examples'

    warning: no files found matching '*.mako' under directory 'examples'

    warning: no files found matching '*.dat' under directory 'test'

    warning: no files found matching 'ez_setup.py'

    changing mode of /root/upgrade_env/bin/mako-render to 755

  Found existing installation: MarkupSafe 0.11

    Uninstalling MarkupSafe:

      Successfully uninstalled MarkupSafe

  Running setup.py install for MarkupSafe


    building 'markupsafe._speedups' extension

    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall

-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o

    gcc -pthread -shared

build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so

Successfully installed Mako MarkupSafe

Cleaning up...

Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.

With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.


Original Comment By: Carl Meyer
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

Thanks Carl, I will wrap this up and start looking at #13


Original Comment By: Kelsey Hightower
@vbabiy
Contributor
vbabiy commented Mar 15, 2011

If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.

https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py


Original Comment By: Kelsey Hightower
@jedie
jedie commented Jul 7, 2011

@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:

#313
jedie@7a31d70

@nisavid
nisavid commented Feb 21, 2012

@jedie i think you meant to direct your comment at @khightower. @vbabiy migrated her comment to here but did not write the upgrade command.

@gimmi
gimmi commented Sep 15, 2012

+1

@fradeve
fradeve commented Sep 25, 2012

+1

@januz
januz commented Sep 28, 2012

+1!

@sebdah
sebdah commented Oct 4, 2012

+1

@nicolargo

+1

@jezdez
Contributor
jezdez commented Oct 14, 2012

Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.

Instead I'd be thrilled to see a comment "patch done!" ;)

@gotlium
gotlium commented Nov 29, 2012

+1

@pradyunsg
Contributor

Any Updates? Is there any plan to add this, it's 3 years old now..

@kelseyhightower
Contributor

I thought this was merged already. I can dust off my python skill and give it another go.

@eproxus
eproxus commented Oct 22, 2013

Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...

@riyadparvez

+1

@chriskrycho

What is the status of this issue?

@eduardocereto

It is blocked by #988

@ariccio
ariccio commented Apr 26, 2014

If it's really important for you, there are workarounds for upgrading all packages; I threw together a script to do so in parallel(https://github.com/ariccio/update-pip-packages), and there are many other implementations elsewhere on the internet.

@rgommers

There's two parts to this issue. upgrade-all may be blocked by gh-988, but I don't see how upgrade is blocked. pip upgrade can be a simple alias for pip install -U --no-deps. This would resolve one of the main problems of using install_requires in setup.py files. Can't this be done sometime soon?

@qwcode
Collaborator
qwcode commented Aug 17, 2014

pip upgrade can be a simple alias for pip install -U --no-deps

from the description:

pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here

that is not pip install -U --no-deps

“non-recursive” in this context does not simply mean –no-deps. A non-recursive upgrade will upgrade dependencies, but only if needed to fulfill parent requirements.

@rgommers

@qwcode thanks for the clarification. Then that's not what I care about. Why would you call this "non-recursive", it's still recursive but just a bit smarter/different?

I was under the impression from the discussion in gh-571 that really non-recursive was the desired default. That would certainly make sense, and prevent having to always write code like scipy/scipy@8e7ee0c.

@qwcode
Collaborator
qwcode commented Aug 17, 2014

in #571, "non-recursive" is not --no-deps it's the "smarter/different" recursive as you say.
notice the test_upgrade_with_needed_recursive_upgrades() test from that PR.

without getting stuck on terms, there's 3 things:

  1. the "smarter/different upgrade", i.e. the kind that occurs in OS packaging that can also imply upgrading dependencies (but only if they actually need upgrading).
  2. what pip does now, which is to upgrade all dependencies, regardless of need or conflict resolution.
  3. --no-deps

some possible ways to distinguish #1 and #2

  1. "non-recursive" vs "recursive"
  2. "normal" vs "recursive"
  3. "only if needed recursive" vs "do it regardless recursive"

I'm open to using the best terms... just not sure what those are.

@qwcode
Collaborator
qwcode commented Aug 17, 2014

I think I'm liking this "only-if-needed recursive" phrase. maybe i should use that here in the docs:

@rgommers

I like it too. If you'd describe the three options all together as

a. non-recursive
b. only if needed recursive
c. recursive (or "do it regardless recursive")

that would be clear.

Then you want to pick good defaults. For both upgrade and install -U, either (a) or (b) could make sense. I strongly prefer (a), but (b) makes sense too given that that's what OS packaging does.

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U.

@rgommers

To clarify why I strongly prefer (a): an unsuspecting user wanting (b) and getting (a) will have to read a message saying "non-recursive upgrade can't satisfy all dependencies, please use only-if-needed recursive", which is not that big a deal. If the default would be (b), that unsuspecting user may end up with an upgrade or even a broken install of a package he really didn't want to be touching. That can take hours or even days to recover from.

@qwcode
Collaborator
qwcode commented Aug 18, 2014

(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U

the reason to leave install -U alone is just for compatibility reasons, and then to eventually deprecate it.

If the default would be (b), that unsuspecting user may end up with an upgrade
or even a broken install of a package he really didn't want to be touching

if a user wants necessary dependency upgrades to go unfulfilled, they should specifically have to ask for that using --no-deps. there's no way in my mind that could ever be the default. that behavior would do more damage than what you're considering (which is the outlier case). Over and over again, users would be left not having the dependency upgrades they needed.

@rgommers

Deprecating install -U sounds good.

I agree (b) is more common than (a). Even if it would be 100x more common, which is not the case I think, more damage is untrue. Reading a clear error message before install begins is so much better than for example a compile error halfway through, that imho (a) is still the better default.

Relying on --no-deps may be fine for very experienced users, but new users are only going to learn about it after things go wrong.

Anyway, looks like I won't be able to convince you. Then back to the manual

try:
   import dependency
except ImportError:
    install_requires.append('dependency')
else:
    if dependency.version < 'x.y.z':
        raise ValueError('Please upgrade dependency to x.y.z')

it is.

@rgommers

@qwcode thanks for the detailed explanation. I hope this will be moving forward in the near future - only if needed recursive would be a huge improvement over the current behavior.

@jseabold

Spending again many hours this week fielding issues on why we don't use install_requires, I'm adding a 👍 to moving away from the current behavior.

@andreabedini andreabedini added a commit to andreabedini/PyTables that referenced this issue Mar 22, 2015
@andreabedini andreabedini Remove install -U switch as they are being deprecated
See discussion in
pypa/pip#59 (comment)
9b9c10c
@bodokaiser

Are there any updates on this issue?

We have now 2015 and I still need to copy from pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U stackoverflow

@ariccio
ariccio commented Apr 16, 2015

Yeah, essentially all the progress is linked to this issue.

pip_upgrade_github

GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)

@bodokaiser

Yeah for sure but I don't get the reason why this "simple" feature has such a delay.

What is the reason for this?

Am 16.04.2015 um 05:28 schrieb Alexander Riccio notifications@github.com:

Yeah, essentially all the progress is linked to this issue.

GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)


Reply to this email directly or view it on GitHub.

@qwcode
Collaborator
qwcode commented Apr 16, 2015

as for pip upgrade-all, the consensus seems to be to wait for work related to #988 before releasing any shiny new commands that are still ultimately flawed and can be dangerous to a working environment. A simple version of upgrade-all that doesn't properly resolve requirements can easily break existing packages

@lomoalbert

+1

@muhasturk

What are you doing along 4 years?

+1

@yonyonson

+1

@RichieB2B RichieB2B referenced this issue in crits/crits Jun 11, 2015
Closed

Fix dependencies for CentOS 7 #511

@Csega
Csega commented Jul 1, 2015

+1

@pradyunsg
Contributor

@muhasturk Currently, waiting... #59 (comment)

@petebachant

+1

@0x90shell

+1

@ryanpcmcquen

+1

@rasmusbrandt

+1

@cztchoice

+1

@romanpitak

+1

@Ir1sh
Ir1sh commented Sep 2, 2015

+1

@IgorFobia

+1

@ekevoo
ekevoo commented Sep 5, 2015

+1

@mwaskom mwaskom referenced this issue in lightning-viz/lightning-python Sep 10, 2015
Open

Improved dependency upgrade handling #31

@timmattison

+1

@ssbarnea
Contributor

+1 ...sorry for the spam, but running pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U hurts!

@laike9m
laike9m commented Sep 28, 2015

+1

@njsmith
Member
njsmith commented Oct 11, 2015

In case anyone is interested in working on this, I think the comments above are somewhat confusing -- AFAICT the actual status is that pip upgrade-all is currently blocked by the need for a proper dependency resolution system, but pip upgrade could be implemented at any time. If someone wants to work on that part then it would be extremely awesome :-).

(See also the thread starting here and @dstufft's reply here and comment here agreeing with the above assessment.)

@qwcode
Collaborator
qwcode commented Oct 11, 2015

here's another discussion from the pypa-dev list from a year ago (that agrees that pip upgrade FOO could be done now) https://groups.google.com/forum/#!searchin/pypa-dev/pip$20upgrade/pypa-dev/vVLmo1PevTg/oBkHCPBLb9YJ

@rgommers

Thanks @qwcode! I also just saw the new description at https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade, that's helpful.

@RonnyPfannschmidt
Contributor

since pip has a different QA model than linux distributions, i'm fairly certain, a random upgrade all command is just a breakage in the making

pypi has no database of package versions and compatibility that is actually verified and tested,
so unlike a distribution with QA pypi has practically no own QA and relies completely on the projects uploaded to it (which vary in quality)

with the current sets of constraints i'm convinced that its more of a curse than a blessing
and if anyone wonders why distributions are so outdated, QA takes actual time and effort ^^

@pfmoore
Member
pfmoore commented May 6, 2016

Everyone, please use the thumbs up reaction on the OP and stop spamming everyone's inbox.

Note that we don't actually need any sort of "I'd like this" response from people. We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

So personally, I'd appreciate it if people would refrain from pinging this issue unless they have working code that at least offers a starting point for the implementation - and they are willing to follow it through to implementation.

Otherwise, we'll get to it when we can. I personally hit this issue regularly, and I'm a core pip developer, so I can guarantee that if I ever have sufficient time myself to work on this I will. In the meantime, "any progress?" comments tend to simply demotivate me because they mostly just feel like people demanding the right to dictate how I spend my hobby time...

@sfriesel
sfriesel commented May 6, 2016

a starting point for the implementation

I thought that was the purpose of #3194? From there on the discussion drifted into several alternatives for command naming and behavior without a consensus by the core developers. Would it help to implement multiple of those alternatives? Or does #3194 just need a refresh?

@pfmoore
Member
pfmoore commented May 6, 2016

@sfriesel You missed out "and are willing to follow it through". Looks like the person who made that PR either ran out of time or interest (a common problem). The next step on that one is probably for either the OP or someone else who's willing to take over ownership of the PR, to write up a PEP-style summary of the current position, what the various outstanding issues are, what he's suggesting as the resolution for these various issues, and a call for comments to restart the discussion. If the debate is too big for github, maybe he needs to take his document to distutils-sig for more input. If he can't decide on a resolution that he's happy with for the various issues, or can't get consensus from the debate, then yes, #3194 is probably dead (until someone else picks up the problem again, at which point I'd hope they incorporate all the work done their into their new PR).

Well over 50% of the work on a proposal like this one is "management" (documenting proposals, managing discussions, drawing out consensus). It's not necessarily the bit people like to do, but with a codebase as widely used as pip, it's an essential part. (And yes, that's a problem - it's much nicer to say "I write code as a hobby" than "I do project management as a hobby" :-))

@pradyunsg
Contributor
pradyunsg commented May 7, 2016 edited

@pfmoore Totally agree.

The number of proposals and amount of discussion, spread across many locations, related this issue over the years makes it extremely difficult to approach this issue and get a clear picture in a short while.


@RonnyPfannschmidt

pypi has no database of package versions

It's true. This is information provided by the packages at run-time (of the setup.py) and it adds a bit more complexity to the process of resolving the dependencies. Moreover, due to how the current logic behaves, some packages (even prominent ones like scipy) don't list a dependency (like numpy) if it is installed to avoid re-compiling it unnecessarily.

That would be solved by PEP 426, among other things but I don't know what's the status of that PEP.

@pradyunsg
Contributor

@pfmoore

Note that we don't actually need any sort of "I'd like this" response from people

It's just tempting to say +1 or put a thumbs up to put it out there that you would like to see this happen. It lets the person feel like they have added a little something to this issue, contributed to it. I'm not saying that pip devs need to be motivated or persuaded, it's just that everyone wants to say that. And...

I'd appreciate it if people would refrain from pinging this issue unless they have working code

Putting a thumbs up is not going to give anyone (annoying) notifications/emails and we still get to see the number of people who have shown interest. I don't think anyone should mind that but I could be wrong.

@pfmoore
Member
pfmoore commented May 7, 2016

Putting a thumbs up is not going to give anyone (annoying) notifications/emails and we still get to see the number of people who have shown interest. I don't think anyone should mind that but I could be wrong.

Ah, I hadn't realised that doing that avoided the notifications. Yes, that's a good idea.

@pradyunsg
Contributor
pradyunsg commented May 9, 2016 edited

The next step on that one is probably for either the OP or someone else who's willing to take over ownership of the PR, to write up a PEP-style summary of the current position, what the various outstanding issues are, what he's suggesting as the resolution for these various issues, and a call for comments to restart the discussion.

For what it's worth, I was earlier doing a write-up on this and dependency resolution, PEP style. I'm going split it up and update it now, adding all the things that have been proposed in the past few months. I'll post the one for upgrade command as a Gist once it's done ~(2 days?) and link to it from here.

I'm not willing to take over ownership of the PR just yet. I intend to help reignite the discussions and get to a final design that can be implemented. If some stuff around me goes to plan, I should be able to follow this through to implementation but I don't want to commit to it following this through just yet.

@pradyunsg
Contributor

/cc @qwcode @dstufft @pfmoore @xavfernandez

Here's the link to my writeup: https://gist.github.com/pradyunsg/a5abeac4af90fbdc54bb266c32c0d2d8

I initially just linked to various places (~30 links) for the reader to refer to and tried to copy-then-edit-to-fit all the comment threads. The document was pretty long and my opinions slipped through. I ended up starting from scratch. It's shorter now though and not opinionated. I've still marked it as WIP though.

If there's any issues in it, please comment on the Gist. I'll make the corrections as soon as I can.

@pradyunsg
Contributor
pradyunsg commented May 15, 2016 edited

WARNING: SLIGHTLY LONG COMMENT AHEAD


I like the idea of fixing the existing --upgrade behaviour. Are there any concerns regarding this other than backwards compatibility? Adding an upgrade command doesn't seem like a nice idea, due to the high overlap between install and upgrade.

FWIW, git did something similar to what we're discussing about --upgrade, changing the default behaviour of git push in 2014. link

If there's interest in changing the behaviour of pip install --upgrade, here's a seeding proposal (refer to this as P1) for that discussion:

  • Starting next major version (9.0), pip install --upgrade prints a warning:

    PipDeprecationWarning: pip will stop upgrading dependencies to their latest
                           version unconditionally by default starting version 11.0.
    
    To maintain current behaviour after the default behaviour changes, to
    the install command pass `--upgrade-strategy eager`.
    
    To start using the new behaviour immediately, to the install command
    pass `--upgrade-strategy non-eager`.
    
    To quelch this message, <perform some action related to config file>.
    
    For more details, read <link-to-documentation-resource>.
    

    The documentation page linked would provide a quick overview of the change and it's effects. It would then address the most common question: "What should I do?" and "Which upgrade strategy should I use?" in the most direct manner reasonable.

    • The message could use a cleanup.
    • The option could even be 2 flags, --upgrade-eager and --upgrade-non-eager or something. That's bikeshedding, i.e. last step.
  • Unless something drastic happens, in version 11.0, pip install --upgrade switches behaviour to be non-eager when upgrading dependencies.

Edit: I put a 3 major version cycle for the change, just to give more time to people to switch over. It might be unnecessary. Maybe, 1 major version printing warnings and the next making the change is enough?

@dstufft had come up with a similar (same?) idea.


If an upgrade command is the way forward, here's a seeding proposal for reinitiating that discussion (refer to this as P2, if you want to):

  1. pip upgrade copies over most options and behaviour from pip install.
  2. --dry-run flag will be added to install and upgrade commands that print what pip will try if it were to actually run.
  3. pip install <out_of_date_pkg> does not change behaviour.
  4. pip upgrade <not_installed_pkg> would fail.
  5. pip upgrade <up_to_date_pkg> would do nothing, say why it did nothing and exit with a zero exit code.
  6. pip upgrade <out_of_date_pkg> would perform a non-eager recursive upgrade.
    • Default to non-eager recursive upgrade
  7. pip upgrade --eager package would behave like today's pip install --upgrade.
    • Allow opt-in to current behaviour
  8. pip install --upgrade would print a RemovedInPip10Warning.
    Because it would be removed.

Additionally, what are your thoughts on:

  1. Making the upgrade command interactive? (something like what apt-get does)
    • making pip install <pkg> behave like OS package manager's install i.e.upstall?
  2. Having --only-binary by default in the new upgrade command?
    • If it's ok, what about adding --only-source and --no-source in upgrade command?
  3. Adding an option to "hold" the update of a package? IMO, this is a requirement for upgrade-all.
  4. Does it make sense to distingush between these cases i.e. require different CLI?
    • install: not-installed -> up-to-date
    • upgrade: out-of-date -> up-to-date
    • upstall: {not-installed, out-of-date} -> up-to-date
@pradyunsg
Contributor
pradyunsg commented May 15, 2016 edited

I didn't propose anything about the "install as upstall" because I do not feel strongly either way about it. It is consistent with OS-level package managers but would be a major breaking change. I'm not sure how useful it would be or what the exact behavior and interface would be like...

Maybe sleeping over this might help.

@ekevoo
ekevoo commented May 17, 2016

Personally I don't see much value in keeping consistent with a behaviour few if any people are happy about. I wonder if having one for Python 2 (embracing stability) and another for Python 3 (embracing change) would be better or worse.

@pradyunsg
Contributor
pradyunsg commented May 17, 2016 edited

@ekevoo

I wonder if having one for Python 2 (embracing stability) and another for Python 3 (embracing change) would be better or worse.

One, That's off-topic. Two, if you do so, there will end up being divergence. That'll alienate people currently using Python 2 because it'll (hopefully) reach EOL one day and they'll have to shift over to Python 3, having to adapt for a differently behaving tool. Thus, pip shouldn't do divergence (and won't, IMO). Might as well keep it together in a single tool then.

OTOH, I do see what you think about how much backward compatibility is acceptable...

@xavfernandez
Collaborator

A warning in pip 9, plus additonal options to tweak the behavior of pip install --upgrade, --eager/non-eager (or some other name) seems fine to me.

@pradyunsg
Contributor

Bump. Another notification for everyone.

@pfmoore @dstufft @qwcode I don't want to be nosy but could you please take out some time for this, over the weekend or coming week?

@pfmoore
Member
pfmoore commented Jun 3, 2016

OK, so my comments:

Re the two ways forwards, I don't have an opinion. Pick whichever one you feel is the better option, implement it and we'll see how it goes from there.

Re your additional questions:

  1. I'm -1 on making the command interactive. Unless there's a specific issue that the user needs to make a decision about, along the lines of "this consequence of what you asked for is something you were clearly unaware of, and could be a problem - do you want to proceed" then it's just useless noise. I'm against "are you sure" prompts in general. If you have a specific proposal for a prompt you think is worth adding then feel free to ask again with the details included.
  2. We shouldn't make upgrades different from installs, so no to --only-binary by default. For my personal use, I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.
  3. I'm not sure what you mean about "holding" updates. Please clarify. But as a general response, I'd say don't bother in the first instance, let's keep the initial PR free from "optional extras" - they can be added later.
  4. Isn't that the main topic of debate on the various discussions of this feature? For a start, doesn't it affect the basic question of install --upgrade vs upgrade? I don't recall there having been a resolution of that question, so I'm a little surprised you expect a "simple" answer to this one? I certainly don't have a good answer.

(By the way, I don't have easy access to the gist, so if there is any content on there that isn't in your summary, I've missed it sorry).

I know it's hard work to write a PR, and demotivating to have it seem to get bogged down in debates and questions, but ultimately, I think someone is going to have to write one, and put it out there as "this is the solution I propose - any comments?"

@pradyunsg
Contributor

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.

If you probably always want something, it should be default1. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

I'm not sure what you mean about "holding" updates.

I'll let this rest until we get to the point of talking about adding upgrade-all functionality.

note to self: apt-mark concepts for pip

I'm -1 on making the command interactive. Unless there's a specific issue that the user needs to make a decision about

Ditto.

Isn't that the main topic of debate on the various discussions of this feature?

Exactly why I wanted to know what you guys think.

I don't recall there having been a resolution of that question, so I'm a little surprised you expect a "simple" answer to this one?

I don't expect a "simple" answer just yet. I'm hoping we arrive at one, preferably simple, answer through discussion.

By the way, I don't have easy access to the gist, so if there is any content on there that isn't in your summary, I've missed it sorry

More than a summary, it's a follow up to the write-up. Please do read it as soon as you can do so.

Pick whichever one you feel is the better option, implement it and we'll see how it goes from there.
[snip]
I think someone is going to have to write one [PR], and put it out there as "this is the solution I propose - any comments?"

I was thinking along the lines of "let's figure out exactly what we want first" and then go ahead and implement that.


1 unless it breaks someone else's world

@pfmoore
Member
pfmoore commented Jun 6, 2016

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.
If you probably always want something, it should be default. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

If everyone probably always wants something, it (maybe) should be a default. But I was merely commenting on my own preferences. I have honestly no idea whether --only-binary is what most people would want (it probably depends on whether we're talking long or short term).

There's a strong difference between binaries (wheels), sources that can be built anywhere (typically pure Python) and sources that need a compiler or other external prerequisites. Sadly, pip can't distinguish the latter two, and that makes --only-binary less useful (particularly as a default).

And I value consistency between commands over "do what I mean" defaults, which again may just be my personal preference.

@pfmoore
Member
pfmoore commented Jun 6, 2016

More than a summary, it's a follow up to the write-up. Please do read it as soon as you can do so.

I've read it, it didn't add much that I wasn't aware of (as I've followed the various threads on this), so my comments stand. But it's a good summary of the state of play, so thanks for writing it.

@pradyunsg
Contributor

We shouldn't make upgrades different from installs [snip] I'll probably always want pip upgrade --only-binary in the first instance, but I want it to be explicit.

If you probably always want something, it should be default. But I think that maintaining consistency across install and a possible upgrade command is a good idea.

If everyone probably always wants something, it (maybe) should be a default. But I was merely commenting on my own preferences.

Clarification: The "you" in my comment was referring to the end-users (plural one), @pfmoore being one in that specific case. I really should have used some other word.

@pfmoore
Member
pfmoore commented Jun 6, 2016

Some thoughts on upgrade strategies. Personal opinion, all of this, of course :-)

Suppose I do pip upgrade foo

  1. If I don't currently have foo installed, I expect an error. For me, "install" and "upgrade" are two separate activities that I don't want merged.
  2. If there's no version available that's newer than what is currently installed (see below re --only-binary) then I expect a notification that there's nothing to do.
  3. Otherwise, I expect foo to be upgraded to the latest version. That's what I asked for, so it should happen.
    • I'm not convinced that adding constraints makes sense. What would pip upgrade foo>1.0 mean if I have 1.1 installed but 1.2 is available? It's not an upgrade if it leaves 1.1 there, but if it upgrades to 1.2, it's the same as pip upgrade foo. Possibly you could ascribe meaning to something like pip upgrade foo<2.0 but IMO that would be a very unusual case, and not worth the complexity. So let's assume pip upgrade just takes a list of package names (as opposed to requirements).
    • Similarly, I don't know how to interpret pip upgrade <path_to_archive/wheel>. Let's assume it's not allowed in the first instance.
  4. I may or may not want to allow pip to attempt to build from source (this has to be a user decision, as pip can't tell whether building from source will work, and has no capability to try again with a different version if a source build fails). So --only-binary must be a user option, and if specified, means that pip should ignore source distribution when working out "what is available". (We could of course default to only binaries, and have a --allow-source option, but either way upgrade should match install in this regard).

OK, so that's handled the explicitly specified packages. I doubt there is anything controversial in there (except maybe for the parts I say we should omit until later). Now onto dependencies.

  • The primary rule in my mind is that we should never do anything to dependencies, unless we are actually upgrading the user-specified package. So that's the starting point. If the target doesn't get upgraded, don't even look at dependencies. Ever.
  • My view would be that all dependencies have to be handled the same way - whether they are direct or indirect dependencies of the user-specified package, is irrelevant. I don't think this is controversial.
  • A fundamental assumption here should be that the user doesn't know what the list of dependencies is. After all, the point of having implicit dependencies is so that the user doesn't have to manage them. So any solution that needs the user to know explicitly about a dependency is flawed, in my view (unless a command fails with a message about a particular dependency, in which case the user might rerun with that dependency specified in an option - but we should be looking to work first time in as many cases as possible, so this should not be seen as the norm).
  • I would say that by default, the approach should be to only upgrade dependencies that have to be upgraded to satisfy the new constraints. There's a nasty problem lurking here, as constraints may change depending on what gets upgraded. But whether we do a complete solution to that issue, or simply attempt some "best effort" solution isn't that important (I doubt complex dependency graphs with conflicting constraints are common in real life). The important point is the principle, that we don't upgrade anything that the user hasn't asked to be upgraded, unless we have to.
  • Having a flag to say "upgrade everything to the latest version" might be useful (for backward compatibility, at least). Even better would be to have tools (probably external to pip) that analyze and report on upgrade options. Then users could decide for themselves. But I don't know that I would ever see a need for either of these options myself.
  • Rather than an "eager upgrade" option, I'd be much more likely to want an upgrade-all command (or upgrade --all if we want to keep to a single command). This should be semantically identical to upgrade with every installed package listed on the command line. Once I'm doing blind upgrades of packages (typically I may not know or have documented the dependency tree of my packages) I probably just want "everything". Even more so if I use virtual environments to isolate things.
  • The --only-binary question appears again here. Certainly if the user specifies --only-binary for the main upgrade, dependencies should be assumed to imply --only-binary. But even if the user allows for source installs when upgrading the main package, introducing the risk of failure involved in doing a source upgrade of a dependency seems unreasonable. So I'd suggest we should only ever consider binaries for upgrading dependencies, unless the user explicitly allows source. That would imply needing an option --allow-source dep1,dep2,.... I expect this proposal to be controversial, but consider a pure Python package foo that is distributed only in source form, that depends on numpy. We have foo 1.0 depending on numpy >=1.10, and foo 2.0 depending on numpy >=1.11. The user has foo 1.0 and wants to upgrade. They don't have the means to build numpy. They have to allow source upgrades for foo, but they don't want to waste time trying to build numpy 1.11 (or worse still, building it might work but give an unoptimised build, which breaks their system). They could of course specify --only-binary numpy but they may not even be aware that foo depends on numpy.

That's about it. For me, the requirements are pretty clear (and honestly, I don't think they are particularly controversial if we ignore implementation issues). However, implementation is the killer here (basically the above pretty much demands a full SAT solver approach as has been discussed previously). What compromises we are willing to make because implementing a SAT solver is too hard, is where the debates will likely lie.

As far as I know, the following will be the implementation challenges:

  1. A full SAT solver is hard. I don't know enough to understand what alternatives are simpler, and how they differ from a SAT solver. (Well, eager upgrade is, I believe, simple enough - that's why it's what we do at the moment). Specifically, when we'll get a situation where we upgrade something that the simple "don't upgrade anything you don't have to" principle says shouldn't have been upgraded.
  2. I've glossed over anything but the most basic dependency constraints. Once we get involved in upper limits on versions, or lists of allowed versions, things get nasty. But realistically, such cases are likely to be pretty rare (I'd love someone to do research on actual dependency information from PyPI - I may try to do that, but I doubt I'll get the time).
  3. When we have multiple targets - pip upgrade a b c - do we treat this as 3 separate actions, "upgrade a", then "upgrade b" then "upgrade c", or do we merge the 3 into a single "combined" action somehow (note that because I propose treating dependencies different from targets, this is not the same as pip upgrade dummy where dummy depends on a, b and c and we assume dummy was upgraded).

If anyone wants to take issue with any of the above comments or assertions, that's great. Most of this is just my opinion, and I honestly don't work on complex environments - my usage is likely to be mostly about maintaining a system install and a few virtual environments on Windows (hence, debates about binary vs source installs are important to me, but complex dependency graphs not so much). The more use cases we can check our assumptions against, the better.

@dstufft
Member
dstufft commented Jun 6, 2016

So we currently have two operations, pip install and pip install --upgrade, however I think that both of these things do something that nobody actually wants to happen in real life.

For pip install, we get this weird sort of behavior where you might get the latest version (or you might not!) based on what is already installed in your system. I think this sort of weirdness makes pip harder to use, and I don't think it really makes a whole lot of sense (what situation would you care about not upgrading but you still want to install the latest if you don't already have it?). I think that the fact that this command has this weird maybe-latest-maybe-not behavior is the reason why we see a lot of people reaching for pip install ---upgrade as their primary command rather than pip install.

However, pip install --upgrade isn't much better, while it gives you consistent behavior, it's overly eager, it extends into a full on upgrade of everything which the user may not even be aware will be implicated in their pip install --upgrade command. I do see some use for this behavior, but I don't see it as a good default for a top level command.

What I think we should do here, is figure out a path to making `pip install ...more consistent. In that I meanpip install <a list of named requires``should always end up having the latest acceptable version (given specifiers and modifier flags like``--only-binary``) regardless of what was previously installed. I think that we should otherwise attempt not to touch anything that wasn't explicitly mentioned, except where required to do so (and if we are required to do so to satisfy a constraint, then we should again upgrade to the latest available to us that matches all constraints).

I think that giving this command some sort of --eager or --recursive or --upgrade-all-the-things behavior would be fine.

I don't think that a pip upgrade command that takes a list of packages is something that we should do. I think that if we add such a command, then it should be used to simply upgrade everything installed to the latest versions (taking into account dependency chain information, as well as modifiers like --only-binary).

@pfmoore
Member
pfmoore commented Jun 6, 2016

Huh? So pip install foo doesn't always fail if foo is installed? That seems wrong to me, I'd expect it to just say "already installed".

I'm not keen on the idea of pip install being "install or upgrade". Better to be explicit and all that.

@dstufft
Member
dstufft commented Jun 6, 2016

Right now pip install foo never "fails" based on if something is installed or not, it will just do nothing and say that X is already installed. My assertion is that behavior isn't very useful. "Assert that some version, any version what so ever is installed" does not seem to be useful behavior to me. It also doesn't match other package managers that I'm used to like apt-get or so.

@pfmoore
Member
pfmoore commented Jun 6, 2016

OK, I can see that behaviour isn't useful (although personally I find silently accepting "already installed" as fairly innocuous). But I'd prefer to see an install of an already installed package fail, rather than have it upgrade.

What would you expect pip install foo-1.0-none-any.whl to do if foo 2.0 was already installed? Downgrade? Silently do nothing? I'd rather see a nice, simple "already installed" error than a complex set of rules that I doubt people would remember in practice.

I don't have much experience with Linux package managers, so I can't comment on the similarities (or otherwise) with pip. But I don't think I'd expect apt-get install foo to upgrade, so if you say it does, I can only reply that I'd find this odd, too.

@nchammas
nchammas commented Jun 6, 2016 edited

"Assert that some version, any version what so ever is installed" does not seem to be useful behavior to me.

Quick side-question about this: What about "Assert that this specific version is installed"? That seems useful to me, having an idempotent install command.

Nevermind, we have that behavior today.

@njsmith
Member
njsmith commented Jun 6, 2016 edited

@pfmoore:

What would you expect pip install foo-1.0-none-any.whl to do if foo 2.0 was already installed? Downgrade? Silently do nothing? I'd rather see a nice, simple "already installed" error than a complex set of rules that I doubt people would remember in practice.

Huh, expectations are surprising things. I would say that obviously in this case pip should downgrade. The user has completely explicitly said that they want pip to install this particular wheel, so pip should install that particular wheel. There's nothing complex about that. But the "distribution path/file is explicitly given" case is #536 -- probably we should keep this discussion focused more on what happens if the user says "pip install foo", which goes to the package index and finds a foo 2.0, when foo 1.0 is already installed.

I very much agree with Donald's position here. If we start from the question "what should pip install do?", then I can see how one might argue that, well, it says 'install' in the name so it should only install things, never upgrade them (well, except when there's some constraint thing). But if we start from the question "what operations do users want?", then a command that might install the latest, or might leave you with an old version, is really weird and counter-intuitive. I assert that in 99% of cases where users type pip install x, it's because they (a) either aren't sure whether it's installed or else know it isn't, AND (b) want to make sure that they do have it installed because they're about to start using it for the first time. (If it weren't the first time, they'd know it was installed, so they wouldn't run pip install.) In this situation, giving them the latest version is the right thing to do.

@nchammas:

What about "Assert that this specific version is installed"? That seems useful to me, having an idempotent install command.

For "specific version" there's pip install x==<version>.

I can also imagine that for some kinds of scripting/programmatic use it might be useful to have a pip require x command that has the same semantics as installing a package with Dist-Requires: x, i.e. it makes sure that some version is installed but with no guarantees about what. But this would be a lower-level command not intended for end-users.

One way to think about the difference between these: if x is not installed, then it would be okay for pip require x to install some random old version. (And heck, maybe it should, to force people to be robust against it.) But no-one would ever accept pip install x installing some random old version.

(This is a thought experiment, I'm not actually advocated that plain Dist-Requires: x or pip require x in an environment with no x should pick a random old version of x to install.)

@njsmith
Member
njsmith commented Jun 6, 2016

Well, I guess there's also one other case where users type pip install x, which is when they already know it's installed, but they're used to systems with the Debian-style behavior where install always upgrades. Obviously these users also want it to upgrade :-).

@xavfernandez
Collaborator

I'd rather not add a pip upgrade command.

pip users already have some expectations of the behavior of pip.
The main pain point comes from the pip instal --upgrade default behavior so let's focus on that.

A warning in pip 9, plus additonal options to tweak the behavior of pip install --upgrade, (--eager/non-eager) followed in pip 10 by a change in its default behavior seems simple enough, should remove the main pain origin and does not break pip users mental model of pip.

@dstufft
Member
dstufft commented Jun 6, 2016

Yea, I'm definitely trying to approach this from a "what operations does a user want to do" versus "what operations should X command do". I am then taking this high level operation that I think users want to do, and trying to map it to a single named command (as explicit as it would be, pip install-the-latest-version`` is not very user friendly).

This is obviously all very fuzzy, but I can say that 99% of the time what I do is pip install -U <whatever> because that best matches what I expect from an installer given what is currently available. I also see in various automation scripts people using pip install -U. It's also what other popular package managers for languages do by default like npm. In the cases where I do see people not using -U, it's because of the recursive nature of it, not because they don't want the latest version of things installed.

@njsmith
Member
njsmith commented Jun 6, 2016

pip users already have some expectations of the behavior of pip.

TBH, though, the main expectation I have about pip as a user is that on about 50% of invocations it will do something surprising and obviously wrong. And I'm not the only one -- see e.g. @glyph's talk at pycon last week where he observed that pip is great except that all the defaults are broken and require unbreak-me flags. This isn't a criticism of the pip developers, I understand you/we are all working under a complex set of constraints, and it isn't an argument that we should just break things willy-nilly for the sake of breaking them -- pip has a lot of pieces and lots of them are fine! But given the overall state of pop's defaults, I'm really not convinced by arguments of the form "pip has always done X, therefore pip should always do X". If you want to argue for pip install to refuse to upgrade then that's cool but I'd much rather see that argument made on the actual merits, not just on inertia.

@dstufft
Member
dstufft commented Jun 6, 2016

Yea, I certainly agree with @njsmith and @glyph here. We have a number of bad default behaviors, and part of moving forward I think needs to be figuring out how we can get rid of those and deal with those breaking changes to move things towards better defaults.

This particular change may not be one of those, but I think it is.

@pfmoore
Member
pfmoore commented Jun 6, 2016

Yea, I'm definitely trying to approach this from a "what operations does a user want to do" versus "what operations should X command do". I am then taking this high level operation that I think users want to do, and trying to map it to a single named command (as explicit as it would be, pip install-the-latest-version`` is not very user friendly).

OK. Suppose I'm willing to count myself as convinced (somewhat) about this. However, if we assume this is the right thing to do, what about dependencies? I'm 100% convinced that "try to install numpy from source" is almost always not what is wanted. So we only install numpy from wheels, unless the user explicitly mentions numpy. Take that as a given for now, and then suppose we have the situation I described earlier.

  • User has foo 1.0 and numpy 1.10 installed, foo 1.0 depends on numpy >= 1.10
  • PyPI has foo 2.0 available, depends on numpy >= 1.11. There are no numpy 1.11 wheels.

What does pip install foo do? Presumably leave the user at 1.0, as it's a working installation? But should it succeed (as foo is installed) or fail (as it couldn't install the latest version)? If the former, how does the user find out that his system is out of date? If the latter, how does the user say "I just want to make sure foo is there"? OK, the user can do pip list --outdated, see that foo 2.0 exists and do pip install foo (BTW, that still seems utterly weird to me, I have foo, but I know there's a new version, so I do pip install??? Nevermind...) And get a success, but 1.0 remains installed?

One reason I prefer two commands is that the user's intent is completely clear, so edge cases like this can be handled correctly because we know the user's expectation.

Maybe this is all obvious if you're used to apt-get. But I can definitely say it's not at all clear to someone like me who isn't.

If you want to argue for pip install to refuse to upgrade then that's cool but I'd much rather see that argument made on the actual merits, not just on inertia.

My argument is on the basis of explicit rather than implicit. Most definitely not on "we've always done it this way". I have no issue with the idea that maybe apt users are used to "install" meaning "possibly upgrade". I'm really not so sure that other users would be.

One thought - does apt have a "package already exists - upgrade?" prompt? I could imagine install-as-upgrade being less surprising to me if it was "install-or-ask-if-I-should-upgrade"... Of course, pip doesn't behave interactively like that at the moment, although obviously making it do so is an option.

@dstufft
Member
dstufft commented Jun 6, 2016

That is an interesting question-- Other package managers mostly get around that by either not really having binary packages at all so it's always by source, or by pretty much only dealing with binary packages so it's always binary. We're in a sort of weird in between that makes it harder.

Given that, I think that by default for right now, we should pull down numpy 1.11 source package and attempt to install it, but if they've specified --only-binary, then our hypothetical resolver (which we desperately need, SAT or back tracking or whatever) would see that foo-2.0 is not a resolvable installation and will then fall back to installing foo-1.0. This is not a great default, particularly for users on Windows where compiling is much harder, but I think it reflects the reality of today.

That being said, one thing I really want to do is to start to try and push things towards a world where we can change pip's behavior again so that by default we can be binary only, and require opt-in for source releases, but I don't think we're at a place yet that we can do that.

@njsmith
Member
njsmith commented Jun 6, 2016

@pfmoore: I think the question of binary vs. source installs is somewhat orthogonal? It seems to me like the exact same questions arise for a dedicated pip upgrade command, so while they're real issues we need to address, splitting upgrade and install just moves the issues around rather than simplifying them? Also, in the particular case of numpy, we now ship wheels for basically all the platforms that we care about supporting :-).

But here's how I would suggest handling these issues for pip install foo (specifically this command -- I'm not talking about pip install foo==whatever or pip install ./foo-*.whl or pip install bar where bar has Requires-Dist: foo):

  1. Query the index to find the latest candidate version of foo; call this $LATEST. If no candidate versions exist, then error out.

  2. If $LATEST is already installed, then finish successfully.

  3. Check if there is a distribution of $LATEST that can be installed on the current environment. (Example reasons why there might not be: there are only wheels, but no sdists, and the wheels don't match the current environment. There isn't any matching wheel, and there is an sdist, but the user has passed --binary-only :all:. There isn't any matching wheel, and there is an sdist, but the sdist has some flag saying "I only work on python 3" and the user is running python 2 -- the ipython/jupyter folks will probably propose this as a new feature soon, b/c they want to drop python 2 support for new releases in January while still providing a python-2-supporting LTS.)

  4. If $LATEST does not have a viable distribution: issue a warning to tell the user that a newer version is available but not for their environment, ideally with a hint as to what they need to do if they really do want the new version (e.g., "ipython 6.0.1 is available but requires python >= 3.4, and you have python 2.7 -- consider upgrading python", or "numpy 1.12 is available, but there is no binary for ppc64 and you have disabled building from source -- consider --allow-source numpy). Then remove $LATEST from the list of candidate versions and go to step 1.

  5. If $LATEST does have a viable distribution, attempt to install this distribution.

  6. If installing it fails (e.g. b/c it's an sdist and there's no compiler), then error out. Otherwise, finish successfully.

@pfmoore
Member
pfmoore commented Jun 7, 2016

@njsmith binary-only is somewhat orthogonal, agreed, but IMO if we're trying to design commands that "do what the user expects" then it's critical to get this right at the same time.

@dstufft the problem with "install numpy unless the user says --binary-only was explained in the example in my previous epic post - (1) say that foo is only available in source form, and (2) the user may not (and indeed should not need to) know that foo depends on numpy. Then the user can't say --only-binary :all: and has no idea they need --only-binary numpy until after a (long) failed compile. Or (possibly worse still) a successful compile that leaves the user with a non-optimised numpy (these days, numpy compiles out of the box on Windows, but gives a non-optimised build).

I completely agree that long-term we should default to binary-only, but we're nowhere near that yet (at a minimum, it should imply that pretty much every pure-python package is available as a wheel).

@pradyunsg As you can see, there's still a lot of unresolved issues here. Are you still interested in taking this forward? I'm reluctant to fire off another long debate if it's just going to stall again...

@pradyunsg
Contributor

@pradyunsg As you can see, there's still a lot of unresolved issues here. Are you still interested in taking this forward? I'm reluctant to fire off another long debate if it's just going to stall again...

I expected there to be. I am interested in taking this forward. Let's do this! 😄

@FichteFoll
FichteFoll commented Jun 7, 2016 edited

I suggest collecting a list with all of what a user wants pip to do and then propose solutions for each of them until all (or a sufficient amount) of them are solved by a pip command (with options/defaults) or can be handled with "no action".

To start things off, here's a most likely incomplete list:

  1. A user wants to install a package that is not yet installed.
  2. A user attempts to install a package that is already installed.
  3. A user wants to upgrade a package that is installed.
  4. A user attempts to upgrade a package that is not installed.
  5. A user wants to ensure they have the latest version of a package installed, whether it was already installed or not.
  6. A user usually does not want all dependencies to be upgraded and instead have them only be satisfied.
  7. A user wants to upgrade/install a package but does not want to build from source (neither the package nor its dependencies). Probably more likely than:
  8. A user is willing to upgrade/install from source.

My personal proposals for this, as a user:

  1. & 7) pip install foo should attempt to resolve to the latest available version (considering dependencies) and install it. Algorithm would be @njsmith's.
  2. pip install foo → show a warning that foo is already installed and propose using pip upgrade foo
  3. & 7) pip upgrade foo attempts to install the latest available version of foo, again following @njsmith's algorithm. If a newer version cannot be installed because it isn't available for the platform and there is no sdist or the user doesn't want to build from source, show that. Succeed in either case and only fail if installation itself failed.
  4. pip upgrade foo fails if the package is not installed.
  5. pip install-uprade foo or pip install foo --ensure-latest or pip install foo --upgrade (basically the same as currently except non-eager).
  6. All operations should be non-eager and an --eager flag would be available to get the old functionality of install --upgrade. If a dependency is not yet installed, install the latest. If it is already satisfied, do nothing. If it is not satisfied, install the latest still satisfying version (for upper-bounded requirements).
  7. pip upgrade foo --allow-source or pip upgrade foo --allow-source numpy, if foo depends on non-binary numpy releases. Edit: I don't know if this would be applicable, see comment below.

Feel free to extend the list and post your own proposals.

@dstufft
Member
dstufft commented Jun 7, 2016

@pfmoore I'm not sure what the alternative is though? The state of the world today is that wheels are gaining ground but they're no where near ubiquitous so I'm not sure I really see a good option that doesn't allow source releases by default right now.

@pfmoore
Member
pfmoore commented Jun 7, 2016

@dstufft my proposal was to allow source for explicitly named packages, but default to only binaries for dependencies. That way the user gets no "surprise" build steps. It's a compromise, sure, but it reflects what I (have to) do manually at the moment.

@pfmoore
Member
pfmoore commented Jun 7, 2016

@FichteFoll Above all else, my main use case is for an "upgrade all" capability. Look for everything that's currently installed, and if there are newer versions available, upgrade them.

The packages I have installed don't typically have anything other that ">=" dependencies (and most don't even have that) so there's nothing complex here. Just grab the latest version. My biggest restriction is that there are some packages I can't build (numpy, scipy, lxml, pyyaml, matplotlib, pyqt) so I only want binaries for those. I can probably just put --only-binary <these> in my pip.ini (or at least I hope I can...)

Secondary: Install package XXX (latest version), plus any of its dependencies I don't already have. Don't upgrade dependencies I already have if they satisfy the new package's constraints. I always know that I don't currently have XXX.

Tertiary: Upgrade a single package XXX that I (know that I) have installed. Don't alter any other packages unless required to maintain dependency constraints (and even that is theoretical - I've never encountered the situation in real life so I don't know what would be the best resolution for me). My intent is always "upgrade to latest version". I have never encountered a situation where this would break dependencies of already-installed packages. If it did, I think I'd like a warning that I hadn't got the latest version (and why) plus an upgrade to the latest version that is acceptable. In my mind, this situation currently translates to pip install -U although the behaviour for dependencies is not what I want. The main reason I'd do this though is because of the current lack of a suitable "upgrade all" (or to deal with cases where a new "upgrade all" command failed to work like I wanted).

All of the discussion about dependencies and constraints is, in my experience, almost entirely theoretical. I currently have 160 packages installed in my system Python (a mix of scientific, data analysis, web and general programming modules). 100 of them have no requirements. For the remainder, none have anything more complex than a list of packages - no version constraints or anything more complex than Requires: six, dask, pillow, networkx. The longest list of dependencies had 9 items.

@dstufft
Member
dstufft commented Jun 7, 2016

@pfmoore Isn't that going to break a lot of things? A quick list of things that I can think of off the top of my head that I know are very popular packages is anything depending on:

  • SQLAlchemy (optionally requires a compiler, will use pure Python if not).
  • PyYAML (optionally requires a compiler, will use pure Python if not).
  • Markupsafe (optionally requires a compiler, will use pure Python if not).
  • PyCrypto (always requires a compiler)
  • pycparser (never requires a compiler)
  • httplib2 (never requires a compiler)
  • anyjson (never requires a compiler)
  • zope.interface (optionally requires a compiler, will use Pure Python if not).
  • docopt (never requires a compiler)
  • Mako (never requires a compiler)
  • itsdangerous (never requires a compiler)
  • amqp (never requires a compiler)
  • ordereddict (never requires a compiler)

and so on, you can see a long list of the most popular packages and whether or not they have wheels at http://pythonwheels.com/.

@FichteFoll
FichteFoll commented Jun 7, 2016 edited

@pfmoore

Look for everything that's currently installed, and if there are newer versions available, upgrade them.

I sourced this command from some SO question a while ago that I am currently using for this. It's suboptimal, but works for most of my packages, except one. (It uses the py launcher because I'm on Windows.)

pip list -o | cut -d " " -f 1 | xargs -n1 py -m pip install -U

Now, the one problem I have with this is the flake8 package, which has these requirements:

Requires-Dist: pyflakes (>=0.8.1,<1.1)
Requires-Dist: pep8 (>=1.5.7,!=1.6.0,!=1.6.1,!=1.6.2)
Requires-Dist: mccabe (>=0.2.1,<0.5)

Specifically pyflakes is a problem since that has a newer version available and gets updated with the above command, causing flake8 to fail doing anything (since it checks the version).
So this is indeed something that needs to be considered and I also would like to have proper upgrade-all functionality (without breaking requirements!).

@pfmoore
Member
pfmoore commented Jun 7, 2016

@dstufft why? If foo depends on pyyaml and I ask to upgrade foo, pyyaml doesn't get upgraded (no new binaries) but foo can still be upgraded, as there's still the original pyyaml present.

For new dependencies (or on install where a dependency isn't always present) you have to install, so if there's no binary you take source. I'd personally consider "choose older version with binary over newer version with source" but that's getting perilously close to defaulting to --binary-only which I agree we're not ready for.

Hmm, maybe the issue I have is actually with the --only-binary option, which is too coarse. If we had a --prefer-binary option, that said "only use binaries, unless that means there are no candidates, in which case retry allowing source", I suspect many of my concerns that over-eager upgrading would result in breakage might be alleviated. Which, as @njsmith suggested, means that the binary/source distinctions I'm focusing on may well be orthogonal to this ticket (although it would just change my position to "there's no satisfactory solution to my requirements without something better than --only-binary being available"...).

@pfmoore
Member
pfmoore commented Jun 7, 2016

Specifically pyflakes is a problem

OK, so that's not a situation I have (as I say, I don't have anything with dependencies that complex installed). I don't have a problem with refining "upgrade all" to upgrade things to "the latest version that doesn't result in breakages" but AIUI that needs the "SAT solver" approach to work out the correct solution. That's an implementation issue though - the design should indeed always give correct results.

@dstufft
Member
dstufft commented Jun 7, 2016

@pfmoore I think a --prefer-binary flag could be a good option, regardless of the outcome of this ticket. Likely bundled with a warning when that ends up not installing the latest version that would otherwise have been installed.

@njsmith
Member
njsmith commented Jun 7, 2016

@FicheFoll: I don't think trying to rederive the entire ui from first principles is going to be very productive. There's a relatively clearly defined piece of the problem that's on topic for this particular issue, and if we try to expand the scope to everything at once then it'll just bog down again.

On that topic, it looks like the key place we differ is this: suppose that a user has the mental model that pip install foo is only for transitioning things from uninstalled to installed, and that they understand that foo is already installed. I assert that a user with this mental model will never type pip install foo. Therefore, when some user does type pip install foo when foo is already installed, we can conclude that their mental model is not like your (2). Either the first part is wrong: they know foo is installed and they expect pip to upgrade like some other popular package managers, or, the second part is wrong: they are unaware that foo is installed, in which case they are expecting install to leave them with the newest version (because that's what install does when packages are not installed, and they think this package is not installed).

@dstufft
Member
dstufft commented Jun 7, 2016

For the record, one of the reasons I don't like the pip install ... only ever goes from uninstalled to installed and pip upgrade ... only ever goes from installed to a newer thing installed is because I find the user experience pretty crummy. Software that knows what you wanted it to do, but instead of doing that thing it tells you to invoke some different command is incredibly frustrating.

$ pip install foobar
I'm sorry, but foobar is already installed, you want to run ``pip upgrade foobar``
$ pip upgrade foobar
...

Would do nothing for me except annoy me, even though it's technically "correct".

The flipside of that, if you say "ok, if pip install foobar already has foobar installed, then we'll act like it's not installed, and if you do pip upgrade foobar then we'll act like it's already installed, we end up with two commands that do basically the same thing, except with maybe some minor differences in exactly how things are processed, which says to me that they belong as a singular command with some --options to deal with the edge cases. I think this is better because it means users don't have to try to make a choice between which one they want up front, there is one command to install stuff and for most users that will generally do the right thing. If some user in a specific scenario that requiers some choices on their part, then they have to pay the cost of making some choices about what --flags to use.

@pfmoore
Member
pfmoore commented Jun 7, 2016

OK. Consider me convinced. I've done some research, and even the Windows installers I'm (supposedly :-)) familiar with do install-as-upgrade (if you install something like VirtualBox, it says "you already have a previous version installed, do you want to upgrade?") Powershell has install-package, which doesn't say anything specific, but there's no upgrade-package. Etc. So I guess just having the single "install" command is the norm.

Which of course means that, unless someone else wants to argue, technically this PR can simply be closed as "not going to be implemented". But it's still as good a place as any to discuss how we want to remodel the install command, I guess.

OTOH, maybe we do close this as rejected, and someone opens up a new issue, with a concrete proposal of how they suggest the install command should be modified. It might at least provide a clearer starting point for discussions.

A question. Does anyone think we're at a point yet where someone could put together a complete proposed behaviour from all of the suggestions here and elsewhere? Covering how dependencies are handled, what happens with constraints (and when they conflict), binary vs source, how we support the "upgrade all" scenario, etc? My personal feeling is that we need someone to make that decision, to give the discussion a reference point, or we could just debate details forever. I could probably do that, but I am unlikely to be able to implement what I propose (e.g., I would propose an "optimal dependency resolution" approach, which implies a SAT solver AIUI). So it would be better for someone who was willing to implement their proposal to step up (and deal with the inevitable debate and bikeshedding :-)).

I'm still concerned about some of the implications in the points made here, but I'm not sure I have the energy to debate them until there's a real possibility of an implementation.

@xavfernandez
Collaborator

I fully agree with @dstufft 's latest #59 (comment)
That's why (once again) I'd advocate for the simple solution of --eager/--non-eager options.

@pfmoore
Member
pfmoore commented Jun 7, 2016

I also agree with @dstufft's comment, as noted (we go for a single install command and no update command).

However, I'm not sure what --eager / --non-eager entails. I guess --non-eager means don't upgrade anything that doesn't have to be updated (either because the user specified it explicitly, or because it's too old to satisfy the new set of dependencies). Does --eager then mean upgrade every dependency to the latest possible version, whether necessary or not? Which would be the default? (I'd argue for --non-eager)

And a question - would this encourage scientific packages to correctly declare their dependencies? That has to be an important consideration.

Bikeshed point. The option names --eager and --non-eager are pretty unintuitive. I think we need better terms. Maybe something explicit like --upgrade-dependencies.

This PR also suggests an upgrade-all command, which is the key use case for me. Are you saying that we reject that command, or simply that you don't have an opinion on it?

@dstufft
Member
dstufft commented Jun 7, 2016

My understanding of what --eager and -non-eager means matches what @pfmoore just said (whether dependencies are only-if-needed or always installed), and I agree that --non-eager should be the default. I also agree the name is kind of crummy though I don't have a better solution that isn't a mouthfull. Maybe --[no-]recursive or something, I don't know.

I think something like an upgrade-all command could be a good addition (as long as it made sure not to violate the version specifiers of anything). I would probably just call this upgrade though and just let it take no arguments to restrict what it operates on.

@pradyunsg
Contributor
pradyunsg commented Jun 8, 2016 edited

tl;dr
Discussion for an upgrade-all-packages - stays here.
Discussion for prefer-binary - Over to #3785
Discussion for install-as-upgrade - Over to #3786


If we had a --prefer-binary option, that said "only use binaries, unless that means there are no candidates, in which case retry allowing source"

This is a good idea. While related to this issue, I do think it deserves it's own issue. (@dstufft's comment makes me think there's interest to pursue it). I took the liberty to open #3785 for further discussion on this.

My understanding of what --eager and -non-eager means matches what @pfmoore just said (whether dependencies are only-if-needed or always installed)

Eager upgrade would install the latest allowed versions of all (sub-)*dependencies. Non-eager upgrade would upgrade a (sub-)*dependency only if it no longer satisfies the requirements of a package.

The option names --eager and --non-eager are pretty unintuitive.

I agree. I like the idea of putting the behaviour behind a single flag.
--upgrade-strategy=eager/non-eager

It's a mouthful too but it does convey the intent more explicitly. Is it too verbose? Maybe.

I think bike-shedding is best done after finalizing semantics.

Does anyone think we're at a point yet where someone could put together a complete proposed behaviour from all of the suggestions here and elsewhere?

I think so. We do, at the least, need to establish some base facts we agree upon. I'm on this.

OTOH, maybe we do close this as rejected, and someone opens up a new issue, with a concrete proposal of how they suggest the install command should be modified. It might at least provide a clearer starting point for discussions.

I think we should leave this issue open for now, because it proposes an upgrade-all. You have already noted that upgrade is not happening. I opened #3786 for futher discussion on the install-as-upgrade command.

@pradyunsg
Contributor

I'm still concerned about some of the implications in the points made here, but I'm not sure I have the energy to debate them until there's a real possibility of an implementation.

I am willing to take this forward all the way through to implementation. I definitely don't want to waste everyone else's effort in getting to a consensus about this.

@njsmith
Member
njsmith commented Jun 8, 2016

I think everyone agrees that an 'upgrade the world' command would be really nice to have, but IIUC it's blocked while we wait for the resolver work to land. In the mean time, I posted a more concrete proposal for single-package upgrades on @pradyunsg's new dedicated issue for that discussion.

@pradyunsg
Contributor

I think everyone agrees that an 'upgrade the world' command would be really nice to have, but IIUC it's blocked while we wait for the resolver work to land.

Indeed this issue is now properly blocked by #988. Mentioning the issue number for cross-linking the two issues.

@pradyunsg
Contributor
pradyunsg commented Jun 8, 2016 edited

I almost forgot...

I'm not sure what you mean about "holding" updates. Please clarify.

Now that this issue is exclusively for upgrade-all, I should clarify.

There might be some situations where it might be useful to prevent an upgrade of a certain package when we run upgrade-all. The specific one I have in mind is... If pkg is installed and I don't want to bother about going re-configuring a potential newer version, so, I want to prevent pip from upgrading that specific package when running 'upgrade the world'. Essentially I'm holding back the upgrade of pkg.

A flag that takes comma-separated list of packages to hold back from an upgrade would be fine.

Adding this once a SAT solver comes along should be easy. It's just a few extra clauses IIUC. (Yes, I'm brushing up on SAT solvers as well)

I don't know if this is something common or something someone wants. But, I don't think this originated in my head though. Someone must have mentioned it in some thread somewhere.

@pfmoore
Member
pfmoore commented Jun 8, 2016

Ah, I see. That makes sense, and seems like a reasonable thing to want.

@pradyunsg
Contributor

Quick request: Could someone please add to the description a tiny note that upgrade command won't happen? Possibly if you want, write a tiny summary of why it is so as well.

goes to sleep

@njsmith
Member
njsmith commented Jun 8, 2016

Two nice features that apt has (and I think other mature system package managers like dnf have similar features):

  • Marking a package as "held", which is a persistent flag attached to the package and causes it to be skipped by upgrade-the-world commands. Often set on packages that you had to downgrade or patch locally.
  • Tracking of which packages were explicitly requested by the user, versus which ones were only installed implicitly to fulfill dependency constraints. The latter can be removed by an upgrade-the-world if they stop being depended-on.

Both of these can be seen as special cases of the thing that more recent project-environment package managers like npm and cargo do, where they distinguish between some sort of human-oriented "requirements" file versus a fully specified "lock" file. An explicitly held package is like a package that has a human-specified version constraint, and am explicitly installed package is one that's listed in the human-editable file.

@pradyunsg
Contributor
pradyunsg commented Jun 9, 2016 edited

Ditto. If we're adding 'upgrade the world', we need to add the ability to mark packages (as held/user-installed and maybe more) to add information to better determine a course of action for the upgrades. I see this as a requirement, not as a nice to have.

I actually like the technique used by Cargo and likes. The use of files and not some form of meta-data-hidden-behind-a-cli-command makes it much more easy to grasp, manage and also possible to create a reproducible environment.

I would actually be happy if I see some form of pyproject.lock...

200th comment. Wow. 😃

@pradyunsg
Contributor

Adding reference to #654 for "marking" of packages that we talked of.

@pfmoore
Member
pfmoore commented Jun 27, 2016

Can you explain what cargo does? Where would you see the pyproject.lock file being stored on the user's machine?

@triplepoint

Rust's Cargo's lockfile gets stored in the project root, to record the version of dependencies currently installed. Committing this file to git allows you to share a consistent set of dependency versions with other developers and CI. PHP's Composer has a similar lockfile, Composer.lock.

I always assumed that pip's 'pip freeze' and 'pip install -r' were meant to do a similar thing, and it was just unfortunate that the lock file format was easily readable/writable by humans and people choose to edit it directly. Was requirements.txt not originally conceived as a lockfile?

@pfmoore
Member
pfmoore commented Jun 27, 2016

@triplepoint Thanks for the explanation. That does indeed sound more like a requirements file. But requirements files are optional, version-based and per project. Marking (as I understand it) should be per environment (virtualenv or system installation) and should simply say "don't auto-upgrade package X" (but allow manual upgrading if the user explicitly requests an upgrade of that package by name).

@rgommers

To help untangle the discussions on upgrade-all and "upgrade behavior", here are comments from @rbtcollins and @ncoghlan in the discussion on the list on the latter about a SAT solver not being required for a first implementation of upgrade-all:

Robert:

I realise the consensus on the ticket is that its blocked, but I don't
actually agree.

Yes, you can't do it right without a full resolver, but you can do
an approximation that would be a lot better than nothing (just narrow
the specifiers given across all requirements). That is actually
reasonable when you're dealing with a presumed-good-set of versions
(which install doesn't deal with).

Nick:

"yum upgrade" has worked well enough for years without a proper SAT solver, and the package set in a typical Linux install is much larger than that in a typical virtual environment (although distro curation does reduce the likelihood of conflicting requirements arising in the first place).

That said, rerunning pip-compile and then doing a pip-sync is already a functional equivalent of an upgrade-all operation (as is destroying and recreating a venv), so I agree there's no need to couple the question of supporting bulk upgrades in baseline pip with changing the behaviour of upgrading named components.

@pfmoore
Member
pfmoore commented Jun 29, 2016

IMO, pip upgrade-all is by far the most important proposal on the table out of all of the various "upgrade functionality" discussions. Having "upgrade all" would give an obvious way to keep your system up to date, making questions about non-eager updates leaving things at older levels much less pressing, as well as filling a gap that currently exists.

While a full solver would be nice, I see no reason why the starting point for pip upgrade-all shouldn't be that it does what pip install -U <list every package that's installed here> does. That's what I'd expect as a user, and for the majority of cases it does exactly what's needed. The complex corner cases around conflicting requirements can be handled in the first instance by reference to the above. If that isn't sufficient, then we can look at either modifying install -U behaviour to address it, or special-casing the update-all command, or even implementing a full solver at that point.

@magicgoose

Are you going to implement this in next 10 years?

@pradyunsg
Contributor
pradyunsg commented Sep 15, 2016 edited

FWIW, I will be re-visiting this, this weekend.


@magicgoose said:
Are you going to implement this in next 10 years?

@pfmoore said it better than I can:

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

So personally, I'd appreciate it if people would refrain from pinging this issue unless they have working code that at least offers a starting point for the implementation - and they are willing to follow it through to implementation.

@RonnyPfannschmidt
Contributor

entirely personal opinion

i think by the very nature of pip (which is install packages from a repo that does not have global QA like say debian, redhat or ubuntu)
i feel it is necessary and/or acceptable to actually completely refrain from implementing and update-all functionality,

since we pip ever guarantee a known got state of a install-able set of python packages

@pfmoore
Member
pfmoore commented Sep 15, 2016

@RonnyPfannschmidt IMO it's perfectly reasonable for users of pip to explicitly disallow the use of an update-all command, if that fits their requirements / workflow. But not all users of pip have the same strict requirements as these folks. For users with more relaxed needs, an update-all command is useful, and its lack makes it significantly harder for them to keep their systems "up to date". So I do think it's reasonable for pip to provide such a command. It's our job to provide the tools people need, not to enforce particular policies on how users choose to use those tools.

@RonnyPfannschmidt
Contributor

@pfmoore my personal experience with just updating all packages in an environment is, that it breaks things very regular ^^

users that need an relaxed update all sound like you mean normal end users (which should just use a normal linux distribution for example)

@ncoghlan
Contributor

Whether upgrade-all is problematic or not depends a lot on how many dependencies you have, and how well disciplined they are about their API maintenance. It can also be used in conjunction with a curated private repository to get control over what upgrades actually happen, rather than having to carefully configure which upgrade commands you run in each virtual environment.

@ncoghlan
Contributor

@RonnyPfannschmidt Windows users still outnumber Linux users ~18 to 1, and they don't have anything comparable to a distro package management community to fall back on at this point (while the core technology is there in recent versions, the packaging and curation communities aren't). That means they're a lot more reliant on user level tools like pip and conda to pick up the slack.

@rgommers

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

@pfmoore are you aware that such comments can come across as denigrating the efforts of contributors? That is how it looks to me. I'm not sure if you followed the whole history of this issue, but in this particular case you're very much off-base. The issue is much more with the pip development team than with any contributors.

Rough summary of just a part of it (PR gh-3194):

  1. There's a documented decision that an upgrade command is welcome (on the pip mailing list as well as the docs and GitHub).
  2. Then a call goes out by a prominent developer (@njsmith in this case) that implementing this feature would be a very valuable.
  3. New contributor shows up, implements everything, addresses all reviews comments quickly. PR ready to merge.
  4. Core contributor changes his mind about wanting upgrade.
  5. Very long discussions follow, without conclusion.
  6. Contributor gives up and disappears (most people would, such things are frustrating).

And that was even before @pradyunsg showed up, who is showing remarkable persistence.

In a well-functioning project, the core devs that really care about the issue would organize a quick hangout and make some sort of decision. Or delegate one or two people to work on it enough to get that conclusion. Or at the very least apologize and say thank you to the PR submitter, instead of blaming him for "not following through".

I know you have the best of intentions, but please be a bit more careful with this sort of comment.

@dstufft
Member
dstufft commented Sep 15, 2016

I think it's perfectly reasonable for requirements and opinions to change through discussion, particularly for a thorny issue like this one where there isn't a right answer. Part of getting a patch landed in any code base is keeping up with changes that get hashed out as part of review and discussion around any change. Some changes are fairly minimal and have less of them, some have more. There's nothing wrong with a person deciding they don't want to deal with that and dropping out (every bar to adding a change will cause some amount of drop off, including tests, documentation, etc).

This particular change is particularly nasty because it's changing the default behavior of the primary usage of pip. That's hard and scary and it would be a disservice to our users if we rushed it and didn't full hash things out prior to committing to one direction or the other. This command is used 10s to 100s of millions of times a month. This is not a small, easy change and it won't be the folks pinging this issue who have to deal with the angry backlash that comes from making any change.

The person who implemented the PR before, their time is appreciated, but it is a fact of the matter they didn't follow through till the end. As core devs here our time is limited, we're either volunteers or spread amongst many projects and we float in and out of participation. There are a ton of different issues that all require attention and this is just one of them. Paul simply stated that pinging this issue isn't helpful (which it isn't) and that if someone wants it they'll need to either wait for someone (including one of the core devs) to decide to put in the considerable effort to change the default behavior of millions or do that themselves.

@pfmoore
Member
pfmoore commented Sep 15, 2016

We're aware that people want this, what is lacking is anyone willing to develop a fix that caters for all of the various concerns and issues that have already been raised, and that will inevitably come up during a PR review.

@pfmoore are you aware that such comments can come across as denigrating the efforts of contributors? That is how it looks to me. I'm not sure if you followed the whole history of this issue, but in this particular case you're very much off-base.

@rgommers Seriously? That comment was from months ago, and was quoted out of context here (but frankly, I have no problem with @pradyunsg quoting it in response to the unhelpful and sarcastic comment he was replying to). I'd suggest that if you'd reviewed the whole history, you would have seen my comment in context, at which point hopefully you would have understood what I was trying to say.

If I was that "off-base", then you could have said so in May at the time I said it, rather than picking on it now, out of context.

If I offended anyone, I apologise, that wasn't my intent. Honestly, I'm as frustrated as anyone else that this issue is proving so difficult to reach a design that is acceptable to everyone. In my comments, I'm trying to strike a balance - on the one hand, I genuinely appreciate the contributions people make, but on the other hand, I feel it's important to make it clear to people that on an issue like this, the coding is frankly the least of the work needed. Very often, contributors don't appreciate this fact, and that's where we get incomplete PRs, where people get frustrated with the work needed to persuade people that their design is OK, and/or to rework the change, possibly in ways they don't really like, to take into account other people's (often conflicting and inconsistent!) views. I'd rather they came into the process with an understanding of what's involved, rather than coming in with unrealistic expectations and as a result having a bad experience.

Everyone involved in the discussions on this issue has put in a lot of time debating the pros and cons of various approaches. I don't imagine there's anyone who's happy over how long it's taking. Personally, the lack of an upgrade-all command (just one part of the change, and not likely to be the one that gets implemented first) hits me on a regular basis. We do occasionally (honestly, it's a lot more than just "occasionally") get people (usually people who haven't actually contributed to the discussion or the code) commenting "this is important, why haven't you implemented it yet?" Frankly, it's hard to remain calm and not snap at such people.

In a well-functioning project, the core devs that really care about the issue

You do realise that this comment is open to being interpreted as saying that the pip developers don't care about fixing this (and by implication about pip)? We're all at risk of wording things in a way that can offend people. I'm sure you didn't mean this as a criticism of the pip devs, please assume I'm equally not trying to offend anyone.

would organize a quick hangout and make some sort of decision.

I'd be surprised if something like that would work here, but I'm open to trying it. Not sure who would get involved or how we'd manage it, but sure, if it helps and someone wants to try that approach. I would say that, as this change has such a large potential to affect pip users, any decision made on a private channel like this should probably be written up as a proposal and published for general comments - and I suspect that doing so would simply trigger yet another round of the same debates we've been having.

Or delegate one or two people to work on it enough to get that conclusion.

So you're saying that a decision this big should be made unilaterally by a couple of people? Maybe that's the only way we're going to get a resolution, but it's not really how decisions are made on pip (unlike Python, we don't have a BDFL with executive decision making authority). You can claim that makes us not a "well-functioning project" if you want, that's not my view but we can disagree on that if you want.

Or at the very least apologize and say thank you to the PR submitter,

I'm not sure what we should be apologising for, but if it helps then I'll happily apologise - for the fact that no-one gave him a clear understanding of what a difficult job it would be getting this proposal through to completion, or helped him to manage the debate and guide the participants to a consensus. But to be fair, I don't think anyone else involved knew at the time that this would be the case, so I think it's mainly hindsight talking here.

He certainly has my thanks. It's rather easy to say "that goes without saying", but it shouldn't - open source projects don't thank contributors enough, and that's a problem. While I'm on the subject, I'd like to thank everyone who's contributed to this debate - and I'm completely serious here - as I know from experience how draining it can be. But especially to @pradyunsg for being the current victim of all the endless discussions and changes of direction. Thanks for not giving up! (Yet!!)

instead of blaming him for "not following through".

Well, I don't think there's any blame attached (although it's hard to know whether he felt blamed, as he's not around any more). But it's true that his original PR didn't get managed through to completion. That's just a fact, though. I hope no-one is suggesting that contributors are entitled to have their PRs accepted simply because they submitted them. Not all PRs are acceptable when first submitted, they need to be revised and updated, and sometimes even after all the work they are still not acceptable. Sorry, but that's life.

[If I seem harsh in the above statement, I apologise (again!). But a lot of my leisure time is spent reading and dealing with complaints that I (or projects I'm involved with) somehow haven't done enough. And it's a vicious cycle - I lose motivation to work on open source in my spare time due to the nagging, which of course means even less gets done. While I try to remain polite, sometimes it's not easy]

I don't plan on saying anything further on this meta-discussion about how the PR is being managed. I've spent an hour this evening working on this response, to try to avoid saying anything that might offend someone (and I bet I failed :-(). And I could have much better spent that time - with my family, relaxing, or doing something more productive.

So can I suggest we get back to trying to help @pradyunsg come up with a PR that we can all be happy with, and leave off the fruitless meta-discussions?

@pradyunsg
Contributor

So can I suggest we get back to trying to help @pradyunsg come up with a PR that we can all be happy with, and leave off the fruitless meta-discussions?

Yes please. 😇

@pradyunsg
Contributor

Oh, I forgot.

@rgommers said:
And that was even before @pradyunsg showed up, who is showing remarkable persistence.

I'll take this as a complement... Thanks.

@pfmoore said:
especially to @pradyunsg for being the current victim of all the endless discussions and changes of direction. Thanks for not giving up!

You're welcome.

(Yet!!)

I really hope it doesn't get to that state. It'll be a really bad state of affairs if it does, IMO. (I'm arrogant that way)

@pradyunsg
Contributor

I wrote the following while I was reviewing the history and figured might as well put it up here, for future reference and letting others correct me just in case.

  • Decided to work on this after realizing how the big non-technical:technical ratio would to be (It's much bigger than I really conservatively thought) and realizing that there has already been an unsuccessful attempt at this.
  • Did a write-up on the state of affairs, because I was bored and I needed to know what's happened anyway.
    • Probably spent more time (and space here?) on it than needed but at least I got a good overview of the issue for myself (and everyone else?).
  • Got over-excited after writing it. 😃 Showed it to the world!
  • Initiated a (long!!!) discussion on implementing an upgrade command.
    • A few useful off-shoot ideas came up in the discussions. New issues were created for the same.
  • Discussion leads to the idea of changing install behaviour to upstall, which would do non-eager upgrades by default and removed a part of the functionality provided by --target option - 3 things.
    • This is where we made a mistake - bundling up the 3 (fairly) independent changes and which would be implemented as one, because no one realized this part.
  • I implemented the same. Since the 3 changes were bundled up, they all got stuck when there was no consensus on changing install behaviour to upstall, the potentially controversial change.
  • The lack of consensus initiated a long discussion which came to the point where people were backing out of the discussion and I guess almost pushed to burnout.
    • Some of the earlier assumptions were broken and it was decided that we would do a minimal disruption change first.
  • I ran out of free time to work on this so I created a few new issues so that someone else can work on them independently and ideally not make the same mistake as we did here.
  • Stalled.
  • I'm back! I'll try to get the switch to non-eager upgrades by default done by Sept 25.
@dstufft
Member
dstufft commented Sep 16, 2016

Yes, let's just focus on fixing pip install --upgrade and leave everything else out for now. That's a requirement for any other work anyways.

@rgommers

@pfmoore @dstufft thanks for the thoughtful replies. I didn't mean to offend anyone, so apologies if it came across that way.

I will not respond to everything, because no one is looking for a long discussion here.

you could have said so in May at the time I said it,

I was away from Github for 2 months then, but it did bother me when I saw that comment first time around.

So you're saying that a decision this big should be made unilaterally by a couple of people?

All options on the table are way better than the status quo. And after 5.5 years and many hundreds of comments spread over multiple issues/PRs here and the mailing list, I'm not confident that yet more comments are going to resolve it. I hope it will get resolved, but if this stalls again then definitely yes - nominate one or a couple of people and just make a choice.

I'm not sure what we should be apologising for, but if it helps then I'll happily apologise - for the fact that no-one gave him a clear understanding of what a difficult job it would be getting this proposal through to completion, or helped him to manage the debate and guide the participants to a consensus.

I meant the latter. Sometimes I change my mind on a decision for one of my projects, that happens. Then I feel responsible for making the new direction clear. And apologize if I don't have the time to deal with the consequences of that change (takes only 30 seconds....).

But it's true that his original PR didn't get managed through to completion. That's just a fact, though.

That's a view, not a fact. In my view, the PR was complete - all that was left to do was either hit the green button or reject it. He did all I would expect from a contributor.

I realized after your reply that the expectations the pip devs have from contributors and core devs are very different from pretty much any other project I'm familiar with. I expect core devs to guide new contributors, encourage them, give them feedback where needed, and help them resolve contentious issues (which most contributors neither have the skills for nor the interest in, and those that do often end up becoming core devs) if needed. You're saying to new contributors: "you have to manage us. we may change our minds, disagree with each other, or lose interest - it's your job to manage that". Maybe that's the nature of this project and it has to be this way, I don't know.

While I'm on the subject, I'd like to thank everyone who's contributed to this debate

Agreed. Thanks to everyone who contributed.

  • and I'm completely serious here - as I know from experience how draining it can be.

It is draining. Personally, I stick around here and on distutils-sig because it's important for the Python ecosystem and the users of my packages, but both places don't exactly give me positive energy.

@pradyunsg
Contributor

Just in case it slipped under the radar - #3972 😄

@pfmoore
Member
pfmoore commented Sep 17, 2016

You're saying to new contributors: "you have to manage us. we may change our minds, disagree with each other, or lose interest - it's your job to manage that". Maybe that's the nature of this project and it has to be this way, I don't know.

I said I wouldn't continue this, but this point struck home. That's not how I feel about our approach, but now that you put it that way I do see that it can come across that way. To be honest "we may change our minds, disagree with each other, or lose interest" is true - after all we're all just people with other commitments as well - but I don't consider it as something a new contributor has to "manage". Rather, it's just a reality of dealing with people, but if making too much of it comes across as dumping the problem on new contributors, that's wrong.

Thanks for pointing this out - I'll try to keep it in mind and avoid giving that impression in future.

@dstufft
Member
dstufft commented Sep 17, 2016

I think a lot of the problem really boils down to the fact that we're vastly undermanned which ends up making it hard to keep up and follow up on everything. There are more potential contributors then there are us so it is easy to feel overwhelmed when there are multiple folks all trying to make changes at once. Bigger changes, or changes where there is no clear cut consensus, tend to be the hardest ones to deal with so they are the ones that tend to suffer the most :(

The sad state of affairs is that while I think we would all like to be here to help guide through every contributors through the process, we simply don't have the man power. This tends to have a bit of a viscous cycle too, because we don't have the man power to do that, we don't readily find new people who seem to have really started groking the mentality behind how pip operates and who have learned enough (because we're not there to teach them) to give them commit rights to pip. This means we're constantly undermanned and struggling just to keep our heads above water (At least, that's how I feel. Constant 70-90 hour weeks is real hard on a person :/).

@dstufft
Member
dstufft commented Sep 17, 2016

@pradyunsg Reviewing #3972 is on my TODO list, just haven't hit it yet.

@pradyunsg
Contributor

@pradyunsg Reviewing #3972 is on my TODO list, just haven't hit it yet.

Thanks!

@rgommers

This means we're constantly undermanned and struggling just to keep our heads above water (At least, that's how I feel. Constant 70-90 hour weeks is real hard on a person :/).

That is hard. Numpy and Scipy were in that situation when I started working on those. Not fun. I appreciate all that you're doing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment