Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

`pip install -U` upgrades already satisfied dependencies #304

Open
ejucovy opened this Issue · 17 comments

9 participants

@ejucovy

If I pip install -U foo, I would expect that the latest version of foo will be installed, and foo's dependencies will only be reinstalled if they're not already satisfied. But in fact the dependencies all get reinstalled even if I already have identical versions installed:


$ pip install -U django-supervisor
Downloading/unpacking django-supervisor
  Downloading django-supervisor-0.2.0.tar.gz
  Running setup.py egg_info for package django-supervisor
Downloading/unpacking supervisor (from django-supervisor)
  Downloading supervisor-3.0a10.tar.gz (438Kb): 438Kb downloaded
  Running setup.py egg_info for package supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
Downloading/unpacking meld3>=0.6.5 (from supervisor->django-supervisor)
  Downloading meld3-0.6.7.tar.gz
  Running setup.py egg_info for package meld3
Installing collected packages: django-supervisor, supervisor, meld3
  Found existing installation: django-supervisor 0.1.1
    Uninstalling django-supervisor:
      Successfully uninstalled django-supervisor
  Running setup.py install for django-supervisor
  Found existing installation: supervisor 3.0a10
    Uninstalling supervisor:
      Successfully uninstalled supervisor
  Running setup.py install for supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
    Skipping installation of /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor/__init__.py (namespace package)
    Installing /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor-3.0a10-py2.6-nspkg.pth
    Installing echo_supervisord_conf script to /usr/local/ejucovy/django/bin
    Installing pidproxy script to /usr/local/ejucovy/django/bin
    Installing supervisorctl script to /usr/local/ejucovy/django/bin
    Installing supervisord script to /usr/local/ejucovy/django/bin
  Found existing installation: meld3 0.6.7
    Uninstalling meld3:
      Successfully uninstalled meld3
  Running setup.py install for meld3
Successfully installed django-supervisor supervisor meld3
Cleaning up...

My "existing installations" of supervisor-3.0a10 and meld3-0.6.7 are both "successfully uninstalled", and then identical versions are installed.

@hltbra
Owner

In my judgement it is a known behavior, don't know if it is really a bug - I guess easy_install has the same behavior.

I would like to see other's opinions.

PS.: There was a question in StackOverflow related to this: http://stackoverflow.com/questions/5937756/why-is-pip-looking-for-download-cache-if-the-same-exact-package-is-already-instal

@hltbra
Owner

Related to issue #49

@carljm
Owner

It is a bug, and it is a duplicate of #49.

@carljm carljm closed this
@ejucovy

I don't think it's a duplicate of #49. I read #49 as saying that install -U foo shouldn't reinstall foo itself if it's already at the latest version -- which is different from whether or not it should reinstall foo's already-satisfied dependencies.

This distinction matters for hard-to-build libraries which have frequent releases but fairly stable APIs -- for the most part, one installation is good enough -- I'd only want to reinstall it if my dependencies start to use newer features from those nested dependencies (i.e. if the requirement is no longer satisfied) -- for example:

  • foo 0.1 depends on lxml>=2.3.0
  • foo 0.2 is released, and depends on lxml>=2.3.0 (same dependency)
  • lxml 2.4.0 is released

If I've already installed foo 0.1 and lxml 2.3.0, and I pip install -U foo, I wouldn't want it to install lxml 2.4.0. It should only install lxml 2.4.0 when foo starts to depend on lxml>=2.4.0.

@carljm carljm reopened this
@carljm
Owner

Ah, yes, that is a bit different. Even if #49 is fixed, there would be some additional code needed to achieve the behavior you're wanting when the dependency is not at its latest version, but still satisfies the dependency requirements.

I think if we made this change, some people would find it surprising. I can see why it's desirable in certain cases -- but I can also see cases where the current behavior (minus #49) is preferable. So I'm on the fence on this one.

@ejucovy

Would an extra command-line option be appropriate? pip install foo --upgrade vs pip install foo --upgrade-recursive? (Or --upgrade-nonrecursive if preserving the current behavior backwards-compatibly is important)

@nisavid

making upgrades non-recursive by default would provide consistency with other package managers, such as APT and Portage. and i think there is a good reason for such behavior, which is that it avoids unintended side effects--if i want to upgrade a package P, then i would like to enter a command along the lines of upgrade P, not upgrade P --but-not-other-things.

on the other hand, i think that an "upgrade all" command (see #59) should be recursive by default, since "all" should really mean "all" by default. in this case, non-recursive behavior would mean "upgrade all packages installed directly, but not any dependencies that weren't installed directly" (like Portage's emerge --update @world without --deep).

@cls

A related issue is that if the package being upgraded has a dependency on another package which is already installed but unavailable from a repository, Pip will fail and is unable to upgrade the requested package, despite its dependencies being fulfilled.

@ejucovy ejucovy referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@ex-nerd

This appears to happen with -I as well as -U.

@carljm
Owner

Since -I stands for --ignore-installed, and is intended to make pip operate as if nothing were currently installed, reinstalling everything is the correct behavior for -I. Thus the behavior here is only a bug for -U, not for -I.

@jvanasco

easy_install does not have the same behavior. easy_install will not re-install dependencies if they are already met.

this isn't a feature or behavior, this is a bug.

this is also really annoying -- testing PIP distribution for a framework's package , or updating a single framework add-on, means needing to re-install the entire framework and all of it's dependencies. these unnecessary downloads and installs are a waste of time and resources.

@fdintino fdintino referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@qwcode
Owner

for those who just want a way to do this now, short of a behavior/code change to "-U"

I think this achieves the desired result, right?

upgrade top-level requirements only:

  • pip install -U --no-deps REQS //upgrades only top-level
  • pip install REQS //this 2nd pass will install any new dependencies from the upgrade
@fdintino

For the simplest case that suffices, but I don't think that works for a requirements.txt file, or if there are dependencies which have updates to their install_requires. We have a complicated script that does a diff of our requirements.txt and more-or-less does what you describe, but it doesn't handle upgrade depth > 1.

What complicates matters to an extent is that many django packages comment out or remove their install_requires lines because of this bug; otherwise they end up with some alpha version of django installed (that has been our experience, and I've seen it in the github issues of many prominent django packages.

@qwcode
Owner

Hey @fdintino , I rigged up a basic test with requirements, -U, and --no-deps and it seemed to work, i.e. the items in the requirements file were upgraded, but no dependencies. that's consistent with my understanding of what the code is doing. it's tedious in there though, so there could be a failing with this idea.

for the case where a top-level requirement has a new "install_requires" dependency, that's why I mentioned the 2nd pass command to install those.

for the case where there are "are dependencies which have updates to their install_requires", I'm not sure I follow you there. you would only discover and want to act on that, if you were wanting to do a full recursive upgrade, right?

@fdintino

Only if you no longer met the requirements. So, for example, if I updated django-sentry, and django-sentry required django-celery>=2.5.4,django-celery<3.0where it previously required django-celery>=2.5.3,django-celery<3.0 (perhaps because of a bug fix or a new feature), and I currently have django-celery==2.5.3, then I would expect it to update django-celery to satisfy the requirement, the way my patch does. However, if I happened to have django-celery==2.5.4 I would not expect it to update celery. In the case of celery it's not a big deal, but when packages have Django>1.2,Django<=1.4, and projects are often written to target Django 1.2, 1.3, or 1.4, an unexpected upgrade of django can be a huge headache.

@qwcode
Owner

thanks @fdintino . I follow now. you don't want to cut off all recursive behavior (that will do necessary upgrades to fulfill requirements), just want to stop recursive "forced" upgrades.

btw, your comment is cropped due to formatting. might want to fix that for others.

@qwcode
Owner

I posted a gist that breakdowns achieving the desired behavior using the 2 commands in sequence mentioned above.
Not that it invalidates the desire for this ticket or the pull, but it was helpful to me to confirm how
it works currently and what's possible currently.
(hint: in the example, "b" is the parallel to django that was mentioned as a concern)

https://gist.github.com/3088149

comment appreciated on the gist if this is not the scenario.

@fdintino fdintino referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.