Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip needs a dependency resolver #988

Closed
cboylan opened this issue Jun 11, 2013 · 177 comments
Closed

pip needs a dependency resolver #988

cboylan opened this issue Jun 11, 2013 · 177 comments

Comments

@cboylan
Copy link
Contributor

@cboylan cboylan commented Jun 11, 2013

pip's dependency resolution algorithm is not a complete resolver. The current resolution logic has the following characteristics:

  • Only one top level specification of a requirement (otherwise pip raises a "double requirement" exception)
  • "first found wins" behavior for dependency requirements/constraints, which results in not respecting all constraints. pip 10+ print a warning when this occurs.

NOTE: In cases where the first found dependency is not sufficient, specifying the constraints for the dependency on the top level can be used to make it work.

pip install project "dependency>=1.5,<2.0"

(2019-06-23)

This is being worked on by @pradyunsg, in continuation of his GSoC 2017 project. A substantial amount of code cleanup has been done, and is ongoing, to make it tractable replace the current resolver, in a reasonable manner. This work enabled pip >= 10 to warn when it is going to make an installation that breaks the dependency graph. (The installations are not aborted in such scenarios, for backwards compatibility.)


(2019-11-29)

The most recent update regarding this is available here.

@dstufft
Copy link
Member

@dstufft dstufft commented Jun 11, 2013

I added this to 1.5 because I think it's a pretty big wart that should be sorted out.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Jun 11, 2013

for now, the "pip solution" is to declare in your requirements file (or as an install argument directly) what specifier you want, and it overrides what happens down in the dependency tree.

i.e. in this case, put "foo>=1.0,<=2.0" in your requirements or add it as an install argument.

also, pip compiles everything it's going to install into a set first, then installs.
it's not installing everything as it's discovered, and it's discovery order is not depth first.
the order of discovery isn't the issue, but rather the lack of doing any resolution as it's compiling the set.
Currently, first found wins.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Jun 11, 2013

I keep meaning to add a "conflict resolution" section to the Internal logic section to the docs that covers this.
practically speaking, pip's override logic is pretty nice IMO, and is all most people need most of the time.

@emonty
Copy link
Contributor

@emonty emonty commented Aug 11, 2013

I believe, based on discussions with @dstufft, that this may be related to the problems people will have if they try to upgrade from old distribute to new setuptools in the same set of software as other things. Or, more specifically, if you depend on a piece of software that depends on distribute with an unbounded high end, and you are currently running on a pre-merge distribute, what will happen is that the dependency discovery will put distribute into the list along with the rest of the things that software A depends on, then it will add distribute's dependency, setuptools, to the list, later. THEN, when it's installing, distribute will be upgraded, which removes the distribute provided setuptools code, then the next thing in the list is installed, which is not setuptools, and which fails because setuptools is not installed yet.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Aug 11, 2013

@emonty the distribute to setuptools upgrade problem you speak of is described in #1064, but also here http://www.pip-installer.org/en/latest/cookbook.html#importerror-no-module-named-setuptools

but this is not related to pip's conflict resolution shortcoming issues described in this issue.

@dracos
Copy link

@dracos dracos commented Aug 29, 2013

Just had this issue installing a project - on a system that already had python-dateutil 1.4 installed - that depended upon python-dateutil (no specific version) and django-tastypie 0.9.16. Even though django-tastypie has a python-dateutil dependency of >=1.5 (and not 2.0), and the parent has no specific version dependency on python-dateutil, python-dateutil remained at 1.4. This seems pretty fundamentally against what a package installer should be doing in such a circumstance :)

@dstufft
Copy link
Member

@dstufft dstufft commented Aug 29, 2013

FWIW I've been experimenting with making a real resolver for pip.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Aug 29, 2013

@dracos there is a "solution" right now (short of a new resolver for pip; although that would be nice). Specifically declare what you want the python-dateutil requirement to be in a requirements file, or as a top-level pip install argument, and that will be honored.

http://www.pip-installer.org/en/latest/cookbook.html#requirements-files
http://www.pip-installer.org/en/latest/logic.html#requirement-specifiers

pip install myproject datetutil>=1.5,<2.0

@dracos
Copy link

@dracos dracos commented Aug 29, 2013

@qwcode Thanks, I know that and will have to do so - but this means anyone working on the project will have to manually check all dependencies. Say someone upgrades django-tastypie and it now requires a later version of python-dateutil, there's no way to detect this besides manual checking of each dependency upgrade/install. Oh well.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Aug 29, 2013

@dracos understood. it's a pain point. but just want others who find this, to at least know, there is some kind of solution.

@qwcode
Copy link
Contributor

@qwcode qwcode commented Aug 29, 2013

@dstufft as you work on a new resolver, keep in mind that top level requirements (i.e. pip install arguments are requirements file entries) will still have to be considered overrides (or dominant). that's a pip feature. Also, to state the obvious, this change will be "backwards incompatible" in the sense that many installations will turn out differently.

@dracos
Copy link

@dracos dracos commented Aug 31, 2013

I've made a script (that is basically a patch to pip's prepare_files(), but I didn't want to patch pip, I can't control pip everywhere I use it for example) - available at https://github.com/dracos/check-pip-dependencies - that notes multiple dependency requests for the same package and outputs a list of conflicts, that you can then manually resolve using the methods suggested above.

@benoitbryon
Copy link

@benoitbryon benoitbryon commented Sep 24, 2013

This ticket looks like #174.

@ghost
Copy link

@ghost ghost commented Dec 19, 2013

The pydata people at continuum analytics have built a packaging toolchain
more suited to scientific packaging (That's good, it can get funky).
Facing the same issues, they've implemented a dependency resolver using a SAT solver,
see here. The paper mentioned there is readable and they've open sourced the wrapper package
around the (open as well) SAT solver engine.

Basically, it's all there.

@Ivoz
Copy link
Member

@Ivoz Ivoz commented Feb 18, 2014

@y-p the only problem with that is that it relies on C code, rather than pure python. I presume that it would notionally be unacceptable to expect that everyone installing pip have a C compiler easily available on their system; thus you would have to provide a compiled pip for every system Python hopes to run on. That is a major ask (anyone compiling and distributing for python arm?).

It might be "all there", but it's in an unusable form for pip's general audience.

@merwok
Copy link

@merwok merwok commented Feb 18, 2014

MIT-licensed pure Python: https://github.com/ActiveState/depgraph
(I just know of it, never tested it on the field)

@Ivoz
Copy link
Member

@Ivoz Ivoz commented Mar 4, 2014

0install used to be based on python (now on OCaml), and also wrote a pure python solver, which can be found in their 1.16 release (also in sat.py); afaik it was py23 compatible.

Also some good notes on sat solving on their page, and an awesome email.

minisat paper; GRASP paper

@Wilfred
Copy link
Contributor

@Wilfred Wilfred commented Mar 14, 2014

pkglib has recently released a pure Python dependency solver, FWIW: https://github.com/ahlmss/pkglib/blob/master/pkglib/pkglib/setuptools/dependency.py#L596

@dstufft
Copy link
Member

@dstufft dstufft commented Mar 14, 2014

Thanks, I'll take a look at it when I take a look at the enthought one as well.

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Jun 30, 2014

I've been reading some of the references. One thing I don't see in any of them is how version dependencies are managed - specifically, if package A depends on B (version >=2.0) how is that encoded. Is each version of a package treated as an entirely separate object for resolution purposes? Does anyone have any pointers to references on how this is handled?

@brainwane
Copy link
Member

@brainwane brainwane commented Jul 30, 2020

Per #8511 we have now released pip 20.2. This release includes the beta of the next-generation dependency resolver. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag. Please see our guide on how to test and migrate, and how to report issues. Please report bugs using this survey or by opening a new GitHub issue, not commenting on this one.

The new dependency resolver is off by default because it is not yet ready for everyday use.

We plan to make pip's next quarterly release, 20.3, in October 2020. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

Please spread the word by pointing to this blog post -- spread the word on Hacker News, Reddit, Twitter, Facebook, Dev.to, Telegram, relevant Stack Overflow answers, your favorite Slacks and Discords, etc. Most of the people this will affect do not keep up with Python-specific developer news. Help them get the heads-up before October, and help us get their bug reports.

@thomasf
Copy link

@thomasf thomasf commented Aug 3, 2020

Please consider adding a minimal version selection to pip argument for this resolver, that means only consider the minimal versions specified by each dependency version spec. Go modules uses this and it's the least intrusive version spec system I have seen.

It enables having one pip requirements.txt file without lock files and always results in a reproducible version dependency tree.

If you want to force an upgrade for a sub dependency, just add it to your requirements file or I guess a constraints file. If all package in the resolver has no minimal version set for one specific dependency maybe that could generate a warning because you probably want to add a constraint if that happens.

I guess exceeding a maximum version should still generate an error.

It's a little bit more effort for the project using a feature like this but it also brings repeatable installs without additional lock files and more programs just to manage than lock file.

It would be really nice to move off massively complex tools like pipenv and poetry and be able to skip the concept of a lock file and still get reproducible builds. I'm never interested in the latest possible version of a dependency, I'm always interested on the one I have tested my software with.

After a brief look this is probably best implemented by writing a separate resolver for resolvelib, right?

@brainwane
Copy link
Member

@brainwane brainwane commented Oct 21, 2020

Thanks to everyone who tested pip 20.2 and provided bug reports and feedback, or who spread the word! We also made a video you can share.

We are aiming to release pip 20.3 about a week from now, on Wednesday or Thursday, Oct 28 or 29. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

For more on the rollout and how you can help, see #6536 (comment) -- starting tomorrow, @di is gathering a volunteer first-response team to help reply to confused users.

@brainwane
Copy link
Member

@brainwane brainwane commented Nov 18, 2020

As I discussed in a comment elsewhere we decided to delay the release slightly, because of some CI problems cropping up and because of some external factors. pip 20.3b1 is available in case you want to try that out.

In today's team meeting we agreed that the 20.3 release will likely be tomorrow or Friday. You can follow #8936 for more.

We've also substantially improved the "what's changing" user guide so please take a fresh look and circulate it!

And the new resolver is already solving some people's issues, which is great!

@brainwane
Copy link
Member

@brainwane brainwane commented Nov 29, 2020

We have now resolved a finicky Mac OS Big Sur support issue and a headache-inducing infinite resolution issue #9011, which were stopping us from releasing. Per #8936 (comment) the pip 20.3 release, in which the new pip resolver will be the default, will very very likely be tomorrow, Monday, 30 November.

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Nov 30, 2020

pip 20.3 has been released, and it has the new resolver by default! Here's our release announcement on the PSF blog: https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html

@pradyunsg pradyunsg closed this Nov 30, 2020
@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Nov 30, 2020

That felt goooood. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.