Bundler resolver blows up with many available versions #1671

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
4 participants
Contributor

pmahoney commented Feb 23, 2012

I've seen Bundler's resolver blow up (takes a very long time, but does complete) in a production situation where we have a moderate number of dependencies (say 10-20) where some of these have many versions available in the repository (three gems have at least 40 available versions).

I think I've reproduced this situation in a unit test. It's not a very good test, but it serves to illustrate the problem.

I don't (yet...?) have a solution to the resolver algorithm. Would be a fun problem if I can find the time.

Owner

indirect commented Feb 23, 2012

In general, I recommend that people add optimistic version limits to their Gemfiles in an attempt to avoid this scenario. For example, gem 'foo', '~> 1.2' when the user wants something between 1.2.x and 1.9.x. Since dependency graph resolution is an NP-complete problem that we can't solve by approximation (ha), it's possible that there is no significant improvement available to be made. That said, if anyone would like to take a stab at it, that would be awesome.

Contributor

pmahoney commented Feb 23, 2012

Indeed... There seem to be possible improvements for this particular case (e.g., try activating each first-level dependency against a blank slate so the 'more' gem would succeed after which it would be trivial to find the right version of all the others). But there would still be other worst-case scenarios. (note that ~> can still run into the same problem. in my real-world case, it was actually some gems using >= and some using ~> that caused the backtracking).

Found this interesting and relevant link for anyone working on improvements: Review NP-complete Library Versioning Problem, also Algorithmic Considerations [of package management systems]

edit: perhaps a note in the documentation about what can happen is warranted? or maybe better feedback in "verbose" mode during dependency resolution (but less than the debug output). Both of these are far easier to implement :) Or maybe not; now that I'm using better search terms, it seems to be a well-known problem.

Contributor

pmahoney commented Feb 24, 2012

What is your opinion of a hypothetical alternate algorithm that would either resolve quickly or bail out rather than go into some exhaustive search (perhaps a --exhaustive switch could enable the current behavior)? I'm not sure the workability of this, but based on reading the edos work it seems Debian's APT (and many other dependency resolvers) does not attempt to perform an exhaustive search.

From a UI perspective, it is not ideal for bundler to take a long time (45+ minutes in my particular real-world case) without any feedback. Would you be interested in patches that allowed pluggable resolvers?

Contributor

radar commented Mar 12, 2012

I agree with Andre on this one. You should be specifying exact dependencies in your Gemfile for quicker resolution.

Contributor

pmahoney commented Mar 13, 2012

radar,

I understand "exact dependencies" to be something like gem 'mygem', '= 1.0.1'. I interpret Andre's "optimistic" version limits to be the opposite of that, e.g. gem 'mygem', '~> 1.0'.

Can you clarify?

It's definitely possible to work around my specific problem by playing with version ranges in various ways. But the problem remains that the bundler resolver may take a "long time" with no user feedback.

Contributor

radar commented Mar 13, 2012

Exact dependencies would allow for quicker resolution than optimistic dependency specifying, that much is true. Either way is better than specifying no version at all. I go for the optimistic kind personally.

This pull request fails (merged 209a2d2 into 720b40b).

Owner

indirect commented Feb 15, 2013

Happily, I believe this particular situation to be resolved by the changes to the backtracking behaviour between 1.3.0.pre.7 and 1.3.0.pre.8. :)

indirect closed this Feb 15, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment