Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

bundle install extremely slow (to resolve dependencies) #2125

Closed
jarl-dk opened this Issue · 22 comments

9 participants

@jarl-dk

On this small (artificial) project (only having a Gemfile and a specfile) bundle spends 33 minutes to resolve dependencies...

The demonstrating project can be seen on https://github.com/softace/slow_bundle_install_demo

I find it kind of odd that it should take so long time.

@alirz23

weird, I does take long to resolve the dependencies, I am on it should be resolved soon

@dblock

#2248 may be related

@jarl-dk

@alirz23: Are you on it? It seems like (from #1083 (comment)) that the problem only occurs when pulling in dependencies from a gemspec file.

@jarl-dk

I have investigated a bit more:
I have run bundle install --verbose and it always hangs after the following lines of output:

Unmet Dependencies: ["tenderlove-frex", "spicycode-rcov"]
Fetching gem metadata from http://rubygems.org/
Query List: ["tenderlove-frex", "spicycode-rcov"]
Query Gemcutter Dependency Endpoint API: tenderlove-frex spicycode-rcov
Fetching from: http://rubygems.org/api/v1/dependencies?gems=tenderlove-frex,spicycode-rcov
HTTP Success
Query List: []

Just like these people describe:
http://stackoverflow.com/questions/13443332/bundle-update-stuck

The problem is that tenderlove-frex and spicycode-rcov are old gems hosted on http://gems.github.com/, see http://gems.github.com/list.html.

It seems like there are old versions of gems out there that depends on
these gems that no longer exists, and while trying to resolve a
dependency, bundler is trying these versions of gems. Most preferably
the versions of gems that depends on tenderlove-frex or
spicycode-rcov should be deleted (authors should be
contacted). Until then bundler should be fixed to behave well in those
scenarios.

I have further figured out that running with DEBUG_RESOLVER=1 will show the activities of the resolver.

@jarl-dk

By applying this patch to edge:

commit 3d6a175fd2630fb13b7d9169ec606c35ec1268d3
Author: Jarl Friis <jarl@softace.dk>
Date:   Tue Jan 29 12:57:24 2013 +0100

    Added dots for activity

diff --git a/lib/bundler/resolver.rb b/lib/bundler/resolver.rb
index 236eedd..1f0a390 100644
--- a/lib/bundler/resolver.rb
+++ b/lib/bundler/resolver.rb
@@ -122,6 +122,7 @@ module Bundler
     # <GemBundle>,nil:: If the list of dependencies can be resolved, a
     #   collection of gemspecs is returned. Otherwise, nil is returned.
     def self.resolve(requirements, index, source_requirements = {}, base = [])
+      Bundler.ui.debug "Resolving"
       base = SpecSet.new(base) unless base.is_a?(SpecSet)
       resolver = new(index, source_requirements, base)
       result = catch(:success) do
@@ -133,6 +134,7 @@ module Bundler
     end

     def initialize(index, source_requirements, base)
+      @iteration_counter    = 0
       @errors               = {}
       @stack                = []
       @base                 = base
@@ -166,6 +168,11 @@ module Bundler
       # gem dependencies have been resolved.
       throw :success, successify(activated) if reqs.empty?

+      @iteration_counter += 1
+      if((@iteration_counter % 10000) == 0)
+        Bundler.ui.debug ".", false
+      end
+
       debug { print "\e[2J\e[f" ; "==== Iterating ====\n\n" }

       # Sort dependencies so that the ones that are easiest to resolve are first.

I figured out it has nothing to do with obsolete gems, it is a pure resolver issue.

@rubycut

+1 for this bug, takes over 5 mins.

@indirect
Owner

If you throw a Gemfile at Bundler that takes a bajillion tries to resolve, it'll take a while. This isn't a bug. That said, if you guys have ideas for how to improve the resolver, I'm very open to pull requests.

@indirect indirect closed this
@rubycut

It is a bug, it takes forever. Even if you already have Gemfile.lock and you just want to update one gem.

@indirect
Owner

@rubycut please see ISSUES.

@indirect indirect reopened this
@indirect indirect closed this
@indirect
Owner

@rubycut @jarl-dk okay, so I actually investigated this particular resolve today. I just applied #2303, and I was hoping that it would fix this problem, too. It doesn't, but that's because this slow resolve is particularly and especially nasty. Here's a excerpt of the final lock:

    activemodel (3.0.20)
      activesupport (= 3.0.20)
      builder (~> 2.1.2)
    builder (2.1.2)
    cucumber (1.2.1)
      builder (>= 2.1.2)

In retrospect, it's obvious that only builder 2.1.x will satisfy the Gemfile, but that's because activemodel (via rails) demands it. If you just look at cucumber, it seems like ANY builder version will be okay. The reason this gemfile takes so long to resolve is that Bundler tries every version of activemodel/rails/etc looking for one that's compatible with builder 3.0, as requested by cucumber.

To speed this up, either add a line to your gemspec noting that you need builder ~> 2.1.2, or switch to a version of cucumber that has a more sane version constraint.

If either of you think there's a way to figure this out in less time without changing the Gemfile or gemspec, I would love to hear about it.

@indirect
Owner

Whoops, I guess that was an incomplete answer. rack-test has the exact same problem (capybara says it will take any version, rails is strict). You need to explicitly state that your gem needs ~> 0.5.7 in order to get the version that will work with the Rails version your Gemfile requests.

@dblock

@indirect I think this is a backtracking optimization problem. Maybe there's a way to sort gems as dependencies are satisfied. In your example bundler tries "every version of activemodel/rails", but is it true that there're simpler constraints to be satisfied further that would reduce this space significantly?

@indirect
Owner

Yes, and yes. The trick is figuring out how to abstract that into an algorithm that doesn't break something somewhere else. :)

@rubycut

Also there should be some kind of warning or progress bar, I am letting bundle update run for 5 minutes and nothing happens, I am not sure how long should I wait, 30 minutes, two hours? If I upgrade one gem only it should last for 30 seconds max, or it should at least print some kind of warning that it will take long time. And I also saw some other issues which suggest there is some kind of recursive loop problem.

@dblock

@rubycut - you can't have a true progress bar as you're backtracking, because you discover new things as you move forward all the time, but some indication of space being searched could be useful - maybe just a "." every 5 seconds to indicate that it's making progress.

@indirect
Owner

@rubycut oh man. If it were possible to know it will take a long time, then I would have a solution for NP=P, and I would be off being a mathematical celebrity instead of working on bundler. :)

I haven't yet found a single recursive loop problem. Every single one of those claims has turned out to be a lot of gem versions. And you have just described the exact uncertainty of an NP-complete problem. Resolving is inherently impossible to predict, timewise. Do you think I should just cut off the resolve after X minutes and tell people "too bad, your Gemfile is too slow, try again"? That's not exactly helpful.

@indirect
Owner

@rubycut that said, if you can figure out it's logically possible to do what you just said, I would love a patch.

@rubycut

@indirect sorry, no time to work on this beyond suggestions. I still think some kind of warning after 30 seconds would be advisable. Or ideally, it should stay after 30 second what is he doing.

@TangMonk

Same problem.
I want to install this gem, only a few dependencies gem.
But when resolving dependencies, bundle Through all of my gem! Is it necessary?

@arijoon

@rubycut you can run the installer with --verbose flag to see what it's doing as jarl-dk explained! Progress bar would be useless since there is no way to predict the completion!

@jriff

Well... you are right about the progress bar but outputting a few periods now and then would probably be enough to let the user know that stuff is actually happening?

@TimMoore
Owner

Printing a dot every second was added in fa909d6 and released in v1.3.5. In addition, v1.6.0 has a completely rewritten resolver that dramatically improves some of these pathological cases.

If you're not seeing the dots, please open a new issue and we'll try to investigate.

If you have other suggestions for improving the resolver, you can add an issue in the bundler-features repo or submit a pull request here. Thanks!

@jaimerson jaimerson referenced this issue in carrierwaveuploader/carrierwave
Closed

Bundle install takes forever #1419

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.