Very slow dependency resolving

[cmhe]

Describe the problem as clearly as you can

The step to resolve the project dependencies with bundler takes multiple hours.

Did you try upgrading rubygems & bundler?

I tried it with 2.3.17 and a couple of other older releases of bundler:

$ bundle --version
Bundler version 2.3.17

Post steps to reproduce the problem

With this Gemfile:

source ENV['GEM_SOURCE'] || 'https://rubygems.org'

  gem "puppet-module-posix-default-r2.7", '~> 1.0', require: false
  gem "puppet-module-posix-dev-r2.7", '~> 1.0',     require: false
  gem "github_changelog_generator",                 require: false
  gem "puppet-module-posix-system-r2.7", '~> 1.0',  require: false
  gem "librarian-puppet",                           require: false

I also tried it inside a Docker container:

FROM ruby:2.7.6-slim

ADD Gemfile /root/Gemfile

RUN gem install --version '= 2.3.17' bundler && \
        bundle --version && \
        cd /root && \
        bundle config set --local path vendor/bundle && \
        bash -c 'time bundle install'

The build inside the Docker container fails, because of missing build dependencies, however bundler mostly spends resolving dependencies there and the final time output, after it fails to build is:

real    332m27.595s
user    331m56.310s
sys     0m21.727s

So over 5 hours.

Which is the same on my host, where the build of those dependencies succeeds.

Which command did you run?

bundle config set --local path vendor/bundle
bundle install

With the Gemfile posted above in place.

What were you expecting to happen?

Resolving dependencies in under 5 minutes.

What actually happened?

Resolving dependencies in over 5 hours.

If not included with the output of your command, run bundle env and paste the output below

I used the docker container, so the environment should be clear. I can however post it as well if required...

[indirect]

Wow, congratulations on finding such a pathological resolver case! It's been years since I've seen one this bad. 😁

I suspect it will take some time to fix this, so in the meantime, here is a workaround you can use to get a complete resolution and a working Gemfile.lock:

Start with this Gemfile:

source "https://rubygems.org"

Then run:

• bundle add puppet-module-posix-default-r2.7
• bundle add puppet-module-posix-dev-r2.7
• bundle add puppet-module-posix-system-r2.7
• bundle add github_changelog_generator
• bundle add librarian-puppet

Commit the resulting Gemfile.lock, and you should have something you can use for now.

[ansonhoyt]

I don't plan to wait 5 hours 😅, but I may be running into this too as I update a very small app.

The OP's report seems clear, but in case it helps, I'll just note the scenarios that I found slow resolution:

• Existing project with bundler 2.3.17.
• It took bundle update in a few seconds on ruby 2.7.5, then ruby 2.7.6, and then again after updating to ruby 3.0.4. (normal)
• Trouble started when tried updating rails:

  # Gemfile
  -gem 'rails', '~> 6.1.4'
  +gem 'rails', '~> 7.0.3'

• It took a couple minutes for bundle update rails to successfully identify version conflicts with a couple gems. (slow)
• After update gemspec dependencies in the conflicting gems, I set aside the Gemfile update to (successfully) update those dependencies with bundle update completing in seconds (normal against rails 6.1.4).
• Now re-trying bundle update rails has been working for tens of minutes (>4.1k progress dots) using 100% of a single M1 Max core. (very slow)

Thanks for the workaround. I'll try that now.

[ansonhoyt]

The workaround helped. 🙏

bundle add rails --version '~> 7.0.3' quickly showed Bundler could not find compatible versions:

Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies.....
Bundler could not find compatible versions for gem "activemodel":
  In snapshot (Gemfile.lock):
    activemodel (= 6.1.6)

  In Gemfile:
    populate was resolved to 0.1.0, which depends on
      activemodel (>= 5.2, < 6.2)

    rails (~> 7.0.3) was resolved to 7.0.3, which depends on
      activemodel (= 7.0.3)

Deleting your Gemfile.lock file and running `bundle install` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

After I allowed Rails 7.0 in populate's gemspec (my private gem), upgrading withbundle update worked normally/quickly. 🎉

[nvasilevski]

✅ UPD: Running into #5385
Most likely not related to the current thread

Hey folks, sorry for chiming in 🙇
I was wondering if @ansonhoyt (or anyone experiencing the same) could run bundler with --verbose flag to see if anything specific is taking the most of the time. We are looking into some dependency resolution slowness and in our case bundler tries to fetch every available gemspec.rz of a particular gem, like:

Most likely called from here:

rubygems/bundler/lib/bundler/fetcher.rb

Line 113 in 46a92b2

Bundler.load_marshal Bundler.rubygems.inflate(downloader.fetch(uri).body)

It becomes better on the second run because of the cache, I assume, but still takes significant time because of the iteration over all available versions:

rubygems/bundler/lib/bundler/fetcher.rb

Lines 110 to 111 in 46a92b2

	elsif cached_spec_path = gemspec_cached_path(spec_file_name)
	Bundler.load_gemspec(cached_spec_path)

It doesn't seem to do it on bundle _2.2.33_ update --verbose

I'll poke around a bit more to have a clear reproduction and maybe a suggested solution

Thanks!

[deivid-rodriguez]

I think you're running into #5385.

[jesse-shopify]

I'm using Bundler 2.3.4 until #5385 is resolved, in apps where there are private gems with many versions.

[deivid-rodriguez]

If you're using a geminabox based server, I recommend you check out geminabox/geminabox#435!

[deivid-rodriguez]

Just tried this reproducer against #5960 and it now resolves in 5 seconds! 🎉