Bundle updates are moving dependencies from global to secondary source

[agrobbin]

Bundler is moving a bunch of dependencies from our primary source (https://rubygems.org/) within Gemfile.lock to a secondary source, when there is an overlap of sub-dependencies between sources. That secondary source includes a gem that has one of the same sub-dependencies as the dependencies that were moved, which I think is the likely cause.

The removals are within the https://rubygems.org/ source, while the additions are within a private source (in our case, a GItHub Packages RubyGems registry).

source 'https://rubygems.org'

gem 'rails'

source 'https://rubygems.pkg.github.com/[redacted]' do
  gem 'foo' # <= some private dependency that includes an overlapping dependency with `rails`, such as `activesupport`
end

I originally thought this was a Dependabot-specific issue (dependabot/dependabot-core#9326), but I've now seen it happen when doing gem updates locally, so I think it's actually a Bundler issue.

This has been happening at least since Bundler v2.5.4, and still on v2.5.6.

[deivid-rodriguez]

Does the secondary source include activesupport and the other gems that were moved? If those exist in the non global source, Bundler does prefer that, although I'm not sure it should be changing the lockfile if they're already locked in the global source section.

[agrobbin]

Here is a snippet of the secondary source in the Gemfile.lock, when no additional shared dependencies are included:

GEM
  remote: https://rubygems.pkg.github.com/[redacted]
  specs:
    [redacted] (1.4.0)
      activesupport (>= 5.2, < 7.2)
      google-cloud-pubsub (~> 2.0)
      hc_metrics (~> 0.3)
      sidekiq (>= 5.0, < 8.0)
    [redacted] (0.4.1)
    [redacted] (0.1.1)
      iterable-api-client (~> 0.5)
      mail (~> 2.8)

The secondary source does include activesupport as a dependency, but it is not actually stored in the secondary source repository itself (at least, it doesn't show up in GitHub's Packages list!).

Let me know if I can provide any other details here to help look into the issue.

[deivid-rodriguez]

Yeah, I meant if there was a gem called activesupport stored in the secondary source. If there's no such gem, then this seems like a bug, yeah. Very intriguing.

If you could create a dummy github packages source that simulates the situation, replacing your private gems with dummy gems, that would be ideal.

[agrobbin]

Hmmm @deivid-rodriguez do you have a recommendation on how to create a dummy GH Packages source that others can access? I wasn't aware that that was possible.

[agrobbin]

@deivid-rodriguez I was able to reproduce this by writing a failing test for Bundler, which fails on v2.5.7, but passes on master. The key combination is a secondary source and use of caching. If you remove the bundle :cache line in the test below, tests pass on v2.5.7.

Also notice how it doesn't matter if are any gems within the source block. Just having the additional source defined is enough to trigger the issue! 🤯

context "with multiple sources and caching enabled" do
  before do
    build_repo2 do
      build_gem "rack", "1.0.0"

      build_gem "request_store", "1.0.0" do |s|
        s.add_dependency "rack", "1.0.0"
      end
    end

    build_repo4 do
      # set up repo with no gems
    end

    gemfile <<~G
      source "#{file_uri_for(gem_repo2)}"

      gem "request_store"

      source "#{file_uri_for(gem_repo4)}" do
      end
    G

    lockfile <<~L
      GEM
        remote: #{file_uri_for(gem_repo2)}/
        specs:
          rack (1.0.0)
          request_store (1.0.0)
            rack (= 1.0.0)

      GEM
        remote: #{file_uri_for(gem_repo4)}/
        specs:

      PLATFORMS
        #{local_platform}

      DEPENDENCIES
        request_store

      BUNDLED WITH
          #{Bundler::VERSION}
    L
  end

  it "works" do
    bundle :install
    bundle :cache

    update_repo2 do
      build_gem "request_store", "1.1.0" do |s|
        s.add_dependency "rack", "1.0.0"
      end
    end

    bundle "update request_store"

    expect(out).to include("Bundle updated!")

    expect(lockfile).to eq <<~L
      GEM
        remote: #{file_uri_for(gem_repo2)}/
        specs:
          rack (1.0.0)
          request_store (1.1.0)
            rack (= 1.0.0)

      GEM
        remote: #{file_uri_for(gem_repo4)}/
        specs:

      PLATFORMS
        #{local_platform}

      DEPENDENCIES
        request_store

      BUNDLED WITH
          #{Bundler::VERSION}
    L
  end
end

[deivid-rodriguez]

Nice! Well, coincidentally, it seems that #7516 has also fixed this problem 🎉. I will release the fix for that next week. Happy to get your test contributed to avoid regressing here.

[deivid-rodriguez]

Closing this as fixed by #7516, please do contribute your test case!

[agrobbin]

Great, thanks @deivid-rodriguez! I've opened #7544.