"bundle install" doesn't respect version in Gemfile.lock when using source option/block (1.7.3)

[mdt]

When a gem (let's call it "MyGem") has a specific remote source option set in the Gemfile, and Gemfile.lock specifies an earlier-numbered version for MyGem, then in some cases, "bundle install" incorrectly selects the latest version of MyGem and changes Gemfile.lock to specify the latest version of MyGem. Reproducing this problem seems to require all of the following:

• Gemfile specifies multiple remote sources.
• Gemfile requires a specific source for MyGem.
• The local gem cache (maybe: bundler installation?) knows about a more-recent version of MyGem than the one specified in Gemfile.lock.

If any one of the above isn't true, then "bundle install" correctly installs the locked version of MyGem and leaves Gemfile.lock unchanged. But if all three bullet-points hold, then "bundle install" always changes Gemfile.lock to grab the latest version of MyGem, even though we could have (should have) used the locked version. This seems like a bug.

This long script reproduces the problem:

https://gist.github.com/mdt/426195a829c9aa681bf1

[indirect]

Thanks for reporting this!

On Tue, Sep 16, 2014 at 11:15 AM, Marc Tanner notifications@github.com
wrote:

When a gem (let's call it "MyGem") has a specific remote source option set in the Gemfile, and Gemfile.lock specifies an earlier-numbered version for MyGem, then in some cases, "bundle install" incorrectly selects the latest version of MyGem and changes Gemfile.lock to specify the latest version of MyGem. Reproducing this problem seems to require all of the following:

• Gemfile specifies multiple remote sources.
• Gemfile requires a specific source for MyGem.

• The local gem cache (maybe: bundler installation?) knows about a more-recent version of MyGem than the one specified in Gemfile.lock.
  If any one of the above isn't true, then "bundle install" correctly installs the locked version of MyGem and leaves Gemfile.lock unchanged. But if all three bullet-points hold, then "bundle install" always changes Gemfile.lock to grab the latest version of MyGem, even though we could have (should have) used the locked version. This seems like a bug.
  This long script reproduces the problem:
  https://gist.github.com/mdt/426195a829c9aa681bf1

  Reply to this email directly or view it on GitHub:
  "bundle install" doesn't respect version in Gemfile.lock when using source option/block (1.7.3) #3175

[TimMoore]

Still reproducible in Bundler 1.7.6. Thanks for providing the script, @mdt. I'll try to get this sorted out soon.

[TimMoore]

I have a tentative fix for the final step of the repro script at https://github.com/TimMoore/bundler/compare/bundler:1-7-stable...issue-3175-unlocked-sources

This is the case where both remotes are already in the lock file, and the gem is there locked to the older version. It was incorrectly calculating that the sources had changed and re-resolving the entire Bundle.

The earlier case (where you add a new source to the Gemfile) is trickier. Bundler intentionally unlocks the bundle and regenerates the lock file when the sources change. This is to allow for the case where a gem changes sources (for example, from a gem server to git).

I think we might be able to get around this by refusing to unlock the version of a gem unless the type of its source changes. @indirect do you think this makes sense? Can you think of any problems that would cause?

[indirect]

@tmoore I think that sounds like the behavior that we want, yeah.

On Sun, Nov 16, 2014 at 4:21 PM, Tim Moore notifications@github.com
wrote:

I have a tentative fix for the final step of the repro script at https://github.com/TimMoore/bundler/compare/bundler:1-7-stable...issue-3175-unlocked-sources
This is the case where both remotes are already in the lock file, and the gem is there locked to the older version. It was incorrectly calculating that the sources had changed and re-resolving the entire Bundle.
The earlier case (where you add a new source to the Gemfile) is trickier. Bundler intentionally unlocks the bundle and regenerates the lock file when the sources change. This is to allow for the case where a gem changes sources (for example, from a gem server to git).

I think we might be able to get around this by refusing to unlock the version of a gem unless the type of its source changes. @indirect do you think this makes sense? Can you think of any problems that would cause?

Reply to this email directly or view it on GitHub:
#3175 (comment)

[TimMoore]

I made a PR for the final step at #3301.

I'm still having some trouble getting it to ignore newly added sources without causing other problems.

[jezstephens]

Still reproducible in 1.8.2. Surely this is a fairly common use case. Is there a workaround I'm missing?

[indirect]

Simply refusing to unlock the gem if the source type stays the same is not okay: if the Gemfile moves a gem from one source to another, the version must be unlocked to allow versions that are present in that source even if the locked version is not present in that source.

On Mon, Feb 16, 2015 at 3:23 PM, Jez Stephens notifications@github.com
wrote:

Still reproducible in 1.8.2. Surely this is a fairly common use case. Is there a workaround I'm missing?

Reply to this email directly or view it on GitHub:
#3175 (comment)

[indirect]

@jezstephens if this is a frequent problem for you, you're welcome to fix it and send a pull request.

Also, please keep in mind that when you say "gem 'rake'", that means you are okay with literally any version of the rake gem. You got one. If you're not okay with the version you got, please specify versions that you are okay with while we fix this problem entirely for free and in our spare time.

[jezstephens]

Apologies @indirect, I didn't intend my comment in the ungrateful tone you seem to have felt. I was simply surprised at the lack of people having trouble with this, though now I understand the issue a bit better I see it may not be as common an issue as I first thought.

I actually did look into this further, but didn't make any real progress before turning in to bed. All I've found is that while PR #3301 fixes the second error case in @mdt's script (once updated to reflect the current version of rake), I still see a similar problem in my real application.

My gems are updating when I run bundle install not only when I add a new source, but also afterwards if I manually roll back the versions in Gemfile.lock then run it again.

when you say "gem 'rake'", that means you are okay with literally any version of the rake gem [...] please specify versions that you are okay with

I don't think this is really about what my Gemfile says, but what my Gemfile.lock says. Being able to lock to specific versions without giving specific versions in the Gemfile is obviously a key feature. In fact I do have version constraints for the affected gems, just not specific versions.

It's also worth noting that this problem results in unwanted upgrades of dependencies not listed in my Gemfile.

Specifying exact versions seems like the best workaround for now though, so thanks.

I'll look into this further when I have more free time. I understand it's frustrating when people expect something for nothing, but please don't be so quick to imply laziness if you want to encourage contributions.

[indirect]

Thanks for clarifying. In general, incredulity that a bug has not been fixed in an open source project is always going to be grating because we are working on this project almost entirely unpaid. As a result, it sounds ungrateful, demanding, or both.

I do believe that it is a bug that versions are unlocked in this case, and we will try to get it fixed. Pull requests are welcome anytime someone has a problem that they genuinely need fixed quickly.

[TimMoore]

@jezstephens could you please add the details from bundle env and reproduction steps (ideally a script)?

My gems are updating when I run bundle install not only when I add a new source, but also afterwards if I manually roll back the versions in Gemfile.lock then run it again.

Can you clarify what you mean by this? Are you reverting the changes to Gemfile.lock entirely? If that's the case, then from Bundler's perspective it's still a new source.

[jandudulski]

reproduction steps

Create a new Gemfile, e.g.:

source 'https://rubygems.org'

gem 'virtus'

source 'https://rails-assets.org' do
  gem 'rails-assets-angular', '1.3.0'
end

and run bundle install

Now change gem 'rails-assets-angular', '1.3.0' to gem 'rails-assets-angular', '~> 1.3.0' and run bundle - for me it updates rails-assets-angular to the newest accessible locally version.

Or change gem 'virtus' to the gem 'virtus', '1.0.3' and run bundle:

[yanoo@hodor]~/Code/gemtest% bundle install
Fetching gem metadata from https://rails-assets.org/..
Fetching gem metadata from https://rails-assets.org/.
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Could not find gem 'virtus (= 1.0.3) ruby' in the gems available on this machine.

In this case I have to manually run gem install virtus -v 1.0.3 and then bundle will work or remove all source blocks - it would also work.

[moxley]

We reproduced this problem in bundler versions 1.7.12, 1.7.3, and it seems to be fixed in 1.8.3 and 1.8.5.

[jandudulski]

Just tested as described above with 1.8.5 - bundler still updates to the highest available version locally.

But there is a progress - issue with not downloading a gem ("Could not find gem ... in the gems available on this machine" issue) seems to be fixed.

[pschambacher]

Still happening with Bundler 1.9. From my point of view Bundler is simply broken for 3 minor versions now.

[indirect]

@pschambacher since users reported this bug as fixed, please open a new ticket for the issue you're seeing in 1.9, following ISSUES so that we can reproduce the bug and fix it.

[pschambacher]

Hi @indirect I've created the issue here with as much details as possible