Provide an option to force a gem version, even it it conflicts with a gem's declared compatibility.

[coilysiren]

From @xaviershay on August 22, 2013 3:22

From #2412

Still debate about how this would be done or whether it's a good idea. See linked ticket for context.

Copied from original issue: rubygems/bundler-features#20

[coilysiren]

From @xaviershay on August 22, 2013 3:25

Also this #1549

[coilysiren]

From @rmetzler on September 3, 2013 23:31

Today I started to look into this.

I read about all proposals for the API. The DSL style I liked best and is the easiest to implement is the following notation:

gem 'mail', '~> 2.4.4', :override => true

This notation means, that the resolver uses the version marked by the override option whenever two dependencies are compared.

Requirements

• Bundler should warn about this, because it is a potentially dangerous option.
• If Bundler compares two conflicting dependencies both having :override => true then Bundler should throw a special VersionConflict error.

Questions

As I'm new to the Bundler source, trying to find my way around I recognized some minor issues. Also I'm not sure whether I should it ask here or on the mailing list. I'll just write it down here because it's easier now. Please advise!

1. I didn't found a test case for Bundler::VersionConflict so I added one. It's modeled after a detailed description of the problem, but could be simpler though.
2. I found this line that doesn't look right, especially when you consider this constructor. Not really sure about that, had a few errors when I changed it and run tests.

[coilysiren]

From @indirect on September 5, 2013 5:42

I think that :force_version => true might make it clearer what the option does, since there are other options that could be passed as well, making :override a bit ambiguous (override what? and with what?).

I'm happy to merge a testcase for VersionConflict. 👍 Unit tests in general are a weak area for the Bundler codebase. 😬

The line you point to doesn't look right to me either, but maybe there's another Bundler::Dependency hiding somewhere?

[coilysiren]

From @rmetzler on September 5, 2013 9:3

Thank you for you reply, very much appreciated. I also think :force_version => true is a clearer option.

I'll send a pull request for VersionConflict and investigate further in dep helper method.

[coilysiren]

From @sheerun on October 17, 2013 13:11

I don't think it's good idea, forking is better solution.

[coilysiren]

From @rmetzler on October 17, 2013 14:46

Hi,

first thing I've to say, I'm sorry I didn't send any pull request for the test cases yet.

@sheerun for discussing the proposed experimental feature in question please read over the discussion first here #1549 and and #2412
I think this is the best description for the current state of how they to fix gem dependency conflicts

I just want to fix the problem I encountered myself more than once. This feature would be experimental and so far I've not written anything else than tests, because the bundler code in question is fairly untested yet. Only "happy paths" were covered with tests in master branch and not conflicting dependency versions.

I also thought about writing a CLI tool (different from bundler) that would know about git, Github and Gemfiles and could do something like the linked comment suggested automatically incl running tests and send pull requests. If Ruby devs think this might be a better idea, please let me know.

Recently I also worked with VersionEye on their open source CLI veye and I realized that there are many issues around dependency versions. This is the case especially in Ruby and nodejs communities where the lib ecosystem is growing rapidly. But every package manager has this problem and if there are good solutions out there, please let me know and I'll investigate the code.

Richard

[coilysiren]

From @aantix on February 8, 2014 0:16

👍 Running into this very issue today. I know the versions are compatible. But one author has been too strict with his requirements..

[coilysiren]

From @korbin on March 11, 2014 20:12

+1 for this feature.

Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override.

[coilysiren]

From @indirect on March 11, 2014 21:24

Again, the issue is not that we actively want to make it harder. The issue is that allowing this means there is no longer a canonical source of truth about what a gem’s dependencies are. When these kinds of changes cause resolver errors (and they will), it will be extremely hard to diagnose. I’m fine with “at your own risk”, I’m not fine with “makes people complain that Bundler is broken when it’s actually something they did themselves”.

If you’re competent enough to debug forcibly overridden dependencies yourself, you’re competent enough to either hack this feature into Bundler or fork and update the dependencies yourself. This feature hands a giant foot-gun to those who won’t know how to debug the issues that come up while using it.

On Mar 11, 2014, at 1:12 PM, korbin notifications@github.com wrote:

+1 for this feature.

Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily, tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override.

—
Reply to this email directly or view it on GitHub.

[coilysiren]

From @TimMoore on March 11, 2014 22:9

And think of the poor gem authors that will suddenly have a pile of mysterious bug reports caused by someone using a gem that was listed as explicitly incompatible!

[coilysiren]

From @strzibny on May 7, 2014 7:41

I would also welcome this feature as this overly strict requirements are happening quite often. Providing an override option seems reasonable. Of course it's always on the risk of the user.

[coilysiren]

From @anri-asaturov on June 14, 2014 14:21

This feature becomes more and more important with number of gems growing. .NET has it. Java has it. Nothing bad happened to them because of it.

[coilysiren]

From @indirect on June 28, 2014 4:16

I’m slowly warming up to this, as long as the override is set explicitly in the gemfile on a line for the gem being overridden, and the overridden version is called out during bundle install. Something like:

# Gemfile
gem "puma", "5.1" # depends on rack <= 2.0
gem "rack", "3.0", force_version: true

$ bundle install
[…]
Installing rack 3.0 (forced version, conflicts with puma)…
Installing puma 5.1…
[...]

[coilysiren]

From @thewoolleyman on November 18, 2014 5:20

+1.

As mentioned on the closed bug thread, one major exacerbating factor of this problem is that pessimistic constraints unnecessarily specify all three components of a semantic version, when they should only need to specify two to prevent API-incompatible changes in dependencies.

Rubygems.org unfortunately encourages this practice by always suggesting a pessimistic constraint with all three versions, which leads gem authors to do the same practice in their gemspecs.

[coilysiren]

From @TimMoore on November 18, 2014 5:53

Maybe Rubygems.org should be changed to recommend setting only major and minor versions?

[coilysiren]

From @endymion on June 27, 2015 22:56

I respectfully add my vote to this feature request. Bundler's lack of a feature for forcing a version that's known to work has cost me hours of extra work today. It's an even bigger issue when you're dealing with older Ruby code that uses gems where the authors have locked the dependencies to something like "~> 1.2.5" and then stopped maintaining a gem. Sometimes a gem doesn't really need to be updated because it works, but it's still full of over-specific dependencies like that. The only way to deal with it right now is to fork the gem itself and update the dependencies. Bundler could easily have made my life a lot simpler today by giving me more flexibility and control and letting me take responsibility for my own QA.

[coilysiren]

From @indirect on June 28, 2015 19:47

@endymion Thanks for the feedback. Today, no one on the Bundler core team has the time to do the work that this feature would involve. If someone joins the team who wants to implement and support the feature, or Ruby Together is willing to fund work on this feature, we'll take another look.

[coilysiren]

From @thewoolleyman on June 29, 2015 19:58

@indirect I do agree with your previous comments that this could make the situation worse for Gem maintainers, because they don't know if a bug report is due to an incompatible overridden dependency version being used.

I still think the long-term solution is to encourage/force people to use pessimistic constraints correct. E.g., docs on rubygems.org, and rubygems itself should complain loudly if you try to put a dependency version in a gemspec with a three-component pessimistic constraint, because in all likelihood You're Doing It Wrong.

Maybe someone (with more time and who cares more than me) can open that discussion on the RubyGems mailing list along these lines (and cross-link it here in a comment), and see what comes of that...

-- Chad

[coilysiren]

From @indirect on June 29, 2015 20:9

The RubyGems.org pages have all changed to default to ~> X.Y version constraints available for copying. I think it might be worth it to warn if a gemspec has a pessimistic constraint on a bugfix release, but unfortunately there are many, many, many, gems that only function correctly at a precise bugfix release. :( It's a chicken and egg problem that I'm not sure how to progress on.

[coilysiren]

From @jfinkhaeuser on July 1, 2015 13:37

If a gem has a constraint on a specific bugfix release because that's the only version it works with, that's a bug in the dependency.

Or, to go by example:

• Gem A exists in version 0.1.1
• Gem B requires precisely version 0.1.1, because it doesn't work with version 0.1.0

Then Gem A uses semantic versioning wrong. That cannot be emphasised enough.

The point of semantic versioning is to provide a stable interface across all releases with the same major and minor component. A bugfix that breaks the interface is simply no longer compatible with the same major and minor component.

Penalizing Gem B for doing the only thing left for it to do seems like entirely the wrong approach.

Gem B should specify dependencies as ~> 0.1 >=0.1.1 in this case. Being helpful about this would IMHO be a good feature.

But really, it's the Gem A maintainers that need to be given a hard time, not anyone else.

(Mind you, there is no easy solution to this. The best solution so far is to differentiate between interface and package versions, and depend on an interface version. That's happening to a degree in Linux/Solaris shared object libraries, and vastly improves some issues, but it's not bullet proof either.)

[coilysiren]

From @booleanbetrayal on October 30, 2015 15:14

would definitely find this feature useful!

[coilysiren]

From @xeger on November 13, 2015 17:10

I deal with a lot of abandonware gems whose authors never learned to manage dependencies correctly. I get bit by falsely-rigid constraints several times per year, preventing me from fixing security or functional defects in my own software. I solve these by forking the offending gem and fixing the constraint.

Once in a very great while I find that there was a genuine reason for the author to be so pedantic. Generally, however, they were just ignorant about the pessimistic versioning notation. I fix the constraints, open a futile pull request in case the author ever wakes up again, and add a tech-debt item to stop depending on the offending gem.

Needless to say, I would find this feature very helpful. That having been said, there's almost always a workaround -- forking the gem and depending on it as gitted, or pulling from a custom source -- so I understand the core team's unwillingness to spend precious cycles on the feature.

Nevertheless, I'd love to see a feature like this emerge! It would save me significant time in the long run. So, err, +1, and perhaps I'll spend some time implementing it, if I become seriously blocked.

[coilysiren]

From @todd-richmond on November 13, 2015 17:28

very well put - thx! That is the exact issue we had... small, but highly useful gems that have been abandoned by their author with restrictions they didn't fully understand before going dark. In this case forking is a pain because we have to manage them and if the author ever does make changes, we probably won't notice

This is somewhat similar to maven's ability to exclude package dependencies so that you can enable the use of newer packages. The developer is owning the conflicts at that point so a warning would be fine, but far better than having to fork and hack

[thewoolleyman]

@lynnco Thanks for cleaning up. Are there plans to implement a feature to address this, or are you just consolidating the tickets?

[segiddins]

Just consolidating at the moment

[ioquatix]

I'd like to add to this proposal.

In some cases, you can write like this:

gem "foo", git: "path/to/fork"

In some cases it would be useful to actually have a fork released as a gem, but replace all instances of it in the Gemfile and related dependencies.

gem "foo", provided_by: "foo-fork"

Where foo-fork is a gem which provides the same API (e.g. forked) as foo.

[arcreative]

As :force_version => true is clearly an opt-in change and extremely visible in a project's Gemfile, I'm finding the suggestion that one should fork every conflicted gem to update its version constraints a bit ridiculous.

This request has very real use cases and is supported in most (all?) other major package managers. The suggestion that every gem maintainer should change to minor versioning isn't a real solution, as there's a large coincidence of needing version overrides as a result of gems being abandoned.

Also worth mentioning, I think you would agree that this option never belongs in a gemspec, just a local project's gemfile. This would allow for both the most safety and flexibility, in my opinion.

What's the timeframe/workload on this if it's agreed upon?

[ioquatix]

@arcreative I think in order for bundler to make this improvement, it needs to be implemented in RubyGems first, per the linked issue.

[ms-ati]

11 months later, what is the status of this feature request?

[segiddins]

@ms-ati we have not even decided it is a feature we would like to implement (it needs a complete RFC first). Though I did toy around with the idea in #5670 just for fun

[arcreative]

@segiddins Your PR looks simple enough, do you have any specific concerns about it?

[segiddins]

The concern is that this is a feature with the potential to introduce a lot of edge cases into bundler, and as such it needs to be well-planned out and considered before merging -- for a change like this, just a patch isn't enough. Design is needed.

[arcreative]

I feel like it should be allowed for gemfiles, but not gemspecs. That would mostly limit the implications of forcing versions to the end user when they opt into this behavior...right? I agree it would be unpredicatable to have forced versions on dependencies of your project when it has other dependencies that may use the same gem, so maybe limiting the scope to gemfiles would be a happy medium?

[thewoolleyman]

Yes, I agree it should only be for gemfiles and not gemspecs.

@segiddins, would there still be many edge cases in this scenario?

If so, what are they? I can't think of many offhand. It's a risk, but one solely incurred by the Gemfile's app itself, never imposed upon any dependent users of a published gem's gemspecs.

[ms-ati]

@thewoolleyman I think you meant @segiddins (misspelled above)

[hswick]

This feature would be nice :D

[rbq]

Maybe instead of forcing versions in the Gemfile and considering this as a special case while resolving the dependencies we could find a way to overwrite the dependency information of a Gem and not touch the whole resolution process at all? Basically a way to patch a gemspec locally within a Gemfile? E.g., if you want to make Bundler act as if your old Rails version was compatible with a newer MySQL Gem:

gem 'rails', '3.2.5' do |spec|
  spec.override_runtime_dependency 'mysql', '< 5.0.0'
end

(Disclaimer: Just an idea, didn't really think this through.)

[onomojo]

Is something going to happen with this or not? This issue was originated way back in 2013 here and still being requested well into 2018. #2412 Five years and you haven't made a decision on if we should add this or not?

[onomojo]

Sorry my mistake, this was requested in 2011. So 7 years now: #1549

[colby-swandale]

@onomojo We have clearly said that this is a feature that needs to go through an RFC process with a well thought out design in mind #4552 (comment). If you would like to see this feature being worked on - then you're welcome to help write an RFC.

[jrochkind]

I think it's actually a terrible idea... and ironically I just ran into the need for it today.

I want to have travis testing my gem under rails master-from-github (as an allowed travis failure), so I can find out when a change in not-yet-released rails breaks my gem. But I can't, because master-from-github is already marked 6.0.0.alpha, and I have a dependency that insists on rails < 5.3.0.

I would never ever use this feature in an actual app, I think it's a terrible idea. It's gonna lead to dependency nightmares of confusion over what is actually compatible with what. I'd rather fork the dependency that's limiting me if I need to. Thus my initial reaction was this was a bad idea. In pretty much any production app Gemfile.

But then... it seems totally reasonable to want to test my gem (not app) on travis against master as an allowed failure!

[onomojo]

You're either in a fork nightmare having to deal with a handful of forks for slowly updated gems or you have to manage those dependency overrides yourself. Either way you're still ultimately responsible for manage it in the end. You're just creating more work for yourself when the result is essentially the same. If you fork, you also end up wondering if you're going to lose upstream changes once the gem is finally updated. Do you fork again on the latest version if you still have to? The only solution right now is to fork but then you have to maintain a forked repos and keep it updated. An override would get you the same results without having to create more headache with managing forks. Here's how I'd envision it working (without jumping through the flaming hoops to create an official RFC of course)

# Gemfile

gem 'blah', '~> 4.2.0' do |g|
  g.dependency_override('slowpoke', '>= 4.2', '<= 5.2')
  g.dependency_override('sloth '>= 1.2', '<= 2.2')
end

Poof. With that I could effectively do what you'd end up having to do by forking the original gem repo just to update some bad dependency versions.

[indirect]

Thanks for the input, everyone.

Since this ticket isn't producing any new points for discussion, I'm going to lock it to collaborators until we have an RFC that discusses the tradeoffs involved and comes to conclusions about those tradeoffs that we can use as a basis for implementing something like this (or not, as the discussion goes).