Provide an option to force a gem version, even it it conflicts with a gem's declared compatibility.

[xaviershay]

From rubygems/bundler#2412

Still debate about how this would be done or whether it's a good idea. See linked ticket for context.

[xaviershay]

Also this rubygems/bundler#1549

[rmetzler]

Today I started to look into this.

I read about all proposals for the API. The DSL style I liked best and is the easiest to implement is the following notation:

gem 'mail', '~> 2.4.4', :override => true

This notation means, that the resolver uses the version marked by the override option whenever two dependencies are compared.

Requirements

• Bundler should warn about this, because it is a potentially dangerous option.
• If Bundler compares two conflicting dependencies both having :override => true then Bundler should throw a special VersionConflict error.

Questions

As I'm new to the Bundler source, trying to find my way around I recognized some minor issues. Also I'm not sure whether I should it ask here or on the mailing list. I'll just write it down here because it's easier now. Please advise!

1. I didn't found a test case for Bundler::VersionConflict so I added one. It's modeled after a detailed description of the problem, but could be simpler though.
2. I found this line that doesn't look right, especially when you consider this constructor. Not really sure about that, had a few errors when I changed it and run tests.

[indirect]

I think that :force_version => true might make it clearer what the option does, since there are other options that could be passed as well, making :override a bit ambiguous (override what? and with what?).

I'm happy to merge a testcase for VersionConflict. 👍 Unit tests in general are a weak area for the Bundler codebase. 😬

The line you point to doesn't look right to me either, but maybe there's another Bundler::Dependency hiding somewhere?

[rmetzler]

Thank you for you reply, very much appreciated. I also think :force_version => true is a clearer option.

I'll send a pull request for VersionConflict and investigate further in dep helper method.

[sheerun]

I don't think it's good idea, forking is better solution.

[rmetzler]

Hi,

first thing I've to say, I'm sorry I didn't send any pull request for the test cases yet.

@sheerun for discussing the proposed experimental feature in question please read over the discussion first here rubygems/bundler#1549 and and rubygems/bundler#2412
I think this is the best description for the current state of how they to fix gem dependency conflicts

I just want to fix the problem I encountered myself more than once. This feature would be experimental and so far I've not written anything else than tests, because the bundler code in question is fairly untested yet. Only "happy paths" were covered with tests in master branch and not conflicting dependency versions.

I also thought about writing a CLI tool (different from bundler) that would know about git, Github and Gemfiles and could do something like the linked comment suggested automatically incl running tests and send pull requests. If Ruby devs think this might be a better idea, please let me know.

Recently I also worked with VersionEye on their open source CLI veye and I realized that there are many issues around dependency versions. This is the case especially in Ruby and nodejs communities where the lib ecosystem is growing rapidly. But every package manager has this problem and if there are good solutions out there, please let me know and I'll investigate the code.

Richard

[aantix]

👍 Running into this very issue today. I know the versions are compatible. But one author has been too strict with his requirements..

[korbin]

+1 for this feature.

Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override.

[indirect]

Again, the issue is not that we actively want to make it harder. The issue is that allowing this means there is no longer a canonical source of truth about what a gem’s dependencies are. When these kinds of changes cause resolver errors (and they will), it will be extremely hard to diagnose. I’m fine with “at your own risk”, I’m not fine with “makes people complain that Bundler is broken when it’s actually something they did themselves”.

If you’re competent enough to debug forcibly overridden dependencies yourself, you’re competent enough to either hack this feature into Bundler or fork and update the dependencies yourself. This feature hands a giant foot-gun to those who won’t know how to debug the issues that come up while using it.

On Mar 11, 2014, at 1:12 PM, korbin notifications@github.com wrote:

+1 for this feature.

Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily, tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override.

—
Reply to this email directly or view it on GitHub.

[TimMoore]

And think of the poor gem authors that will suddenly have a pile of mysterious bug reports caused by someone using a gem that was listed as explicitly incompatible!

[strzibny]

I would also welcome this feature as this overly strict requirements are happening quite often. Providing an override option seems reasonable. Of course it's always on the risk of the user.

[anri-asaturov]

This feature becomes more and more important with number of gems growing. .NET has it. Java has it. Nothing bad happened to them because of it.

[indirect]

I’m slowly warming up to this, as long as the override is set explicitly in the gemfile on a line for the gem being overridden, and the overridden version is called out during bundle install. Something like:

# Gemfile
gem "puma", "5.1" # depends on rack <= 2.0
gem "rack", "3.0", force_version: true

$ bundle install
[…]
Installing rack 3.0 (forced version, conflicts with puma)…
Installing puma 5.1…
[...]

[thewoolleyman]

+1.

As mentioned on the closed bug thread, one major exacerbating factor of this problem is that pessimistic constraints unnecessarily specify all three components of a semantic version, when they should only need to specify two to prevent API-incompatible changes in dependencies.

Rubygems.org unfortunately encourages this practice by always suggesting a pessimistic constraint with all three versions, which leads gem authors to do the same practice in their gemspecs.

[TimMoore]

Maybe Rubygems.org should be changed to recommend setting only major and minor versions?

[endymion]

I respectfully add my vote to this feature request. Bundler's lack of a feature for forcing a version that's known to work has cost me hours of extra work today. It's an even bigger issue when you're dealing with older Ruby code that uses gems where the authors have locked the dependencies to something like "~> 1.2.5" and then stopped maintaining a gem. Sometimes a gem doesn't really need to be updated because it works, but it's still full of over-specific dependencies like that. The only way to deal with it right now is to fork the gem itself and update the dependencies. Bundler could easily have made my life a lot simpler today by giving me more flexibility and control and letting me take responsibility for my own QA.

[indirect]

@endymion Thanks for the feedback. Today, no one on the Bundler core team has the time to do the work that this feature would involve. If someone joins the team who wants to implement and support the feature, or Ruby Together is willing to fund work on this feature, we'll take another look.

[thewoolleyman]

@indirect I do agree with your previous comments that this could make the situation worse for Gem maintainers, because they don't know if a bug report is due to an incompatible overridden dependency version being used.

I still think the long-term solution is to encourage/force people to use pessimistic constraints correct. E.g., docs on rubygems.org, and rubygems itself should complain loudly if you try to put a dependency version in a gemspec with a three-component pessimistic constraint, because in all likelihood You're Doing It Wrong.

Maybe someone (with more time and who cares more than me) can open that discussion on the RubyGems mailing list along these lines (and cross-link it here in a comment), and see what comes of that...

-- Chad

[indirect]

The RubyGems.org pages have all changed to default to ~> X.Y version constraints available for copying. I think it might be worth it to warn if a gemspec has a pessimistic constraint on a bugfix release, but unfortunately there are many, many, many, gems that only function correctly at a precise bugfix release. :( It's a chicken and egg problem that I'm not sure how to progress on.

[jfinkhaeuser]

If a gem has a constraint on a specific bugfix release because that's the only version it works with, that's a bug in the dependency.

Or, to go by example:

• Gem A exists in version 0.1.1
• Gem B requires precisely version 0.1.1, because it doesn't work with version 0.1.0

Then Gem A uses semantic versioning wrong. That cannot be emphasised enough.

The point of semantic versioning is to provide a stable interface across all releases with the same major and minor component. A bugfix that breaks the interface is simply no longer compatible with the same major and minor component.

Penalizing Gem B for doing the only thing left for it to do seems like entirely the wrong approach.

Gem B should specify dependencies as ~> 0.1 >=0.1.1 in this case. Being helpful about this would IMHO be a good feature.

But really, it's the Gem A maintainers that need to be given a hard time, not anyone else.

(Mind you, there is no easy solution to this. The best solution so far is to differentiate between interface and package versions, and depend on an interface version. That's happening to a degree in Linux/Solaris shared object libraries, and vastly improves some issues, but it's not bullet proof either.)

[booleanbetrayal]

would definitely find this feature useful!

[xeger]

I deal with a lot of abandonware gems whose authors never learned to manage dependencies correctly. I get bit by falsely-rigid constraints several times per year, preventing me from fixing security or functional defects in my own software. I solve these by forking the offending gem and fixing the constraint.

Once in a very great while I find that there was a genuine reason for the author to be so pedantic. Generally, however, they were just ignorant about the pessimistic versioning notation. I fix the constraints, open a futile pull request in case the author ever wakes up again, and add a tech-debt item to stop depending on the offending gem.

Needless to say, I would find this feature very helpful. That having been said, there's almost always a workaround -- forking the gem and depending on it as gitted, or pulling from a custom source -- so I understand the core team's unwillingness to spend precious cycles on the feature.

Nevertheless, I'd love to see a feature like this emerge! It would save me significant time in the long run. So, err, +1, and perhaps I'll spend some time implementing it, if I become seriously blocked.

[todd-richmond]

very well put - thx! That is the exact issue we had... small, but highly useful gems that have been abandoned by their author with restrictions they didn't fully understand before going dark. In this case forking is a pain because we have to manage them and if the author ever does make changes, we probably won't notice

This is somewhat similar to maven's ability to exclude package dependencies so that you can enable the use of newer packages. The developer is owning the conflicts at that point so a warning would be fine, but far better than having to fork and hack

[vbabu75]

+1

[jirihradil]

+1

[teadur]

+1

[maximgurin]

+1

[yehormasalitin]

+1

[anton-fedorov]

+1

[oleksandr-oksenenko-zz]

+1

[yuliiakorzhuk]

+1

[batytskyy]

+1

[asukhorukova]

+1

[andrii-arnautov]

+1

[a-paschenko]

+1

[spopkov]

+1

[eugeneugrevatov]

+1

[segiddins]

This deluge of :+1:s is unhelpful -- please add a 👍 reaction to the original post.

[yuri-taran]

+1

[vstepchik]

+1

[coilysiren]

Moved to rubygems/bundler#4552