Proposal to add :override_version_dependency_with => "lib >= 1.5.0"

[naleksi]

Hello!

I recently faced a situation, which I believe is somewhat common, where one lib required nokogiri 1.4.0 and another nokogiri 1.5.0. Bundler barfed.

The rescue was to:

1. fork the other library source,
2. bump up the version dependency into nokogiri 1.5.0 for it,
3. run the test suite to check all is ok
4. push the change back
5. change my original Gemfile into gem "nokogiri", :git => "git@foo.com:myfork_of_lib_updated_to_nokogiri_1_5_0.git"

Then it worked. I also tried first to use local version of the gem through :path, but naturally the deployment to Heroku failed as it was not copying the sources.

Now, this is too much work for such a simple thing. Thus I'm proposing alternative rescue:

1. change original Gemfile into

gem "library_with_old_nokogiri_dep_to_1_4_0", 
    :override_version_dependency_with =>    ">= 'nokogiri' >= '1.5.0'"

Yes, the new option should be named sensibly. Yes, the part with => and >= after each other looks horrible. Likewise for the contents of the option. Maybe the contents could be an array of ['lib', '>=', '1.5.0'] instead, requiring no further parsing. And naturally this one step does not take into

Well, I guess the idea is clear. I posted this after it was suggested at freenode#ruby. Any opinions?

[yfeldblum]

Ultimately, Bundler should provide a simple way to override a gem's stated dependencies in the Gemfile. Bundler should take the override into account when resolving and writing the lockfile as well as when activating gems (possibly by hooking rubygems gem activation).

gem "something-old", :overrides => { "nogokiri" => "~> 1.4" }

[wuputah]

👎

If you want to modify the gem (or its dependencies), fork it or vendor it.

[yfeldblum]

That would require maintaining a complete fork, including polling for updates to upstream and pulling them into the fork.

When all that one really wants is just a one-line annotation.

[wuputah]

If the upstream author is maintaining his version, he'll probably accept a pull request to make his gem compatible with more than one version of nokogiri.

[yfeldblum]

Suppose any of:

• the upstream isn't being maintained
• the upstream maintainer is away
• legal needs to approve any pull request, and legal is vacation

There are many reasons why fixing upstream may not be possible, or may not be possible immediately.

One of the serious benefits of ruby is monkey-patching. Many Ruby/Rails apps have their own lib/patch directories with a nice handful of patches on various other gems. Some of these patches fix bugs, some override behavior that the app author disagrees with, some are changes that upstream has rejected and some are changes that upstream may merge and release in the future, etc. Any number of scenarios, and all great reasons to patch that gem. None of these warrant a fork.

This is another type of monkey-patch (essentially, patching the gemspec), and may be equally necessary in one app or another.

In this hypothetical scenario, the best approach would be to patch something-old's gemspec dependencies now, issue a pull request (and defend it over the course of two weeks), wait another three to five weeks for a release, and finally upgrade and remove the patch.

[indirect]

I'm sorry, but this isn't something that we're interested in adding, at least not right now. The entire reason for Bundler's existence is to make sure that you can't use gems together if they conflict. We provide :git so that you can edit a gem if you need to. The sort of half-editing described here would be a massive amount of work that would just allow you, in the end, to use conflicting gems together as if Bundler didn't exist. Hopefully that makes sense. Thanks for the feedback!

[autotelik]

A big shame. Good proposal. Sometimes you just need/want to crack on with stuff. Sometimes you are just trying things out, so don't need a huge dependancy checking monster stopping you at every turn. This kind of over ride would be a god send and save loads of faffing about.

[indirect]

It's completely trivial to load whatever you want at any time: add it to the load path and call require. It's not useful to duplicate that built-in ruby functionality in Bundler.

On Feb 1, 2012, at 9:55 AM, tom statterreply@reply.github.com wrote:

A big shame. Good proposal. Sometimes you just need/want to crack on with stuff. Sometimes you are just trying things out, so don't need a huge dependancy checking monster stopping you at every turn. This kind of over ride would be a god send and save loads of faffing about.

---

Reply to this email directly or view it on GitHub:
#1549 (comment)

[autotelik]

Really ? Nothing in your response to original idea suggested that. Given the scenario above how could this load approach work ?.

I have faced similar issue as first described, my Rails app includes the Spree gem which has it's own Gemfile.

It proposes paperclip version 2.5.0. This will simply not install on Windows as the paperclip guys have added some fixtures with weird chartcters in the names thus causing the whole gem install to fail.

I need to specify paperclip 2.5.2 as I know that release fixes the Windows install issue, I'm happy
to take responsibility for potential consequences of using a different version than recommended by Spree

bundle install says No.

In Gemfile:
spree (>= 0) java depends on
paperclip (= 2.5.0) java

paperclip (2.5.2)

I'm now stuck, not really sure what bundle has installed or what state my app is in if I try to run it.
So how do I trivially use load to over come this situation?

I apologise if I've missed in your docs some simple to get round this, but as far as I can see your recommended
approach is to go find the offending gem on gitbub, in this case spree, a huge complexe engine.

clone it myself

fix the Spree Gemfile to use paperclip

issue pull request

while we wait, change my Gemfile to use the Spree gem as a path

yfeldblum clearly highlights issues with this approach and given that I may just want to have a quick play with Spree, I find that quite startling compared to the proposal

[naleksi]

Thanks for Indirect to patiently answering, and to Autotelik to keep discussion going. One more thing I'd like to add here are installations to the locations where you cannot "override" Bundler and nor how it's being run. Like installation to Heroku. There are ways around it, so for example instead of being forced to use :git => "git://mygitrepo.com/overridden_dependency.git" one could go around with Private gem (like explained in http://underpantsgnome.com/2011/01/05/how-to-install-private-gems-on-heroku).

Maybe Indirect meant so other way to mess up with LOAD_PATH and install stuff into Heroku. Nevertheless it all seems somewhat contrived compared to the intention: "I know this is a version mismatch but I'm willing to take risk, so please just override and install".

BTW. I'm happy with whatever workaround that just can be executed in minutes instead of tens of minutes, as this seem to be somewhat recurring problem. I believe you can multiply that frequency with the headcount of Ruby dev population to get some estimate of "lost productivity".

[indirect]

I appreciate the discussion. Loading a specific gem in ruby is pretty straightforward:

$LOAD_PATH.unshift("/path/to/paperclip-2.5.3/lib")
require 'paperclip'

That's it. Ruby require management since time immemorial. Well, I'm pretty sure Matz remembers before that, but whatever. :)

As for Spree "recommending" a version... that's not what they're doing. They're demanding, with absolute certainty, a specific version. It's not a recommendation, it's a declaration that the gem they released won't work with any other versions. Given that Bundler already includes the ability to fork gems and use said forks directly out of the git repo, it doesn't seem worth it to add a particular special case to the Gemfile for version compatibility. Pretty much what I said when I closed this ticket before, I guess.

[autotelik]

ok, yes seen requier before, unless I'm missing something, not sure how this answers the issue in hand though, which is bundle install failing to install all the gems I need when one gem fails.

The issue is that Spree may be 'demanding with certianty' but we all make mistakes. I'd be happy to use that version of paperclip but attempting to install that version crashes gem install and then by association bundle install on Windows.

Bundler may insist that I can't use gems together if they conflict, but in this situation it is enforcing a broken system, indeed one that causes itself to fail

Ok, the fork options works. It just seemslong winded and inelegant when working in try and see mode when the comfort and security of bundler is not necesarily required.

anyway, thats just my 2ps worth, thanks for taking time to respond.

[myabc]

Too bad this got voted down/closed, as this is a very common situation with Rails development where library authors are often too cautious in specifying their dependencies.

Imagine you want to use edge Rails (4.0) along with Sass or Haml (see this tweet). Although there may have been no significant changes to ActionView or asset pipeline APIs, because of strict gemspec dependencies your only option to make the stack work is to fork the dependencies and modify the gemspecs.

Being able to do the following strikes me as a much cleaner approach:

gem 'sass-rails' do
  override 'railties', '4.0.0'
end

Clearly this is an "at your own risk" approach – but then so is forking a gem's source.

[ralph]

+1 from me.

While I understand that bundler was not build for this reason, it's currently hard to just try something with a newer version of gem xyz. For example, I just tried switching an app to the Rails 4 beta, just to check how hard the update would be at the current point of time (see what works, what does not work etc.). So yes, I had to go through this fork/update gemspec/bundle from your own fork loop several times. Overriding the gemspec's dependencies if you need to would be a much more convenient way to do so. Plus, deploying your experiment to Heroku would be easier, because you would avoid building lots of private gems which don't have anything changed but the gemspec. This gives you more freedom to experiment.

[indirect]

This request is fundamentally requesting the ability to monkeypatch a gem's code (specifically the gemspec) inside your Gemfile. The entire reason we added the :git option to Bundler was to allow a way to fix gems without monkeypatching. It is technically possible to monkeypatch a gemspec using plain ruby code in your Gemfile, just like you can anywhere else that ruby code is run. However, Bundler is never going to make it easier to monkeypatch, because that is completely opposed to the reason Bunlder was built in the first place. Hopefully that explanation helps.

On Feb 12, 2012, at 4:22 AM, Alex Colesreply@reply.github.com wrote:

Too bad this got voted down/closed, as this is a very common situation with Rails development where library authors are often too cautious in specifying their dependencies.

Imagine you want to use edge Rails (4.0) along with Sass or Haml (see this tweet). Although there may have been no significant changes to ActionView or asset pipeline APIs, because of strict gemspec dependencies your only option to make the stack work is to fork the dependencies and modify the gemspecs.

Being able to do the following strikes me as a much cleaner approach:

gem 'sass-rails' do
 override 'railties', '4.0.0'
end

Clearly this is an "at your own risk" approach – but then so is forking a gem's source.

---

Reply to this email directly or view it on GitHub:
#1549 (comment)

[myabc]

@indirect I guess we disagree here. I do not consider a .gemspec to be application code. Instead I think it should be thought of as metadata. It is a description of the officially supported dependencies (supported by the author), but contains no business logic. It serves no purpose other than to allow easy installation of dependencies through a package system, but could (theoretically) be blown away and replaced by some other solution.

As metadata, it should be overrideable (albeit at a user's own risk). Overriding a .gemspec is not akin to monkey-patching. It is simply giving the power to a gem user (rather than the original gem author) to make a determination on API compatibility.

[peterpunk]

I think that the monkey patching of gemspec happens either the fork or if Bundler does it, if Bundler does it will save a lot of time to the developers, just say "at your risk" when bundling ;-). I hope this change should be considered.

[naleksi]

indirect, I appreciate your reasons. How about reaching a compromise? Could you give a sample how the "technical monkeypatching of a gemspec in plain ruby in Gemfile" could be done?

Perhaps the monkeypatch could be both:

• easy enough for people to use instead of unnecessary forking, and
• yet ugly enough it shouts out "Beware of me!"

[indirect]

@myabc, the gemspec is not application code, it is metadata. However, it is metadata that happens to be a) written in executable ruby code, and b) stored in the gemspec along with the rest of the lib's source code. It doesn't matter, from my perspective, whether the underlying reason you want to monkeypatch some ruby code is business logic or metadata. Either way, Bundler was created (in part, at least) to replace monkeypatches with forking and git repos. The fork-and-git-gem workflow includes fixing bugs in gem metadata just as much as it includes fixing bugs in executable gem code.

@peterpunk, Bundler doesn't exist to save time (insert joke about "Fetching remote index..." here, lol). The reason that monkeypatching is so dangerous is that it saves time now, but creates problems later. This is why the entire ruby community unanimously moved away from monkeypatching bugfixes to using the fork-and-git-gem approach for bugfixes.

@naleksi, I don't have time to come up with a working monkeypatch approach, sorry. But even if I did, I would tell people to fork the repo and use a fixed version from git instead. Monkeypatches for bug fixes are a bad idea, even in gem metadata.

In summary, bugs in gem metadata are just that, bugs in the gem. Even though it would technically be possible to allow buggy gems to be fixed ad-hoc in each individual user's gemfile, it's a bad idea. It's a bad idea for the same reasons that it's a bad idea to fix other bugs in a gem by monkeypatching that gem: maintainability, understandability, future compatibility and a host of other concerns. In short, it's a bad practice, and Bundler is not going to encourage it. Especially since Bundler invented the idea of a git gem specifically to address fixes for gems that have bugs (including metadata bugs!)

[autotelik]

Bundler doesn't exist to save time.

So true, so true, from this little exchange I'd say it exists to drive Windows Ruby developers into the arms of Python right.

[software-project]

Override its sth I definetelly want to have.

[myabc]

I ended up blogging on this issue (and some other pain points with Bundler). Feedback welcome.

I'd definitely love to work on a patch for this, but I guess there'd have to be at least some chance it would be accepted to make it worth spending time on.

[mpapis]

👍 for the proposal, @myabc you can always gem install maybc-bundler let me know when you have it so I can fix it for myself too.

[dball]

By @indirect's logic, bundler itself seems unnecessary. Resolving the dependency graph for all your gems and their dependencies is something you can already do manually, just as you can fork a gem, edit its metadata, and publish it manually. As a practical matter, doing it manually sucks for all the reasons outlined in this thread.

I've been working on upgrading a big rails-2.3 project to rails-3.2, and I could have used this feature at least twice.

[taylor]

+1 to support overriding dependencies

bundler is very similar to package managers like rpm and dpkg. Most package managers allow you to override dependencies, but when you do use those capabilities you are very aware that you are doing a dangerous operation.

It's good that you are required to jump through some hoops to specify the overrides for the package managers. It's good that you are warned with messages and the names of the options to do the overrides.

In the end though they let you do these overrides because sometimes it is desired, necessary, etc...

BTW, non-git/mecurial repos means you can't do the fork and point to the repo work-around.

[rickbyington]

+1

[JustinAiken]

+1

:override would be a lot easier than forking a gem when you want to test it quickly.

How often have you seen a situation where the dev of a gem responds to a bug report with "can you try it with version x of somegem and post logs?"

[bithive]

So when I tried to update to Rack 1.5.2 today to take advantage of important security fixes, Bundler complains:

Bundler could not find compatible versions for gem "rack":
In Gemfile:
rails (= 3.2.11) ruby depends on
rack (~> 1.4.0) ruby

rack (1.5.2)

I don't think it's helpful for Bundler to be so opinionated that I flat-out cannot move past this issue without forking Rails. In situations like this the balance between time-saver and time-waster can shift pretty quickly; I urge the Bundler developers to reconsider this.

[indirect]

The Rack security fixes are part of Rack 1.4.5 as well. Add this line to your Gemfile: gem 'rack', '~> 1.4.5'.

[bithive]

Thanks, I missed that. My desire to put a hack in the Gemfile was premature in this case but I still think an override feature would be useful. It's not about loading arbitrary gems so much as being able to force Bundler to be flexible. I appreciate the care that's been taken in Bundler's design but it can be oddly rigid for a developer tool.

[hoffmanc]

👍

[robotmay]

I'm also going to chip in on this ancient issue. In light of all the recent security vulnerabilities coming to light in the Ruby community, this is a feature which doesn't seem to have been given the proper consideration. My GitHub account is pretty much just full of gems I've forked just to update the 'rake' requirement, or something equally inane. Today I'm going to have fork the entirety of Padrino v0.10.7 just to update the 'mail' gem for a security vulnerability. I know the new one doesn't conflict, why can't I just specify it as an override in the Gemfile and be done with it until they get around to releasing an update with the new version?

This is a feature supported by most package managers which I know of. You might not agree with it but it's a feature quite a lot of people want.

[indirect]

@robotmay if you'd like to start a new discussion, I'd suggest opening a new ticket with your proposed interface and arguments. Thanks!

[robotmay]

@indirect Ah ha, good point; I'll write it up a bit better and submit it as another issue.

[ur5us]

👍

[lastobelus]

@indirect A Gemfile littered with forks whose only purpose was to resolve gem conflicts is also dangerous. Doing a bundle update doesn't get the latest fixes -- unless I go through everything forked and merge with the upstream. If your long term work is on a single slowly evolving app, this may not seem onerous to you, but for contractors who work on many apps, often not touching them for months in between maintenance intervals, it is a huge pain point.

[indirect]

This is a closed issue. Discussion is over at bundler-features#20, where I have commented, and where it's possible that this will become a feature in the future.