gemspec runtime dependencies not included in DEPENDENCIES section of Gemfile.lock #1041

Closed
myronmarston opened this Issue Feb 14, 2011 · 37 comments

Projects

None yet
@myronmarston

Here's a simple example project that demonstrates the bug.

This commit demonstrates the bug. Note that the runtime dep is not included in the DEPENDENCIES section of Gemfile.lock, but the development dep is.

The next commit demonstrates a workaround, but it would be nice to not have to do this.

The README lists my environment info.

@jfirebaugh

This functionality (#gemspec directive pulling in both development and runtime dependencies) used to exist -- in fact it is documented to work:

The gemspec method adds any runtime dependencies as gem requirements in the default group.

I know this because I wrote that documentation back when it did work.

However, @wycats removed it between 1.0.7 and 1.0.8 (a breaking change) in 0337810. The commit message was "Make it possible to override a .gemspec dependency's source in the Gemfile".

I too would like to see this behavior restored.

@ndbroadbent

On Bundler's rubygems support page, it says:

Runtime dependencies in your gemspec are treated like base dependencies.

But it looks like runtime dependencies aren't even being touched by Bundler at all when calling #gemspec.

I've solved my problem by hacking up my Gemfile with the deleted code, and overriding the gem method so that I can re-define the sources for gems in the gemspec.

Here's what my Gemfile looks like now:

# Removes a gem dependency
def remove(name)
  @dependencies.reject! {|d| d.name == name }
end

# Replaces an existing gem dependency (e.g. from gemspec) with an alternate source.
def gem(*args)
  remove(args.first)
  super(*args)
end

# Load development dependencies from gemspec
gemspec

# Bundler no longer treats runtime dependencies as base dependencies.
# The following code restores this behaviour.
# (See https://github.com/carlhuda/bundler/issues/1041)
spec = Bundler.load_gemspec(Dir["./{,*}.gemspec"].first)
spec.runtime_dependencies.each do |dep|
  gem dep.name, *(dep.requirement.as_list)
end

gem 'ransack',  :git => "git://github.com/ndbroadbent/ransack.git"

It's not the best, so it would be nice if Bundler supported this.

@lgierth

Snip. Didn't see @ndbroadbent's edit and @jfirebaugh's comment.

@lgierth

#1245 is also related to this, it overwrites existing Gemfile dependencies with others.

@keppy

Not sure if this is related to the hack yet, but I am experiencing an error that seems fishy when attempting to deploy the fat_free_crm app on engine yard. Note that I have definitely executed 'bundle install' and pushed my repo to github before attempting to deploy. I thought it odd that the six gems which are being added and then deleted are exclusive to the development_dependency portion of fat_free_crm.gemspec:

:: running ssh -i /home/deploy/.ssh/internal -o StrictHostKeyChecking=no -o PasswordAuthentication=no deploy@SERVERADDRESS.compute-1.amazonaws.com 'sh -l -c '\''export GIT_SSH="/data/fat_free_crm/shared/config/fat_free_crm-ssh-wrapper" && export LANG="en_US.UTF-8" && unset RUBYOPT BUNDLE_PATH BUNDLE_FROZEN BUNDLE_WITHOUT BUNDLE_BIN BUNDLE_GEMFILE && cd /data/fat_free_crm/releases/20120322210609 && ruby -S bundle 1.0.21 install --deployment --gemfile /data/fat_free_crm/releases/20120322210609/Gemfile --path /data/fat_free_crm/shared/bundled_gems --binstubs /data/fat_free_crm/releases/20120322210609/ey_bundler_binstubs --without test development'\'
You are trying to install in deployment mode after changing
your Gemfile. Run bundle install elsewhere and add the
updated Gemfile.lock to version control.

You have added to the Gemfile:

  • capybara
  • spork
  • database_cleaner
  • fuubar
  • factory_girl (~> 2.6.1)
  • factory_girl_rails (~> 1.7.0)

You have deleted from the Gemfile:

  • capybara
  • database_cleaner
  • factory_girl (~> 2.6.1)
  • factory_girl_rails (~> 1.7.0)
  • fuubar
  • spork
@ndbroadbent

@keppy, I can't seem to reproduce this problem, but I've removed the development dependencies and put them in the Gemfile instead. I would be grateful if you could please update and try again

@keppy

@ndbroadbent,
First I want to say THANK YOU for doing that little fix. I should have been able to see what was going on, and now I see that it was so simple. I'm new at this and I had a major "ah-ha" moment when i looked at your changes.
So yeah, no problems at all now, my instance is running, and i actually understand a lot more about the way gem dependencies work. 1+ karma to you sir.

@christophermanning

This happens to me in Bundler 1.1.4 as well:

require "bundler"
Bundler.require(:default, :development)

doesn't require gems added with add_runtime_dependency in the gemspec

@DCarper

I'm experiencing the same thing, though with a twist.

My gem has two runtime dependencies, rspec and nokogiri. Bundler picks up rspec just fine from the .gemspec, but only requires nokogiri when included in the gem file.

@indirect
Bundler member
@DCarper

@indirect hmm could you by chance provide a reference for that? gemspec - dependencies has nothing about needing to explicitly require dependencies that I've seen. I could totally be missing something though.

Isn't that what this and at least one other duplicate issue is about anyway? Bundler not requiring dependencies from the .gemspec?

@indirect
Bundler member

@DCarper, this issue is about whether Bundler adds gems to the load path. It has nothing to do with requiring said gems. The link you provided explicitly states "gems that must be installed for this gem to work". That means the gem must do its own requiring, just like it needs to when the gems are provided via Bundler.

@indirect indirect closed this Aug 7, 2012
@indirect indirect reopened this Aug 7, 2012
@indirect
Bundler member

(Sorry, accidentally closed -- this issue is about whether dependencies added to the Gemfile via gemspec are added to the Gemfile.lock, which I have not yet investigated. Either way, dependencies added via gemspec are in fact added to the $LOAD_PATH by Bundler, and can be required by gems as appropriate.)

@DCarper

@indirect okay I just want to read and understand the documentation for myself, can you please link to where the gemspec says all dependencies must be required by hand? I can't seem to find it

Also, given that's the standard this is not a duplicate issue

@indirect
Bundler member

@DCarper Okay, let me recap for you. The Bundler docs for using Gemfiles with gems tell you that Bundler will helpfully install your dependencies, but "otherwise, use gemspecs normally". The Rubygems docs, which you already linked, tell you that listing deps in your gemspec will not install or require said deps. Therefore, by process of elimination, your gem must require it's own dependencies.

You are always free to use Bundler.require anywhere that you would like to, but any code that calls that method can no longer be shipped as a .gem package that is installed with the gem install command.

@DCarper

@indirect I'm with you and after thinking about it I really do understand why its bad practice to use Bundler to require dependencies in a gem. I also don't mean to keep polluting this thread so I'll leave it be after this but..

Based on the Rubygems docs I don't draw the same conclusion as you do. "listing deps in your gemspec will not install or require said deps" to me only implies that it's on ME to get them required, nothing to do with manual or automated. Having really only worked on apps and not gems as much I'm very conditioned to using Bundler to require dependencies, so without putting a whole lot of thought into it, "of course I'll use Bundler to require some dependencies".

If I made that assumption (even if it's wrong) then others are making that assumption as well. My last note is that an explicit statement of "Do not use bundler or any other tool to require dependencies, you must manually require them" could go a long way both in the Rubygems docs as well as the Bundler documentation, so that others don't make the same mistake that I did.

Thanks for the point in the right direction and like I said I'll get out of this thread now ;)

@cnstaging cnstaging referenced this issue in nesquena/rabl Nov 11, 2012
Open

Templates not found in view path #339

@jfelchner

@indirect I just ran into this issue when building a Rails engine. I had assumed since Bundler.require was being called from my dummy app that all would work correctly. From the dummy application.rb:

Bundler.require
require "my_gem_name"

But I have to use the workaround @myronmarston pointed out here in order to get it to work properly.

I obviously don't understand what you're saying and you have a better grasp on it than I do. Do you know of some way to tell my dummy app to use my gem's Gemfile so that when it Bundler.require's it it will do what I expect? Is this even something you would recommend? Or is it just the lazy, error-prone way out?

I looked through the Bundler docs and source code a bit and although I saw bundle-config could set the Gemfile, I didn't see anything in the source code that corresponded.

I attempted to do:

ENV['BUNDLE_GEMFILE'] = File.expand_path('path/to/my/gems/gemfile')
Bundler.require

But that didn't seem to work either.

@indirect
Bundler member

@jfelchner I think the very short answer is that you need to make sure your engine works without Bundler existing at all. Engines are distributed as gems, and will simply be required by Bundler as a shortcut so the developer working on the application that uses your engine doesn't have to write require 'your_engine' by hand.

@jfelchner

@indirect Perfect. Thank you for taking the time to reply.

@mscottford

I've confirmed that gems added via add_runtime_dependency don't show in the DEPENDENCIES section of Gemfile.lock. https://gist.github.com/4650109

However, I don't think this is a bug. The gem is still included in Gemfile.lock it just won't be required for you when running Bundler.require. As @indirect has mentioned, that's by design. Gem authors should call require to load runtime dependencies as they are needed.

It looks to me like this should be closed. Alternatively, we can clarify documentation around this point, since it seems to be a point of confusion.

@myronmarston

FWIW, as I've stated elsewhere, I think Bundler.require is an antipattern that should generally be avoided in any project (whether a gem or an app).

That said, I didn't know that the DEPENDENCIES section of Gemfile.lock was simply what bundler would require with Bundler.require. Does it have any purpose beyond that? If so, I'd still consider this a bug; if not, it'd be good to have some documentation around this (as @mscottford suggested) to lessen confusion.

@indirect
Bundler member

I've tried to document the non-Bundler.require on the Rubygems page, possibly poorly. :) If there's a way to make that clearer, suggestions or pulls would be great. Thanks!

@pmahoney

@indirect We have a toplevel app with both a Gemfile and a gemspec. Our Gemfile has a gemspec line and a few gems with :path specifiers. The gemspec itself is not used to build a gem, but is used to manage dependencies.

Just to be clear Bundler.require, will not require any dependencies listed in the gemspec, ever, even if I'm not developing a gem per-se? This is what I observe, and what I understand from reading the above. I find it unfortunate that there are now two places in which to manage dependencies; one for top-level apps and one for gems (for one thing, it precludes analysis/build/etc. tools from being able to pull information from one place).

@xaviershay

Closing because it's not clear to me what the next step nor if anything is actually still broken. Please re-open with clarification in the likely event I'm being dense. A concise summary of the above and where we're at today would be appreciated ... this thread is really long.

@xaviershay xaviershay closed this Aug 10, 2013
@altV

@indirect @mscottford @myronmarston Can you please clarify, why exactly auto-requiring dependencies of my gem, which are listed in the gemspec, with one command, is bad?

@briangonzalez

I kept all of my logic in my Gemfile, and then inside of my gemspec, I did the following:

# From gemfile.
Bundler.definition.specs.each{ |r| gem.add_dependency r.name, r.version }
@xaviershay

That seems really bad. Gemspecs are evaluated all over the place, and also need to be static. Tying them to a dependency on bundler and dynamically adding dependencies is going to break things in weird ways.

You need to do it the other way around: using the gemspec option in your Gemfile.

@briangonzalez

What am I to do if Bundler.require(:default) does not work with the gemspec line added to my Gemfile? See issue #1096.

@xaviershay

I'm not convinced #1096 is an exact duplicate of this bug. I can't replicate the behaviour (i.e. Bundler.require(:default) does include runtime dependencies for me.) If that isn't true for use, please open a new issue with repro steps. This ticket is large enough that I wouldn't want to overload it :)

@briangonzalez

You can delete this comment after you've read it if you'd like, but the first comment in #1096 outlines exactly how to reproduce it -- don't want to open another issue.

@xaviershay

Sorry I screwed up, my repro case wasn't exactly right and I was accidentally including the "missing" gem in another file (I cheated by just using an existing gem I had).

This comment above is the concise explanation of why this behaviour is the way it is.

@altV

@xaviershay Can I ask, what's bad in including your dependencies with one command (like Bundler.setup) instead of including them in a long list one by one?

@xaviershay

I don't know how to explain it better than the comment I just linked. There would have to be a feature in rubygems, not bundler, that supported this. Bundler should not ever be a runtime dependency of gems.

@paddor

So what's the status here? Isn't Bundle.setup(:default, :development) supposed to load the gem's runtime and development dependencies? (In my case it doesn't load anything.)

@TimMoore
Bundler member

@paddor Bundler.setup sets up the load paths, but does not load anything. Maybe you meant to use Bundler.require instead of Bundler.setup?

@paddor

OMG! Sorry. Totally mixed those two up. Of course I meant Bundler.require. Thanks a lot!

@manuelmorales manuelmorales added a commit to manuelmorales/mini-object that referenced this issue Jul 9, 2015
@manuelmorales manuelmorales I have been bitten by this bug or no-bug bundler/bundler#1041 . I gue…
…ss I should not commit my Gemfile.lock

Conflicts:
	Gemfile.lock
47b76cd
@malakai97

I am seeing this error in 1.10.6.

Steps to reproduce:
$ bundle gem bundler_bug_test
add 'spec.add_runtime_dependency "httpclient"' before default development dependencies.
$ bundle install --path .bundle
And now you can see that httpclient is not required if you do "Bundler.require" in e.g. bundle exec irb.

@grossadamm grossadamm pushed a commit to grossadamm/activerecord-tableless that referenced this issue May 20, 2016
agross gems must require their own dependencies, relying on bundler to requi…
…re them is wrong and causes issues if the order of the gems for a project is different then expected. (bundler/bundler#1041 (comment))
957e9d6
@grossadamm grossadamm referenced this issue in softace/activerecord-tableless May 20, 2016
Open

gems must require their own dependencies #29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment