Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add the ability to add a secondary Gemfile (that can be global to the system for example) #183

Closed
carllerche opened this Issue · 89 comments
@carllerche
  • the information in the secondary gemfile are lower priority
  • Do not include the secondary gemfile in a lock
  • the secondary Gemfile should not effect the resolve of the main Gemfile. It is a superset on top of it.
  • Add the ability to specify this secondary gemfile with an ENV variable, .bundle/config, or flag.
@qmx
qmx commented

That solves my concern with OS-specific-gems, like autotest-growl

@dolzenko

+1
Meanwhile, what's the best workaround to the issue? My .irbrc requires http://github.com/oggy/looksee from gem, by accident I discovered that the following lets me break out of the Bundler jail (at least in development mode):

Gem.refresh
Gem.activate('looksee')

so probably I can just detect bundler running and use this hack to require needed gem?

@indirect
Owner

I just put wirble.rb into ~/.irb/wirble.rb, and I require it from there in my .irbrc file.

@dolzenko

Right, but you can't just vendor a gem into your ~/.irb folder if it has native extensions (like the looksee gem does). Well maybe you can but this is going to be a mess, since you'd have to support compilation, different Ruby versions etc.

@indirect
Owner

Bundler is (obviously, I hope) not going to support putting things into your ~/.irb folder. You could do it as a workaround, though, even if the gem has native extensions. Just copy the files from the installed gem's folder.

@dolzenko

That doesn't scale when you have multiple extension incompatible Ruby versions (think rvm with 1.8 and 1.9) :)

@indirect
Owner

If you're using rvm, I don't understand how this is a problem at all. You can just bundle the gem straight out of your rvm system gems. Just add it to the Gemfile, and it will use the one you already have installed automatically, and running bundle install will install it to your rvm system gems.

@dolzenko

That is a problem because you will be adding developer specific gem to the project wide Gemfile, which is

  1. repetitive

  2. Gemfile may even not be owned by you

that's basically why I believe this feature request was submitted.

@indirect
Owner

I completely agree, and that is why we have committed to adding this feature to bundler. You asked about workarounds, however, and that's what I have been talking about this entire time.

@rickmark

I'd suggest Gemfile.local for the file...

It can be hacked together with some require 'Gemfile.lock' if File.exists?... until the implementation is ready

@Dykam

I would like this feature, and the way penwellr suggested seems good. Then you can just ignore it in the vcs ignore settings.

@tomkurt

Easy solution is just to require with full paths
require "/usr/local/lib/ruby/gems/1.8/gems/wirble-0.1.3/lib/wirble"

if the gem does relative requires like lookse use Dir.chdir("path') do ...;end

Dir.chdir("/usr/local/lib/ruby/gems/1.8/gems/looksee-0.2.1/lib") do
require "looksee/shortcuts"
end

@dolzenko

I just needed to include "ruby-debug" gem and since it depends on a couple of another gems that approach won't work.
(what I had to do is to manually add all the required load paths with $LOAD_PATH.unshift but that's really awkward)

@pwnall

Would like to add my use case to the thread: I'd like this feature so that my Rails deployment software can add the application server gem (thin / unicorn) to bundler's list of gems without messing with the Gemfile.

Right now, heroku asks developers to include "gem 'unicorn'" in their gem files. Ideally, the application's source code shouldn't be modified to reflect implementation details in the deployment environment.

@maxim

+1, can't use irb-enhancing gems without polluting everyone's Gemfile, or creating mess with ugly workarounds.

@ticktricktrack

+1, We're developing on OSX and Windows. We can either load additional Gemfiles OR put Gemfile.lock in our git repo. Both together would be nice.

@mynameisrufus

I thought this would work:

https://gist.github.com/724422

But no because we need to commit the Gemfile.lock

I don't like the idea of changing the gems that the app 'actually' needs but for testing its seems like win!

@jtoy

any progress updates on this?

@indirect
Owner

No, not really. It's an extremely hard problem, and everyone who works on Bundler has dayjobs now. :\

@skojin

my workaround https://gist.github.com/794915
add this method to .irbrc, allow include gems from rvm @global gemset, you should have bundler installed in this gemset

@rkh
rkh commented

any updates on this?

@skojin that link is broken.

@skojin

@rkh link works, copy/paste instead of click (should ends with 15)

@raggi
Collaborator
module Bundler
  # Activate a gem from outside the Gemfile. Could be considered harmful.
  # Might be useful for .irbrc and friends. It's a slow activation, but
  # after activation, files from the activated gems will be available for
  # normal require.
  def self.activate_outside(*gems)
    # Bundler doesn't cripple this:
    Gem.source_index.refresh!
    # Or this:
    Gem.activate(*gems)
  ensure
    # Re-enable the bundler lockdown via bundlers #initialize hack
    Gem.send(:class_variable_set, :@@source_index, nil)
  end
end
@indirect
Owner

Unfortunately, Gem.source_index is deprecated now, and is going to be removed. :(

@raggi
Collaborator

I might recommend applying bundlers hacks to Gem.source_index singleton.

That would enable me to simply instance a new Gem::SourceIndex and even cache it, rather than the hack-back included above.

@raggi
Collaborator

@indirect sure, i'll fix it when that goes and cripple_rubygems is updated.

@mschulkind

If you just want to be able to require gems from the global gemset, here's an alternative that's working for me. Just stick this in your .irbrc :

# Add all gems in the global gemset to the $LOAD_PATH so they can be used even
# in places like 'rails console'.
if defined?(::Bundler)
  global_gemset = ENV['GEM_PATH'].split(':').grep(/ruby.*@global/).first
  if global_gemset
    all_global_gem_paths = Dir.glob("#{global_gemset}/gems/*")
    all_global_gem_paths.each do |p|
      gem_path = "#{p}/lib"
      $LOAD_PATH << gem_path
    end
  end
end
@phocke

I tried all the solutions you proposed, but I still can't get over this issue

janlelis/irbtools#12 (comment)

any idea?

@kidpollo

+1

@imajes

+1 (for saving etc)

@Salzig

+1

@indirect
Owner

Don't remove any gems from Gemfile.lock. That will stop bundler from working entirely.

@ssoroka

+1

@delynn

+1

@ghost

+1

@thomsbg

+1

@phocke

Seems like an update:

https://gist.github.com/2063855

@ConradIrwin

I've posted a workaround for this issue when using Pry at https://github.com/ConradIrwin/pry-debundle (it uses hacks similar to the one posted by @raggi above, but updated for rubygems 1.8 — should be easy to adapt to irbtools if you want).

The basic approach is to keep everything locked down until the user is actually interactively working on their program; at which point the locks are removed and we allow them to include any gems they have installed. (that's the plan, the implementation might be buggy)

I could envisage this being less of a hack if there was a Bundler.relax! method which could be called by plugins, or the user themselves, when they want to enter interactive mode.

@wallace

+1

@skalnik

Yes, this would be super useful! :+1:

@cdlm

+1

@zaius

Here's my method of requiring from outside the Gemfile. Feedback welcome. https://gist.github.com/2643079

@gerrywastaken

+1

https://github.com/gerrywastaken/Gemfile.local

I ended up creating a Gemfile.local which loads the regular Gemfile, however for some strange reason I can't get the gemfile config to be written to or read from the .bundle/config file, so it's still only a partial resolution.

@terlar

I have assembled a few functions to deal with my needs for a local Gemfile.

The last added feature that did it for me is to "unlock" and run "unlocked" bundling. This is just a simple method to replace the local lock-file with your projects lockfile and do the bundling from there. Since we in our projects use a lockfile checked into the git repo.

In case anyone is interested I broke it out from my dotfiles and assembled it in a gist:
https://gist.github.com/3689896

@juanibiapina

I'd love to see this feature too. I have several gems that are specific to my needs, like tmux connectors, rspec formatters and so on.

@stephenmckinney

:+1: useful for test formatters, static analysis gems, and irb replacements

@FoGhost

+1

@ognevsky

Any progress on this feature? Let us hope, please :neckbeard:

@azach

+1, would love to use gems like https://github.com/charliesome/better_errors without forcing it on everyone else

@tmann-ttc

I have a solution that loads a Gemfile.local file, if you don't mind adding this line to end of your Gemfile:

eval(File.read(File.dirname(__FILE__) + '/Gemfile.local'), binding) rescue nil

A little ugly, but effective. Would love to see better syntax for this in bundler.

Thanks to Nathan's post for this idea.

@stephenmckinney

@tmann-ttc I like this on small apps that aren't generating serious $$$ but I couldn't convince the rest of my team at work that this should go into the Gemfile of a production application. This should really be a part of the Bundler feature set.

@azach

Another issue is that it doesn't address Gemfile.lock (which many teams commit to source control), which hopefully bundler would natively address as well.

@indirect
Owner

Guys, are you really aware of what you're asking for here? The main reason that Bundler exists (really, truly) is to make sure that the gems and versions you used in development are the same gems and versions in production. If you do any of the things discussed here (or if this makes it into Bundler as an official feature), you will be giving that up completely. Every deploy will become "hmm, does the production set of gems that no one has ever tried before actually work?". Is that something that you actually want? :confused:

@azach

Your concern is valid and it's true that such a feature could be abused, but that's already the case. If you use one database in development and another in production, you run into the same issue.

I feel like there is a valid use case for this, and that's tooling. See the gem I linked above -- it sits completely outside the app, so there's no issue with using it in production. But I don't see the reason it should be forced on the entire team if only I want to use it.

@stephenmckinney

My use case would be test formatters, static analysis gems, irb replacements, debuggers, etc. Gems that are tools for development, are not deployed to production, and may not be the tool used by everyone else on the team. So yes, this is a feature I would like to have. I understand and am willing to manage the risks.

I could see how this could be a support issue on the Bundler team. For that reason, it would be great if there was some convention to prevent users from doing stupid things that would affect production otherwise caveat emptor.

@tmann-ttc

What I really want to do is install my dev gems globally using gem install and not put them in the Gemfile at all. However, for my colleagues using RVM this does not work, apparently because of the way bundler interacts with RVM's gemsets. They found a workaround by forcing RVM to use its global gemset as the default, but it felt hacky. So maybe that is the real problem that needs solving. I don't use RVM (I use rbenv), so forgive my ignorance.

@duksis

+1

@sheerun

+1

@mattdbridges

We can do stuff like this in Rails, IRB, Pry, etc. Why not Bundler? :+1:

@sheerun

I understand that's because bundler would overwrite Gemfile.lock with dependencies defined in secondary Gemfile

@mattdbridges

@simi I understand that issue. Who would want that in a production environment? Besides, the idea is the global Gemfile is to be secondary to the app's Gemfile, thus making sure that if any conflicts between them arise, the primary one always wins. See then next two comments to understand what I was thinking about.

@amolpujari

+1

I run a command(which is result of a gem install from Gemfile), now to run this command I obviously need to point to to a Gemfile having that gem, right?

But my this command tries to execute bundle pointing out to different Gemfile

So need a way to tell bundler that now point out to this new Gemfile, and do your job

http://stackoverflow.com/questions/16336964/chef-solo-rails-bundle-install-when-having-separate-gemfiles

@localredhead

+1 . This would be useful

@amolpujari

gems_requires 'path to other Gemfile from this one'

@brandondrew

+1.
Gemfile.local seems like a good name for files that are specifically for a given project (but not kept in the repo). But if you want something that is global to all your project, but unique to you as a developer / kept out of repos, then I think ~/.Gemfile is a much better name. Both have use cases, so it seems useful to allow either to be used. (Though personally I'd probably only use the ~/.Gemfile, so I can have things like Jazz Hands in all my Rails projects.)

@simi

@brandondrew you can achieve this behaviour by installing your gems manually (or by creating own meta development gem with only dev. dependencies and no code) and little tricks in $HOME/.irbrc. The problem with secondary Gemfile is written here #183 (comment).

@stephenmckinney

@simi What if we documented some of these unsupported methods on a wiki page, caveat emptor? "Using gems in a development environment that are not in the gemfile" perhaps. Searching thru outdated gists is no fun.

If Bundler team thinks this is a horrible idea, I trust their judgment. This ticket can just be closed out if there is no intention of working on it. Personally, I'm less concerned about this being officially supported by Bundler and more concerned with just getting things done.

@djspinmonkey

+1

I've read this whole thread, including #183 (comment), and I don't think that's actually a relevant point. Bundler already allows different sets of gems to be used in development environments than are used in production. That is exactly what groups are most commonly used for -- almost every Gemfile I've ever seen had at least a test group, and usually a development group, too.

So, the new thing that Gemfile.local would allow is not "differences between development and production gem environments", but rather "differences between individual development environments," which I think most people would agree is far less scary.

As somebody who works in a fairly heterogenous environment (both in terms of OS and preferred tooling), this feature would be very useful to me.

@shlub

+1
I would really like to use the 'minitest-reporters' gem if it's installed on my local machine.
Don't want to bother my colleagues, who prabably won't like it.

@fxn
Collaborator
fxn commented

To better understand this proposal, how is bundler going to ensure dependencies are equal? Wouldn't a local Gemfile add more constraints that could end up in a different set of versions?

@shlub

You're totally right. I just ran into this problem as I tried to include a Gemfile.local via instance_eval within Gemfile.
As set in my local file 'minitest-reporters' relies on 'minitest < 5' and I want to specify the latest version in the Gemfile...
Too bad

@juanibiapina

It could resolve dependencies as usual and only after that check if the local dependencies are met (plus any new ones).

If they are not able to resolve properly considering the restrictions of the non local gems, there could be a warning or even an error. The local file wouldn't change the resolved versions of the gems. It's like a lower priority and it becomes your responsibility to fix it or drop it.

@hoff2

I'd like to +1 this with another use case. Not all Rails apps have only one production deployment, or if they have multiple, those are totally identical. I maintain a CMS app that has 50+ production deployments, each for a different client, and thus each has its own "theme" views and assets. A few of these clients also have some additional custom functionality added to the CMS and site via a plugin in vendor/plugins. Some of those plugins need certain gems. I need to be able to install those gems for those specific sites and have them work. If the Gemfile could include secondary Gemfiles from those plugins that would solve a big problem for me.

@indirect
Owner

@centipedefarmer the ruby code to make this possible isn't actually very hard or complicated. that said, we don't want to make this an officially supported part of Bundler because it just explodes the ways that gemfiles and gems can go horribly wrong. Since Bundler is a small team of volunteers, and we don't have time to provide support for those complicated situations, it's up to people who need it to do it themselves.

@indirect indirect closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.