Add :dest and :path sub-options to :git

[dchelimsky]

I'd like the ability to specify both the :path and :git keys in the same gem declaration:

gem "rails", :path => "./where/i/want/to/store/it", 
  :git => "git://github.com/rails/rails.git"

During bundle install, if the repo exists at :path, use it. If not, clone from the :git location and store it at :path. During bundle update, do a git pull from the repo located at :path.

[dchelimsky]

Here is the specific use case for this:

rspec-rails' development environment depends on rails being included in the bundle. It also requires that rails be stored in a local repo so that we can easily get from one version of rails to another by simply checking out the appropriate tag. This is stored ./vendor/rails.

The Gemfile in the root of the rspec-rails project includes:

gem "rails", :path => "./vendor/rails"

This leads to a catch 22, because when you first clone the rspec-rails repo and run "bundle install" the rails repo hasn't been cloned at ./vendor/rails yet. This can't be managed with a rake task, because the Rakefile operates in the same bundle, which won't load because the ./vendor/rails repo is not present. So the only way to get into a working state is to manually clone the rails repo, after which all is right with the world.

If bundler added support for :path and :git in the same declaration, bundle install would "just work." Yay.

[indirect]

So :path by definition implies that you are saying "I'm going to manage this myself. Bundler, don't mess with it."

The reason I am wary of implementing this is that I had previously thought about path + git support, but it did the exact opposite: if the path is not present, it would use the git repo as a standard git gem instead. That's because a lot of people use :path locally but need to use :git when they deploy.

While I'm not sure about adding this to bundler without some more discussion, have you thought about creating a simple bootstrap.sh or similar file that checks out rails for you?

[dchelimsky]

This isn't about me, it's about my users. I want rspec-rails development to be as easy as possible for contributors, and having to clone, bootstrap, and bundle install is 50% more steps than necessary :)

[indirect]

Well, the bootstrap would obviously clone and then bundle install... so it's still a single step, just a different one than bundle install. :)

[dchelimsky]

Sure, but git clone and bundle install are widely known and understood commands. It'd be nice to just rely on those. Also, if a person doesn't know to use the bootstrap script, then he'll just get an error when trying to run rake or bundle. What I did to work around this is check at the top of the rakefile if the repos are cloned already, and then raise a helpful error if not. I don't have any way of doing that with the bundle command.

So perhaps another solution would be to have a way to manage messages emitted by bundler.

gem "rails", :path => "./vendor/rails", :message => "./vendor/rails does not exist. Try running 'git clone git://github.com/rails/rails.git ./vendor/rails'

WDYT?

[indirect]

I feel like managing the version of vendored rails in this fashion is outside the scope of what bundler is intended to do, but I'll continue to think about this. I may become persuaded.

[indirect]

I'm also curious to hear your thoughts on this issue as it interacts with #231, which is a request to use :path in development and :git in production.

[dchelimsky]

I'm uncomfortable with expecting different behavior in different environments. I think it would lead to confusion, especially when the local and remote HEAD commits are different.

[indirect]

So how would you handle the case of local development and production deployment? Simply say "that's not allowed"? Suggest the hacky ENV workaround explained in #231?

[dchelimsky]

Why not just do that by groups:

group :development do
  gem "mongoid", :path => "#{ENV['DEV_BUNDLE']}/mongoid"
end
group :production do
  gem "mongoid", :git => "git://github.com/durran/mongoid.git"
end

[indirect]

Because unfortunately that doesn't really make sense -- --without doesn't actually remove groups from the Gemfile resolve, because that would break bundling entirely.

Using groups in that manner declares two gems, both named mongoid, one of which is at a path that may or may not exist, and the other is in git. How can bundler arbitrate between two contradictory gem declarations in a single manifest? :|

[dchelimsky]

Couple of thoughts come to mind.

1. :only and/or :except

gem "mongoid", :git => "git://github.com/durran/mongoid.git", :only => :production
gem "mongoid", :path => "local/path/to/mongoid", :except => :production

1. :env key

gem "mongoid", :env => {
  :development => { :path => "local/path/to/mongoid" },
  :production => { :git => "git://github.com/durran/mongoid.git" }
}

Both have benefits and drawbacks. Just floating ideas.

[josevalim]

I need something like this for Devise. Devise depends on Rails and I have both checked out as a git repository in the same folder. Since I don't want bundler to checkout Rails once again, I do this in my Gemfile:

git "rails", :path => "../rails"

But that obviously won't work if you don't have Rails there. So it would be nice to be able to say: use this path, if not, check it out. For me, I don't care if rails will be checked out at "../rails" or BUNDLER_DIR, both would work great in my case.

[indirect]

I talked this over with yehuda, and I think this is a really nice option to add to Bundler, but we should name it something else because it totally conflicts with the idea behind :path. How about this:

 gem "rails", :git => "...", :dest => "vendor/rails"

We could go with :dest, or :destination, or :clone_to, but I think I like :dest best. What do you guys think?

[josevalim]

I think David and I have different use cases. David wants to vendor it inside, I simply want to use that path/dest and, if not available, checkout the git repo (probably at the usual ~/.bundle location). In any case, :dest as option sounds great!

[indirect]

I think I am proposing that :path + :git be used for José's case (try this path, if it's not there, then clone this git repo as usual), and :dest + :git be used for David's case (try this path, if it's not there, then clone this git repo into that path). I will of course add documentation for these options to the git page of gembundler.com, since those uses aren't exactly obvious. Hopefully, though, the combination of :path and :git I am proposing is such that nothing will break.

[josevalim]

Awesome! Victory! Success!

[dchelimsky]

Both use cases say "try this path, then do something else". Where they differ is in the "do something else" part, not in the "path" part, and the difference is the destination. José's case is to store it in the bundle, mine is to store it in the :path. So how about this:

gem "rails", :git => "..", :path => "vendor/rails", :dest => :bundle
gem "rails", :git => "..", :path => "vendor/rails", :dest => :path

The default could be :bundle, so you José can just say:

gem "rails", :git => "..", :path => "vendor/rails"

WDYT?

[dchelimsky]

Either that, or something like this:

gem "rails", :path => "vendor/rails", :alt => { :git => "..", :dest => :bundle }

gem "rails", :path => "vendor/rails", :alt => { :git => "..", :dest => :path }

Very explicit.

[indirect]

I guess they don't seem like the same thing to me because in the one case, :path is the main option, and :git is a fallback in case :path doesn't work. In the other case, :git is the main option, and :dest is a sub-option to :git, like :ref is.

I'm not sure yet how to make that clear in the API, though.

[josevalim]

"I think I am proposing that :path + :git be used for José's case (try this path, if it's not there, then clone this git repo as usual), and :dest + :git be used for David's case (try this path, if it's not there, then clone this git repo into that path)."

+1 for this approach.

[hedgehog]

Is it not better to make any src-dest and fallback behavior explicit, unlimited and consistent between :path and :git?
Allow both :git and :path to accept a:

• string (specific)
• hash (specific)
• array (common)

Current behavior to remain (call these git-strings and gem-strings use cases):

gem "nokogiri", :git => "git://github.com/tenderlove/nokogiri.git"
gem "rails", :path => "./vendor/rails"

Inroduce a 'Git-hash'

git_hash= { :git => "<git-string>", :dest => "./where/i/want/to/store/it", :ref => "...", etc.}

Also allow a 'Gem-hash':

gem_hash= { :path => "gem-string", :dest => "./where/i/want/to/store/it", :version => "...", etc.}

In both hashes :dest => :bundle and :dest => :path are valid, defaulting to the current behavior.

try this path, if it's not there, then clone this git repo as usual
etc.

gem "rails", :path => gem_hash
gem "rails", :path => ["./vendor/rails", git_hash, gem_hash, "./other/path"]

gem "rails", :git => git_hash
gem "rails", :git => ["./vendor/rails", git_hash2, gem_hash2, "./other/path", etc.]

Note that when the :git/:path array element contains a string, it behaves just as the current deafult:

gem "rails", :path => "./vendor/rails"

Or when the string is any valid Git URI:

gem "nokogiri", :git => "git://github.com/tenderlove/nokogiri.git"

Specifically

• When :git => points to a hash a valid git-hash is expected.
• When :path => points to a hash a valid gem-hash is expected.

Finally, the :git/:path behave identicaly when given an array, and are able to process:

• git-strings (unchanged from current behavior)
• gem-strings (unchanged from current behavior)
• git-hashes
• gem-hashes.

I believe this solves dchelimsky's:

I'd like the ability to specify both the :path and :git keys in the same gem declaration

and

store it in the :path

and also josevalim's:

try this path, if it's not there, then clone this git repo as usual

and

store it in the bundle

[indirect]

Your "gem hash" doesn't make any sense. Why would you ever combine :path and :dest? That is a totally meaningless combination. Your "git hash" is what I'm already planning on doing, without the hash. No reason to make things more complicated if it's not needed.

[hedgehog]

Fair enough, drop it. From my reading of the above comments fallback behavior was really valued, and it'd be nice to have that behave consistently between :path and :git. Admittedly in my proposal it is the case that when given an array both :path and :git behave as if they are aliases.
Personally I alwsays thought :path was ambiguous, :dest is more explict, but it seems too late now - or should :dest be introduced and :path deprectade, but retained unitl 1.1? I suspect the vote would be to keep it, then introduce :dest at 1.x and try unring the :path bell later ;)

[hedgehog]

As my comment above makes clear, the function of :path is not unambiguous to users. :path cannot be deprecated without some replacement and I think I've decoded what :path is supposed to do and it is not as generic as its name suggests it is....

On reflection I think the correct way to address the 'issues' with :git and :path is in Issue #577.
I've seen some of josevalim and dchelimsky efforts so I know they are not dim - if they are having issues then I am not alone and I suspect most others will too.

indirect: please think then count to 100 before dismissing and closing Issues #577, your perfromance in both Issues #319 and #320 didn't leave an impression of taking thoughtful care in closing tickets - hopefully it was just a bad week.

[stephanschubert]

+1 for the "Try :path first, use :git otherwise" or "Different sources in different groups" solutions..

This issue is really annoying when developing a main application and several gems at the same time :/

[dburkes]

What's the status on this? From the comments, it looks like it has been fallow for almost a year. Is this planned?

[indirect]

The status is "on the list of things that would be nice to implement when we have time". The core team has limited time and is spending most of it on more universally applicable issues like speeding up bundle install. If someone writes a patch for this, I will try to get it in. If no one writes a patch for this, I will add it to Bundler when I have time to do that.

[timuckun]

Just like to interject this idea.

There should be a :host option for gems. This way each developer can specify their own path to the gem and also specify different gems for different machines whether production or development.

[axsuul]

Wasn't this feature supposed to be scheduled for 1.1 as per this blog post (scroll to bottom)? http://www.cowboycoded.com/2010/08/10/using-2-sources-for-a-gem-in-different-environments-with-bundler/

+1 from me, this feature is something that would help gem development immensely! Hope it gets implemented soon.

As of right now, I'm doing it like this:

require 'rails'

if Rails.env == "production"
    gem "foo", :git => "/path/to/repo"
else
    gem "foo", :path =>  "/path/to/gem"
end

[timuckun]

On Tue, Oct 4, 2011 at 7:15 PM, James
reply@reply.github.com
wrote:

Wasn't this feature supposed to be scheduled for 1.1 as per this blog post (scroll to bottom)? http://www.cowboycoded.com/2010/08/10/using-2-sources-for-a-gem-in-different-environments-with-bundler/

+1 from me, this feature is something that would help gem development immensely! Hope it gets implemented soon.

Does anybody know what the status is? I hate doing "bundler situps"
just to be able to work on a gem.

[indirect]

This will be implemented once there's a pull request with tests or someone on the core team has time to do it. We decided to pull the trigger so that everyone could have immensely faster installs, and we'll keep working on other features (like this one) as we have time.

On Oct 4, 2011, at 11:32 AM, Tim Uckunreply@reply.github.com wrote:

On Tue, Oct 4, 2011 at 7:15 PM, James
reply@reply.github.com
wrote:

Wasn't this feature supposed to be scheduled for 1.1 as per this blog post (scroll to bottom)? http://www.cowboycoded.com/2010/08/10/using-2-sources-for-a-gem-in-different-environments-with-bundler/

+1 from me, this feature is something that would help gem development immensely! Hope it gets implemented soon.

Does anybody know what the status is? I hate doing "bundler situps"
just to be able to work on a gem.

Reply to this email directly or view it on GitHub:
#396 (comment)

[indirect]

Whoa, didn't realize this ticket was still open. :) As of version 1.2, Bundler allows you to use a git repo that is checked out onto your machine without editing your Gemfile via a particular bundle config setting. Feel free to check out the git locals documentation for more information.