Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

bundle pack does not work with git sources #67

Closed
carllerche opened this Issue · 45 comments
@carllerche

No description provided.

@joevandyk

So you guys thinking it would zip up the sources when packed, and unzip on install?

@KieranP

Ideally it would create a gem file from the git repo.

i.e. gem build fromgit.gemspec

Then pack that. Although this might have issues if library authors only update the gemspec on releases.... that said, doesn't bundler bundle using gemspecs in git repos anyway?

@indirect
Owner

@KeiranP that idea is #94.

@stouset

#94 is in, but this still needs to be resolved.

@indirect
Owner

Just in case anyone needs a temporary workaround before this issue gets resolved, you can check out the git repositories you need at the refs that you need and then add the gems to you gemfile. For example:

cd vendor/git
git clone git://github.com/foo/foo.git

Then, in your Gemfile, gem "foo", :path => "vendor/git/foo".

@mjfreshyfresh

Has there been any traction on this? This would solve a major problem for us. It would also complete the functionality of bundle package.

@indirect
Owner

The core team doesn't need this for anything, so we'd like to do it, but it's lower priority than actual bugs and some other features. That does mean we would accept a patch to fix this, mjfreshyfresh. :)

@benhamill

I have done some looking into this issue, and wrote a test for it, but have been having some (tangential) problems that've been keeping me for working on it more.

The feature branch is here, mjfreshyfresh, if you want to fork and beat me to the punch with the pull request.

@kyledrake

I'd like to try to help with this, but I may lack sufficient knowledge in this area. This would be a huge feature, because you could make private gems for internal code and deploy it to cloud hosting like Heroku.

I spent the evening looking at the code in this part of bundler, and I'm puzzled as why the git sourced gems are implemented as a custom repository in the /usr/lib/ruby/gems folder. Would it make more sense install the git source as a gem in the normal gems folder? Basically clone a repository, run gem build, and then install it the same way as with regular rubygems.. and package from that source as well.

I'm sure there's a reason why this cannot happen, but I wanted to hear your thoughts so I could understand the depth of the problem.

@indirect
Owner

It's frequently not possible to run "gem build". It also opens up a massive issue of version conflicts, since it would mean you have a gem that (for example) claims to be version 1.0.4, but is in reality some version later than 1.0.4 but earlier than 1.0.5. The only effective way that we have found to resolve this is to identify git gems by the sha, and keep them separate from system gems.

@kyledrake

Thanks for the info indirect, good to know.

I've made an initial attempt at this, which you can see here.

I'm not sure (yet) if I need to account for things like branches/tags here, or if it already comes pre-prepared within that folder. So I'm not sure if this will work for all cases yet, but it works for a simple :git source example. More tests (and eyes) are needed. This could be the completely wrong approach, so I wanted to make sure it wasn't before continuing.

More details at the pull request.

@Rylon

This would be very useful for us too, any news on whether this will make it into 1.1?

@kyledrake

Could someone from the Bundler team look at this and tell me if this is even a sane approach to this? benhamill#1

I'm not familiar with the internals of Bundler so it's possible this is a lot more complicated than I'm seeing it. If not this may be a good starter..

@pastorius

@indirect,

Which version of bundler is required for the workaround you mentioned? As of 1.0.12, it appears that bundle-package does not include gems specified with the :path option. Am I missing something?

@kyledrake

@pastorius,

That allows you to containerize the gems with your vendor incase you need to. It basically circumvents bundle package for git sources completely. I haven't tried it but it should work.

@kyledrake

Or you could try my patch, but I haven't gotten anybody to look at it yet. It's probably missing something important.

@indirect
Owner

@pastorius, bundle pack does not and will not ever copy :path gems into vendor/cache. That's not how it works. If you want to, you can copy the gem's files into your source control repository (perhaps into vendor/gems?) and then include them in your gemfile with an option like :path => "./vendor/gems/gemname".

@kyledrake

No love for my patch at all? It works! Try it: kyledrake/bundler@c02b43b

@helabed

I will try it this weekend and let you know... my bad, didn't know patch existed !!
I Read about my solution here: http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-applies/ ( search for zubin)
before getting to this page.

@kyledrake

It sortof does. It works for me, but nobody's told me why it's not going to work yet (it probably won't).

@indirect
Owner

@kyledrake, it worked in my naive test too, but I don't use pack, so I'm afraid to officially bless it without someone else trying it. Worst case, we can pull this in for the 1.1 release candidates and hope someone uses it before the final. :P

@kyledrake

@indirect, Yeah I understand. I'm going to roll this patch into the bundler branch, and issue a direct pull request soon, so that it can be evaluated within the main project. I'll be issuing the pull request to master, let me know if you'd prefer it on a different branch.

@vincentp

Hi, any news about this feature? Would be nice to have everything packaged in the cache folder (even gems pointing to git)!

@kcrawford

Ran into this today. Would be very useful to have this fixed.

@marioizquierdo

Why not include a force cache option or something similar?

gem "private-gem", ">=2.1", :git => "private-repo", :cache_path => "vendor/gems/private-gem-2.1.gem"

So someone could easily update the cached version (with bundle update) and do not have to worry about private permissions on heroku, etc.

@pzol
@indirect
Owner

@kyledrake, any news on the direct pull request? I went to try to pull your patch into the next 1.1 prerelease, and your repo appears to be gone. Thanks! :)

@kyledrake

@indirect, this is probably not going to merge smoothly anymore unfortunately, I put it on @benhamill's repository: benhamill#1

If you'd like I could attempt to re-implement this again, or this (may) be a good start if you were interested in working on it. I never got a code review to see if I wasn't missing something really important. It worked great for me though!

@indirect
Owner

@kyledrake, it would be awesome if you could rebase and submit a pull against the the main repo, yeah! We're pedaling away trying to get a final-ish 1.0 bugfix release out and the master branch released as 1.1.rc. Thanks!

@ahawkins

@kyledrake can you please submit the pull to the mainline repo? This is such an important feature.

@jspooner

Yeah this is an important on. I was able to pull this patch benhamill#1 into my bundler and verified that bundle package places the source in vendor/cache. Although running bundle install --deployment still tries to download the source from github.

@wildchild

Much useful. Any status update?

@indirect
Owner

No one has submitted this patch to the main Bundler repo for review, yet. Please do.

@kyledrake

I cannot submit my original code because Bundler has changed substantially since I did that. It (my original pull request) will likely require a complete refactor to make this work, and unfortunately I'm really busy ATM. Honestly, there was probably a lot more thought required to this problem than my original patch suggested. I more wrote it to get a code discussion going. So if you're interested in this feature, take a look at what I did and use that as a base point to think about the problem.

@keo

I would really need this to be able to use private gems with Heroku and bundle pack.

@kyledrake

Yeah, and that was my original reason for needing this. I no longer use Heroku, so I don't need this anymore. I tried to get them to work on this to no avail but perhaps you will have better luck.

@benhamill

I'm not a member of Bundler core, so maybe I can be a bit callous on their behalf... Pull requests are welcome, y'all. Everyone involved has a day job. Please try to remember that, rather than implying the Bundler folks are being negligent or something for not having this done already.

@Rylon

There were several pull requests with possible approaches over a year ago when I originally came across this same issue. The problem is they never got merged and the core has now changed substantially, someone will have to have another attempt from scratch I guess.

@indirect
Owner

@rylon there has never been a single pull request to the main repository with a working, tested implementation. If you would like to provide one, that would be wonderful.

@hedgehog

As @indirect indicated, the really tricky part is fixing this while keeping the rest of Bundler working.

I did try address this issue as part of an effort to try and improve Bundlers git support.

I ended up playing whack-a-mole with Bundlers psuedo-spec's. Often (90% of the time) the specs would pass but Bundler (used via the cli) would be broken.

From memory @indirect did pull/evaluate in my changes but rejected them as being too much, or some such reason. That wasn't an inaccurate conclusion.

From my experience, a simple solution awaits the Bundler spec suite having full unit test coverage.

Of course all this was some time ago and matters may have improved since I lasted looked.

I suppose I'm saying: If you want to tackle this don't expect it to be a trivial exercise.

@benhamill

I had a similar experience as @hedgehog. I got into it and quickly found myself lost and alone in the woods at night. I didn't get as far as having anything even worth putting in a pull request for discussion.

@Rylon

My mistake - I thought there were a few different implementations in pull requests when I was originally looking into this issue last year.

@kyledrake

@benhamill I'd like to consider myself a member of the bundler team (I've contributed code, see the pull request above). "Them" in my comment was referring to Heroku, apologies that that wasn't obvious. I asked them a few times to work on this, but they closed my tickets and my request didn't get to the right people. Since this principally affects Heroku users, I think it makes sense to have a dev from Heroku implement this feature. Their day job is to (arguably, I suppose) fix stuff like this. This could be as simple as e-mailing somebody that works on Ruby stuff at Heroku and asking them to work on it.

If anyone is looking for a workaround, there is a private gem cloud app called Gemfury. Worth looking into, it's made by the guy that did Gem in a Box and he's a great guy, I totally trust him to do a good job.

@josevalim josevalim was assigned
@hone
Owner

@kyledrake Hi! I'm the ruby guy from Heroku that you should email :) Sorry, we never got around to implementing this feature, but luckily @josevalim did! <3 <3. You can try it out right now in bundler 1.2.0.pre using the bundle pack --all AND it runs on cedar right now. Please post any bugs/issues you run into.

@hone hone closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.