Selectively Cache Gems

[Fluxx]

I have what I imagine to be a not-so-uncommon use case. I have a project hosted on Heroku that uses mostly public gems, save for a couple private ones. Since I don't want embed my Github un/pw in my Gemfile or pay/run my own private gemserver like Gemfury, the best solution at present for bundling my private gems on Heroku is to cache all my gems in vendor/cache where Bundler will find and use when Heroku runs bundle install for me.

This certainly works, but the down side is that I have to cache all my gems, not just the ones that are private. Caching all the gems bloats the size of my git repo, which isn't a huge deal but does slow things down. One solution to this to .gitignore the vendor/cache directory, and then manually git add my private gems as needed. This also certainly works, but has two small downsides:

1. Bundler still copies over all the .gem files locally, which is redundant if they're already on my file system in the normal rubygems cache.
2. It requires me to remember to git add the new version of the gem every time I upgrade it. I might forget to do this, and confuse other developers and potentially cause hard to track down problems.

Again, these are small annoyances, but it seems like a way to selectively cache gems would be best. That way you could only cache the certain gems that you needed/wanted cached.

My thought on how to do this was the following:

1. Add a cache config option to bundle config, which would allow users to configure whether caching was used by default or not. Setting it to false would not automatically cache gems during a bundle install. This config option could also be called autocache.
2. Add support in the Gemfile for a :cache => true option, which would force the gem to be cached, overriding the cache bundle config option. This config option could also be called :force_cache or something more descriptive.

I looked through the existing issues and source code, so I apologize if this has been suggested before or if there is a better way to do it. Also, I'm happy to actually do the work and submit a pull request, I just wanted to check and make sure this feature has a chance of being merged before I do.

[indirect]

I may be mistaken, but isn't there already a --no-cache option that turns off autocaching?

On Mar 25, 2012, at 2:02 PM, Jeff Pollardreply@reply.github.com wrote:

I have what I imagine to be a not-so-uncommon use case. I have a project hosted on Heroku that uses mostly public gems, save for a couple private ones. Since I don't want embed my Github un/pw in my Gemfile or pay/run my own private gemserver like Gemfury, the best solution at present is to cache all my gems in vendor/cache where Bundler will find and use when Heroku runs bundle install for me.

This certainly works, but the down side is that I have to cache all my gems, not just the ones that are private. Caching all the gems bloats the size of my git repo, which isn't a huge deal but does slow things down. One solution to this to .gitignore the vendor/cache directory, and then manually git add my private gems as needed. This also certainly works, but has two small downsides:

1. Bundler still copies over all the .gem files locally, which is redundant if they're already on my file system in the normal rubygems cache.
2. It requires me to remember to git add the new version of the gem every time I upgrade it. I might forget to do this, and confuse other developers and potentially cause hard to track down problems.

Again, these are small annoyances, but it seems like a way to selectively cache gems would be best. That way you could only cache the certain gems that you needed/wanted cached.

My thought on how to do this was the following:

1. Add a cache config option to bundle config, which would allow users to configure whether caching was used by default or not. Setting it to false would not automatically cache gems during a bundle install. This config option could also be called autocache.
2. Add support in the Gemfile for a :cache => true option, which would force the gem to be cached, overriding the cache bundle config option. This config option could also be called :force_cache or something more descriptive.

I looked through the existing issues and source code, so I apologize if this has been suggested before or if there is a better way to do it. Also, I'm happy to actually do the work and submit a pull request, I just wanted to check and make sure this feature has a chance of being merged before I do.

---

Reply to this email directly or view it on GitHub:
#1795

[Fluxx]

Ah there is, but it's only documented on gembundler.com, not in bundle help install. That said, installing with --no-cache still has problem #2 from my message above. I'd still like bundler to be able to autocache certain specific gems.

[rohitarondekar]

@Fluxx As an aside, if it's not documented in the man page/bundle help install, do you mind adding it there and making a PR?

[Fluxx]

@Rohit Will do.

[kbrock]

On a similar note:

When I use bundle install --path .bundle, bundler stores 2 copies of my gems.

Both in .bundle/ruby/1.9.1/gems/cache and vendor/cache

And in the .bundle directory keeps older gems.

But that is probably just my user error.

[indirect]

@kbrock that is completely correct, and exactly how Rubygems is supposed to work. The gems in vendor/cache are the "source" gems, taking the place of rubygems.org. The gems in .bundle/ruby/x/gems/cache are the locally installed copies, put there by Rubygems/Bundler.

[Fluxx]

@indirect Was there any word on this? Again I'm happy to put in the time to do with work, I just wanted to know it was a feature that Bundler wanted. If you guys don't want this feature too, that's understandable. Just wanted to know either way.

[indirect]

@Fluxx, I think Bundler should support caching some gems but not all of them. Right now, that is possible if you manually cache gems (or run bundle package and then delete unwanted gems) and then install with --no-cache. A better API seems good, and a :cache option in the Gemfile might be it. @hone, @wycats, any thoughts?

[ghost]

I find myself in need of this due to the difficulty of deploying an app using gems hosted in private GitHub repos to Heroku.

[indirect]

After thinking about this more, I think the best UI for this looks something more like this:

• Gems already in vendor/cache are updated automatically during install
• Gems not in vendor/cache are not added automatically during install
• Adding all gems to vendor/cache is done by running bundle cache
• Adding a specific gem to vendor/cache is done by running bundle cache foo

Does that seem reasonable to everyone?

[peter]

+1 for this feature. I also have private gems that I need to share between apps on Heroku. I was initially using the git directive and fetching the private gem from Github credentials in environment variables. Unfortunately I ran into issues with this so I'm looking for alternatives, i.e. caching the private gem or using a private repo like Gemfury.

Ideally I would also like to be able to experiment with changes in the gem and have them live reloaded (or at least loaded on server restart) in my Rails app. Any pointers on how to do that? I tried putting a path/git switch in the Gemfile based on environment variables but ran it to various issues with that.

[indirect]

@peter You want the "local git" feature. Check out the local git docs.

[peter]

@indirect Thanks! "local git" looks really useful!

[xaviershay]

Closing because it's a feature request, please continue discussion over at https://github.com/bundler/bundler-features or provide a patch.

[md5]

Forgive me for commenting on a long-closed issue, but did this issue ever get raised over at bundler/bundler-features? I wasn't able to find it over there.

As far as I can tell, there's still no way to selectively cache certain gems other than running bundle package --all and manually deleting all but the desired gems.

[indirect]

I think that's the workaround that people use today. If you still want this, please open a feature request ticket explaining the scenario and why you want to cache only some gems. Thanks!

[md5]

Thanks @indirect. I opened rubygems/bundler-features#98