Skip to content
This repository has been archived by the owner on Apr 14, 2021. It is now read-only.

Bundle install running out of memory #356

Closed
dbackeus opened this issue Jun 2, 2010 · 16 comments
Closed

Bundle install running out of memory #356

dbackeus opened this issue Jun 2, 2010 · 16 comments

Comments

@dbackeus
Copy link

dbackeus commented Jun 2, 2010

Using Bundler while deploying to a 64 bit ubuntu 10.04 vps with 256MB ram I get this output/error.

$ bundle install --without test development

Updating git@github.com:streamio/logparser.git
Updating git://github.com/plataformatec/devise
Updating git://github.com/justinfrench/formtastic.git
Updating git://github.com/dbackeus/monque.git
Updating git://github.com/durran/mongoid.git
Updating git://github.com/rails/rails.git
Fetching source index from http://rubygems.org/
Updating git://github.com/rails/rails.git
/usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:352:in ``': Cannot allocate memory - git fetch --force --quiet "git://github.com/rails/rails.git" refs/heads/*:refs/heads/* (Errno::ENOMEM)
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:352:in `git'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:382:in `cache'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:413:in `chdir'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:413:in `in_cache'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:382:in `cache'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/source.rb:320:in `specs'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/environment.rb:51:in `send'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/environment.rb:51:in `resolve'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/environment.rb:49:in `each'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/environment.rb:49:in `resolve'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/installer.rb:66:in `resolve_remotely'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/environment.rb:28:in `specs'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/installer.rb:21:in `run'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/installer.rb:6:in `install'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/cli.rb:90:in `install'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/task.rb:33:in `send'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/task.rb:33:in `run'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/invocation.rb:109
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/invocation.rb:116:in `call'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/invocation.rb:116:in `invoke'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor.rb:137:in `start'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor/base.rb:378:in `start'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/lib/bundler/vendor/thor.rb:124:in `start'
from /usr/lib/ruby/gems/1.8/gems/bundler-0.9.25/bin/bundle:13
from /usr/bin/bundle:19:in `load'
from /usr/bin/bundle:19

If I monitor the memory usage while running the command it's clear that bundler is eating up more and more memory as it goes along (seems to syncronize with every "Updating git://..." output) starting out at 24MB and crashing at around 150MB (even though there is still 100MB to go on the system).

This seems like rather a lot of memory usage for a tool like bundler to use.

Let me know if you'd like more information about the VPS I'm running on etc. I noticed "privmpages" failures in the admin interface for the VPS but I'm not knowledgeable enough to tell wether that can mean there is a problem with the VPS setup rather than bundler itself.
@indirect
Copy link
Member

You need to lock your bundle if you're using Bundler 0.9, with the bundle lock command. Run that on your development box, and then check in the Gemfile.lock file. For more information on deploying with Bundler 0.9, see http://gembundler.com/deploy.html

@dbackeus
Copy link
Author

I did that already.

When comparing the difference between the 32 and 64 bit setups I noticed that the 32 bit system showed git being run in separate processes while the 64 bit system seemed to run git inside of the ruby process and thus bloating the memory.

I have since abandoned the 64 bit systems for the time being.

@jackdempsey
Copy link
Contributor

FWIW I've seen this as well. Fighting through a bunch of issues tonight--hard to know exactly what's going on given that i'm trying to bundle to edge rails and a couple other gems on a 384mb slicehost instance, and of course bundle install seems to work when I run it manually, fail from capistrano. Still, wanted to mention other's have seen it just in case it helps.

@mjankowski
Copy link

I'm seeing this with bundler 1.0.0.beta9 deploying via capistrano.

@hgtesta
Copy link

hgtesta commented Jul 22, 2010

It also happens to me with bundler 1.0.0.beta9. I can't setup my application on a 512 Mb linode instance. I tried also to package (bundle package) all gems but bundler eats all memory when fetching source index from gemcutter.org. BTW, I don't know why fetch index if all the gems are versioned and on vendor/cache.

@dolzenko
Copy link

Same thing here. Running on Rackspace 64-bit 246Mb Ubuntu 10 instance bundler RC1 eats 348 Mb of memory and never completes. I have Gemfile.lock and gems cached in vendor/cache

@khy
Copy link

khy commented Jul 27, 2010

Agreed - during Capistrano deploy, bundler rc.1 takes ~250 Mb and ultimately fails with Errno::ENOMEM on bundle install.

@dolzenko
Copy link

After being really irritated with that issue I just did cd vendor/cache && gem install --local --no-rdoc --no-ri *.gem %) (having done bundle lock on dev machine before that of course)

@ghost
Copy link

ghost commented Jul 29, 2010

bundle install vendor/bundle --local finally licked the memory issue for us.

@indirect
Copy link
Member

indirect commented Aug 6, 2010

This is (hopefully) resolved as of 43fe70e and 1.0rc3, which drastically reduces the memory used by bundle install.

@johngrimes
Copy link

I'm having this problem with Bundler 1.0.0. I'm unable to issue a bundle install --deployment on my 512MB slice, the bundle process gets up to > 150MB of memory usage.

@warrenseen
Copy link

Using Bundler 1.0.1 on a 384MB slicehost slice, I needed to shutdown all other daemons just to run bundle install --deployment. Clearly whatever improvements have been made in memory handling still are not enough to fully resolve this :(

@francois
Copy link
Contributor

Bundler 1.0.15, REE 1.8.7-2011.03 on a Linode VPS with 256 MiB of RAM, I got an Errno::ENOMEM as well. I had to bundle cache and bundle install --path vendor/local --local before I could run with no memory errors.

@indirect
Copy link
Member

@francois, unfortunately the master index contains the gemspec for every single gem that exists. That means loading it into RAM is pretty costly. Try out the Bundler 1.1 prerelease with gem install bundler --pre, which tries to load only the gemspecs that it actually needs to resolve the gemfile and let me know if the RAM usage there is lower.

@c10l
Copy link

c10l commented Nov 15, 2011

I know this is closed, but I wanted to second @indirect's solution.
I had the same issues mentioned, but upgrading bundler to the pre 1.1 version nailed it.
Thanks!

@fideloper fideloper mentioned this issue Mar 10, 2014
Merged
@clayrisser
Copy link

I had this issue with Bundler version 1.13.4. bundle lock seemed to fix it. I have no idea why.

This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests