Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

3.1.1 assets:precompile running out of memory #3694

Closed
samlown opened this Issue · 11 comments

8 participants

@samlown

Hi,

assets:precompile spawns another process for assets:precompile:nondigest before terminating itself. Each instance of the rake task on our setup uses over 150MB of RAM (according to top), so the second run ends up gobbling a total of 300MB. The linux kernel on the 512MB node just ends up telling the rake task its a greedy SOB by killing it. Capistrano then aborts the deploy.

Using two capistrano tasks for assets:precompile:primary and assets:precompile:nondigest solves the problem. I don't know enough about the precompile process and Sprockets to recommend an proper solution (the whole "if digest, compile nondigest" thing is very confusing [1]) but perhaps a Kernel.exec can be used instead to pass control to the new process instead of leaving the old lingering?

Cheers,
sam

[1] #3225 (comment)

@millisami

+1.
The linode instance of 768MB Ram is also leaking out.
All the Swaps are occupied and capistrano task error out with zlib(finalizer): the stream was freed prematurely
What could be the soln??

@arunagw
Collaborator

One way is to compile them in local and then attach it will the application. :-)

@millisami

Any response from the core team?

@yob
yob commented

We've run into this as well - seems a bit excessive to launch three processes

@vitaly-pushkar

I have faced to this too on 500Mb VDS

@mjtko

Hi guys,

I did some work on this back in #3225 in October, so I have a pretty good understanding of how the assets Rake task works.

It is irritating that in order to accomplish the assets precompilation three processes are required - as stated in the OP it can be minimized to two independent tasks (or one if you only want one of digestless or digested assets).

How to fix 'properly'? Well, that will require work on sprockets core as it internally caches what has already been compiled on each compilation run - hence the need for two separate taks. In fact, even if the sprockets work is done, the trivial invocation will still require two tasks as rake needs to be reinvoked with RAILS_GROUPS=assets and RAILS_ENV=production (though that can, ofc, be mitigated by supplying those variables on the command line).

I suppose that, as there are workarounds for the issue at the moment, this won't receive high priority for a fix.

@millisami

For now, I've changed the strategy of assets compilation.

I compile the assets locally, tar n gz it, upload it and untar and place in the /public/assets dir while deploying via custom deploy task.

@kennyj kennyj referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@kennyj
Collaborator

Hey Guys!

I tried several ways to improve.
As a result, the way to improve is the above commit (I think that I maintained compatibility).

I executed bundle exec rake assets:clean assets:precompile

Before:

Every 1.0s: ps aux | grep ruby | grep -v grep | grep -v vim                                                                                                                   Fri Jan  6 00:33:36 2012

kennyj   21818  9.0  6.0 171996 30920 pts/0    Sl+  00:33   0:01 ruby /home/kennyj/.rvm/gems/ruby-1.9.3-p0/bin/rake assets:clean assets:precompile
kennyj   21854 25.2 10.4 212652 53148 pts/0    Sl+  00:33   0:03 /home/kennyj/.rvm/rubies/ruby-1.9.3-p0/bin/ruby /home/kennyj/.rvm/gems/ruby-1.9.3-p0/bin/rake assets:precompile:all RAILS_ENV=product
kennyj   21919 85.0 10.0 209236 51372 pts/0    Sl+  00:33   0:03 /home/kennyj/.rvm/rubies/ruby-1.9.3-p0/bin/ruby /home/kennyj/.rvm/gems/ruby-1.9.3-p0/bin/rake assets:precompile:nondigest RAILS_ENV=p

We had 3 processes, because assets:precompile:all(primary) 's process was not terminated.

After (with the commit):

Every 1.0s: ps aux | grep ruby | grep -v grep | grep -v vim                                                                                                                   Fri Jan  6 00:35:56 2012

kennyj   22552 10.5  6.0 172000 30916 pts/0    Sl+  00:35   0:02 ruby /home/kennyj/.rvm/gems/ruby-1.9.3-p0/bin/rake assets:clean assets:precompile
kennyj   22588 51.4  9.6 206604 49284 pts/0    Rl+  00:35   0:07 /home/kennyj/.rvm/rubies/ruby-1.9.3-p0/bin/ruby /home/kennyj/.rvm/gems/ruby-1.9.3-p0/bin/rake assets:precompile:nondigest RAILS_ENV=p

We had 2 processes.

I use Kernel.exec only if assets:precompile:nondigest is invoked in assets:precompile:all.
@samlown mentioned it :)

@millisami

So, is this gonna be included in the next rails upgrade!

@kennyj kennyj referenced this issue from a commit in kennyj/rails
@kennyj kennyj Don't fork a process when assets:precompile:nondigest is invoked in a…
…ssets:precompile:all. Improve GH #3694.
a342df4
@kennyj kennyj referenced this issue from a commit in kennyj/rails
@kennyj kennyj Don't fork a process when assets:precompile:nondigest is invoked in a…
…ssets:precompile:all. Improve GH #3694.
2056744
@kennyj
Collaborator

@samlown PR 4350 was merged to master, and PR 4352 wad merged to 3-2-stable.
If you have no more problem, please close this issue. thanks.

@vatrai vatrai referenced this issue from a commit in castlerock/rails
@kennyj kennyj Don't fork a process when assets:precompile:nondigest is invoked in a…
…ssets:precompile:all. Improve GH #3694.
1358b8c
@guilleiguaran

Closing since the PR of @kennyj was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.