New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assets:precompile freezes after 10 minutes #55

Closed
lukaszsliwa opened this Issue Jan 29, 2014 · 32 comments

Comments

Projects
None yet
@lukaszsliwa

lukaszsliwa commented Jan 29, 2014

I have a problem with assets:precompile. It freezes after 10 minutes of precompiling assets. Process ends but it still waits for something and no response given.

INFO [2fc05e34] Running ~/.rvm/bin/rvm 2.0.0-p247@apptamers do bundle exec rake assets:precompile on x12s2.staging.apptamers.com
DEBUG [2fc05e34] Command: cd /home/webservice/project/20140129075713 && ( RAILS_ENV=production ~/.rvm/bin/rvm 2.0.0-p247@apptamers do bundle exec rake assets:precompile )
DEBUG [2fc05e34]    /home/webservice/.rvm/rubies/ruby-2.0.0-p247/bin/ruby /home/webservice/project/shared/bundle/ruby/2.0.0/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets

CPU is 100% usage but it's normal when I precompile assets on this server.

I have set up ssh_config entries to make sure that it's not a ssh problem. So it's not a timeout problem.

$ cat ~/.ssh/config 
Host *
   ServerAliveInterval 15

I wonder if it's not a sshkit problem and maybe someone has the same issue.

capistrano-rails 1.1.0
capistrano       3.0.1
sshkit           1.3.0

When I do CTR+C after 1h, i get:

/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/sshkit-1.3.0/lib/sshkit/runners/parallel.rb:15:in `join'
/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/sshkit-1.3.0/lib/sshkit/runners/parallel.rb:15:in `map'
/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/sshkit-1.3.0/lib/sshkit/runners/parallel.rb:15:in `execute'
/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/sshkit-1.3.0/lib/sshkit/coordinator.rb:21:in `each'
/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/sshkit-1.3.0/lib/sshkit/dsl.rb:8:in `on'
/home/user/.rvm/gems/ruby-2.0.0-p247@apptamers/gems/capistrano-rails-1.1.0/lib/capistrano/tasks/assets.rake:60:in `block (3 levels) in <top (required)>'
...

This runners/parallel.rb looks interesting, maybe this is a sshkit runner problem?

@kirs

This comment has been minimized.

Show comment
Hide comment
@kirs

kirs Jan 30, 2014

Member

I faced with the similar issues when I deployed with poor internet connection. Actually, it was wifi point-to-point bridged over 2 buildings.

Could you try it on another connection?

Member

kirs commented Jan 30, 2014

I faced with the similar issues when I deployed with poor internet connection. Actually, it was wifi point-to-point bridged over 2 buildings.

Could you try it on another connection?

@lukaszsliwa

This comment has been minimized.

Show comment
Hide comment
@lukaszsliwa

lukaszsliwa Jan 30, 2014

@kirs Thanks for the response. Connection speed test:

Download: 35.42 Mb/s
Upload:   6.74 Mb/s
Ping:     13 ms

My wifi router is about 1m from my computer.

Anyway, i have found that sshkit has Know issues https://github.com/capistrano/sshkit#known-issues and frist two are:

...
No handling of slow / timed out connections
No handling of slow / hung remote commands
...

I was trying to play with SSHKit::Backend::Netssh.pool.idle_timeout = 0 but nothing was happen. I was trying to change a runner from parallel to sequence. Still the same.

lukaszsliwa commented Jan 30, 2014

@kirs Thanks for the response. Connection speed test:

Download: 35.42 Mb/s
Upload:   6.74 Mb/s
Ping:     13 ms

My wifi router is about 1m from my computer.

Anyway, i have found that sshkit has Know issues https://github.com/capistrano/sshkit#known-issues and frist two are:

...
No handling of slow / timed out connections
No handling of slow / hung remote commands
...

I was trying to play with SSHKit::Backend::Netssh.pool.idle_timeout = 0 but nothing was happen. I was trying to change a runner from parallel to sequence. Still the same.

@lukaszsliwa

This comment has been minimized.

Show comment
Hide comment
@lukaszsliwa

lukaszsliwa Jan 30, 2014

I have added:

# config/environments/production.rb
config.assets.logger = Logger.new $stdout

and now it responses, shows assets precompilation log and capistrano successfully finishes a deploy. It doesn't freeze like before.

In general, it doesn't solve a problem (what if someone doesn't want to see the log?) but it works.

lukaszsliwa commented Jan 30, 2014

I have added:

# config/environments/production.rb
config.assets.logger = Logger.new $stdout

and now it responses, shows assets precompilation log and capistrano successfully finishes a deploy. It doesn't freeze like before.

In general, it doesn't solve a problem (what if someone doesn't want to see the log?) but it works.

@Jesus

This comment has been minimized.

Show comment
Hide comment
@Jesus

Jesus Feb 10, 2014

I'm having exactly the same issue, but your trick didn't work for me.

The SSH connection silently fails. I'm guessing it's due to inactivity in the stream, as the command is taking a lot of time in the server.

I presume that it'd work fine if I managed to put some content in the stream. Ie. enabling the logger as you suggested, for example.

Jesus commented Feb 10, 2014

I'm having exactly the same issue, but your trick didn't work for me.

The SSH connection silently fails. I'm guessing it's due to inactivity in the stream, as the command is taking a lot of time in the server.

I presume that it'd work fine if I managed to put some content in the stream. Ie. enabling the logger as you suggested, for example.

@yourivdlans

This comment has been minimized.

Show comment
Hide comment
@yourivdlans

yourivdlans Feb 23, 2014

I'm also having this issue. Tried setting the ServerAliveInterval and enabling the logger. But I don't get any output so the connection silently fails.

When I remove all the precompiled assets before a deployment I see the files getting precompiled, the next time I deploy I don't get any output but ruby does use 100% cpu for about 6 minutes and the deployment never finishes.
There are a lot of assets though. Locally it takes around 3 minutes. The second time I run the command its much faster because nothing changed.

When I try this sequence manually on the actual machine (clobber assets -> precompile -> precompile again) the first precompile takes around 6 minutes and the second precompile just takes seconds to finish.

Any thoughts?

yourivdlans commented Feb 23, 2014

I'm also having this issue. Tried setting the ServerAliveInterval and enabling the logger. But I don't get any output so the connection silently fails.

When I remove all the precompiled assets before a deployment I see the files getting precompiled, the next time I deploy I don't get any output but ruby does use 100% cpu for about 6 minutes and the deployment never finishes.
There are a lot of assets though. Locally it takes around 3 minutes. The second time I run the command its much faster because nothing changed.

When I try this sequence manually on the actual machine (clobber assets -> precompile -> precompile again) the first precompile takes around 6 minutes and the second precompile just takes seconds to finish.

Any thoughts?

@kirs

This comment has been minimized.

Show comment
Hide comment
@kirs

kirs Feb 24, 2014

Member

@yourivdlans which rails version do you have?

Member

kirs commented Feb 24, 2014

@yourivdlans which rails version do you have?

@yourivdlans

This comment has been minimized.

Show comment
Hide comment
@yourivdlans

yourivdlans Feb 24, 2014

I'm running rails 4.0.2 with ruby 2.0.0p353

yourivdlans commented Feb 24, 2014

I'm running rails 4.0.2 with ruby 2.0.0p353

@asanger

This comment has been minimized.

Show comment
Hide comment
@asanger

asanger Mar 3, 2014

I've actually also been running into this. I'm on Rails 4.0.2 with ruby 2.1.0. The strange thing is that I reverted to a much older commit and the deploy was successful. Then I immediately returned to my latest commit and deployed, which worked. The last deploy used the exact same source that the first deploy froze up on. Very very strange...

asanger commented Mar 3, 2014

I've actually also been running into this. I'm on Rails 4.0.2 with ruby 2.1.0. The strange thing is that I reverted to a much older commit and the deploy was successful. Then I immediately returned to my latest commit and deployed, which worked. The last deploy used the exact same source that the first deploy froze up on. Very very strange...

@mattbrictson

This comment has been minimized.

Show comment
Hide comment
@mattbrictson

mattbrictson Mar 3, 2014

Member

Have you tried enabling net-ssh keepalive? Something like this might work:

server "example.com",
       :user => "deployer",
       :roles => %w(web app),
       :ssh_options => {
         :keepalive => true,
         :keepalive_interval => 60 #seconds
       }
Member

mattbrictson commented Mar 3, 2014

Have you tried enabling net-ssh keepalive? Something like this might work:

server "example.com",
       :user => "deployer",
       :roles => %w(web app),
       :ssh_options => {
         :keepalive => true,
         :keepalive_interval => 60 #seconds
       }
@asanger

This comment has been minimized.

Show comment
Hide comment
@asanger

asanger Mar 3, 2014

Doesn't look like that helped. I just attempted to revert to the same old commit that previously worked, and now that deploy properly either. Everything is sticking on assets:precompile with 100% cpu.

asanger commented Mar 3, 2014

Doesn't look like that helped. I just attempted to revert to the same old commit that previously worked, and now that deploy properly either. Everything is sticking on assets:precompile with 100% cpu.

@mattbrictson

This comment has been minimized.

Show comment
Hide comment
@mattbrictson

mattbrictson Mar 3, 2014

Member

@asanger does the precompile ever finish? Or does it peg the CPU indefinitely? For the OP the precompile finished but capistrano remains stuck. I'm wondering if you are seeing the same thing or if it is a different problem.

Member

mattbrictson commented Mar 3, 2014

@asanger does the precompile ever finish? Or does it peg the CPU indefinitely? For the OP the precompile finished but capistrano remains stuck. I'm wondering if you are seeing the same thing or if it is a different problem.

@seku

This comment has been minimized.

Show comment
Hide comment
@seku

seku Mar 4, 2014

+1
ruby 2.0.0-p247
rails 4.0.0

seku commented Mar 4, 2014

+1
ruby 2.0.0-p247
rails 4.0.0

@mattbrictson

This comment has been minimized.

Show comment
Hide comment
@mattbrictson

mattbrictson Mar 4, 2014

Member

I added 500 SCSS files to my app in an attempt to reproduce this issue, but the deploy worked fine. The precompile pegged the CPU at 100% for 18 minutes (as expected), but once the precompile finished, then the capistrano deploy proceeded without any problems. The deploy did not hang.

My gem versions:

$ bundle show |egrep ' rails|capistrano|ssh'
  * capistrano (3.1.0)
  * capistrano-bundler (1.1.2)
  * capistrano-rails (1.1.1)
  * net-ssh (2.8.0)
  * rails (4.0.3)
  * sshkit (1.3.0 723d0a0)
$ ruby -v
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-darwin13.0]

Relevant deploy.rb configuration:

set :ssh_options, :compression => false, :keepalive => true

Server environment is Ubuntu 12.04 LTS.

Member

mattbrictson commented Mar 4, 2014

I added 500 SCSS files to my app in an attempt to reproduce this issue, but the deploy worked fine. The precompile pegged the CPU at 100% for 18 minutes (as expected), but once the precompile finished, then the capistrano deploy proceeded without any problems. The deploy did not hang.

My gem versions:

$ bundle show |egrep ' rails|capistrano|ssh'
  * capistrano (3.1.0)
  * capistrano-bundler (1.1.2)
  * capistrano-rails (1.1.1)
  * net-ssh (2.8.0)
  * rails (4.0.3)
  * sshkit (1.3.0 723d0a0)
$ ruby -v
ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-darwin13.0]

Relevant deploy.rb configuration:

set :ssh_options, :compression => false, :keepalive => true

Server environment is Ubuntu 12.04 LTS.

@seku

This comment has been minimized.

Show comment
Hide comment
@seku

seku Mar 5, 2014

Hi. I should remove my post. The reason of hanging compilation of assets in my case was failing rake command not capistrano itself

seku commented Mar 5, 2014

Hi. I should remove my post. The reason of hanging compilation of assets in my case was failing rake command not capistrano itself

@yourivdlans

This comment has been minimized.

Show comment
Hide comment
@yourivdlans

yourivdlans Mar 12, 2014

Thanks for checking that @mbrictson. What I am interested in is what did your precompile do when you immediately deployed again? And what happens when you edit just one file? Will it run for 18 min or just the time it took to update that one file.

yourivdlans commented Mar 12, 2014

Thanks for checking that @mbrictson. What I am interested in is what did your precompile do when you immediately deployed again? And what happens when you edit just one file? Will it run for 18 min or just the time it took to update that one file.

@mattbrictson

This comment has been minimized.

Show comment
Hide comment
@mattbrictson

mattbrictson Mar 18, 2014

Member

Hi @yourivdlans. I ran the test you requested and got these results:

Test Time elapsed
Immediately redeploy the same SHA 15 seconds
Change one SCSS file (not a partial) and deploy 18 seconds
Change a partial SCSS file (one that is @imported by all 500 stylesheets) 18 minutes

As expected, if a change affects all stylesheets, they all need to recompiled. If a change is local to a small number of stylesheets, the sprockets cache is used and compilation is fast.

(To ensure the sprockets cache works, you need tmp/cache in your capistrano :linked_dirs so that the cache is reused between deploys.)

This is an argument for avoiding global variables and mixins when structuring your Sass projects, and using @import sparingly, as I've written about here: http://blog.55minutes.com/2013/01/lightning-fast-sass-reloading-in-rails-32/

Member

mattbrictson commented Mar 18, 2014

Hi @yourivdlans. I ran the test you requested and got these results:

Test Time elapsed
Immediately redeploy the same SHA 15 seconds
Change one SCSS file (not a partial) and deploy 18 seconds
Change a partial SCSS file (one that is @imported by all 500 stylesheets) 18 minutes

As expected, if a change affects all stylesheets, they all need to recompiled. If a change is local to a small number of stylesheets, the sprockets cache is used and compilation is fast.

(To ensure the sprockets cache works, you need tmp/cache in your capistrano :linked_dirs so that the cache is reused between deploys.)

This is an argument for avoiding global variables and mixins when structuring your Sass projects, and using @import sparingly, as I've written about here: http://blog.55minutes.com/2013/01/lightning-fast-sass-reloading-in-rails-32/

@yourivdlans

This comment has been minimized.

Show comment
Hide comment
@yourivdlans

yourivdlans Mar 20, 2014

Thanks for running these tests!

Adding tmp/cache to my linked_dirs fixed my problem! Thanks

yourivdlans commented Mar 20, 2014

Thanks for running these tests!

Adding tmp/cache to my linked_dirs fixed my problem! Thanks

@umhan35

This comment has been minimized.

Show comment
Hide comment
@umhan35

umhan35 Mar 25, 2014

Contributor

I'm having the same issue. But is anybody thinking make the precompile process run locally?

Contributor

umhan35 commented Mar 25, 2014

I'm having the same issue. But is anybody thinking make the precompile process run locally?

@nicolasgarnil

This comment has been minimized.

Show comment
Hide comment
@nicolasgarnil

nicolasgarnil Apr 22, 2014

Adding keepalive: true to ssh_options fixed my problem.

nicolasgarnil commented Apr 22, 2014

Adding keepalive: true to ssh_options fixed my problem.

@kirs

This comment has been minimized.

Show comment
Hide comment
@kirs

kirs Jun 7, 2014

Member

Can be closed?

Member

kirs commented Jun 7, 2014

Can be closed?

@terraplane

This comment has been minimized.

Show comment
Hide comment
@terraplane

terraplane Oct 9, 2014

I also have described above problem. No success with config.assets.logger = Logger.new $stdout or keepalive: true option. Process on the server is silently stopping and no reaction on my side in the capistrano console output.

What is really srange: we have two servers and I have the problem only on one of them. The second server's deployment works perfect.

terraplane commented Oct 9, 2014

I also have described above problem. No success with config.assets.logger = Logger.new $stdout or keepalive: true option. Process on the server is silently stopping and no reaction on my side in the capistrano console output.

What is really srange: we have two servers and I have the problem only on one of them. The second server's deployment works perfect.

@sebfie

This comment has been minimized.

Show comment
Hide comment
@sebfie

sebfie Oct 22, 2014

Same issue with on of my collaborator... It works for me but not for him

sebfie commented Oct 22, 2014

Same issue with on of my collaborator... It works for me but not for him

@alexfouche

This comment has been minimized.

Show comment
Hide comment
@alexfouche

alexfouche Dec 19, 2014

Problem solved by adding

ClientAliveInterval 30
ClientAliveCountMax 10

to /etc/ssh/sshd_config on the server. It seems the sshkit gem used by Capistrano responds to these SSH probes, which makes it work.
I chose every 30 seconds, because one of the network equipment, between our collaborator in Spain and the servers in France, seems to be quite aggressive to cut apparent inactive connections.

alexfouche commented Dec 19, 2014

Problem solved by adding

ClientAliveInterval 30
ClientAliveCountMax 10

to /etc/ssh/sshd_config on the server. It seems the sshkit gem used by Capistrano responds to these SSH probes, which makes it work.
I chose every 30 seconds, because one of the network equipment, between our collaborator in Spain and the servers in France, seems to be quite aggressive to cut apparent inactive connections.

@alexfouche

This comment has been minimized.

Show comment
Hide comment
@alexfouche

alexfouche commented Dec 19, 2014

@AlfredoRoca

This comment has been minimized.

Show comment
Hide comment
@AlfredoRoca

AlfredoRoca Feb 4, 2016

In my case, assets precompilation failed (using both capistrano3 and mina-deploy) due to not enough memory. I realize that free memory almost went to 0 and I hadn't any swap file/partition. Then, I added a swap file and It began to work fine.

AlfredoRoca commented Feb 4, 2016

In my case, assets precompilation failed (using both capistrano3 and mina-deploy) due to not enough memory. I realize that free memory almost went to 0 and I hadn't any swap file/partition. Then, I added a swap file and It began to work fine.

@yanaizmene

This comment has been minimized.

Show comment
Hide comment
@yanaizmene

yanaizmene Apr 4, 2016

Having same problem but not only on precompile stage but for every thing that took some time to finish and outputs nothing. For example bundle install.

image

Now have the same problem with rubber, when task rubber:bootstrap tries to install packages.

I'm deploying from OS X yosemite to ubuntu 14.04 x64 on digital ocean.

ClientAliveInterval 30 and ClientAliveCountMax 10 didn't solve the problem.

yanaizmene commented Apr 4, 2016

Having same problem but not only on precompile stage but for every thing that took some time to finish and outputs nothing. For example bundle install.

image

Now have the same problem with rubber, when task rubber:bootstrap tries to install packages.

I'm deploying from OS X yosemite to ubuntu 14.04 x64 on digital ocean.

ClientAliveInterval 30 and ClientAliveCountMax 10 didn't solve the problem.

@will-in-wi

This comment has been minimized.

Show comment
Hide comment
@will-in-wi

will-in-wi Aug 6, 2016

Contributor

Is this still an issue?

Contributor

will-in-wi commented Aug 6, 2016

Is this still an issue?

@zx1986

This comment has been minimized.

Show comment
Hide comment
@zx1986

zx1986 Sep 12, 2016

I got this issue too.
But it happened sometimes, not every time, weird.
Ran with in docker container, it goes ok on the another machine (totally the same VM).

root@1dbae00efbef:/home/app# rake RAILS_ENV=production assets:precompile --trace
** Invoke assets:precompile (first_time)
** Invoke assets:environment (first_time)
** Execute assets:environment
** Invoke environment (first_time)
** Execute environment
** Execute assets:precompile    <= sometimes half here for 70 sec above.


** Invoke ckeditor:nondigest (first_time)
** Invoke environment
** Execute ckeditor:nondigest
** Invoke ckeditor:nondigest

I just can't dump the details logs :-(

zx1986 commented Sep 12, 2016

I got this issue too.
But it happened sometimes, not every time, weird.
Ran with in docker container, it goes ok on the another machine (totally the same VM).

root@1dbae00efbef:/home/app# rake RAILS_ENV=production assets:precompile --trace
** Invoke assets:precompile (first_time)
** Invoke assets:environment (first_time)
** Execute assets:environment
** Invoke environment (first_time)
** Execute environment
** Execute assets:precompile    <= sometimes half here for 70 sec above.


** Invoke ckeditor:nondigest (first_time)
** Invoke environment
** Execute ckeditor:nondigest
** Invoke ckeditor:nondigest

I just can't dump the details logs :-(

@lukaszsliwa

This comment has been minimized.

Show comment
Hide comment
@lukaszsliwa

lukaszsliwa Sep 12, 2016

One of the possible reason why it happens, is server limitations like less RAM. Once it happened the process is killed by the system.

lukaszsliwa commented Sep 12, 2016

One of the possible reason why it happens, is server limitations like less RAM. Once it happened the process is killed by the system.

@vedant1811

This comment has been minimized.

Show comment
Hide comment
@vedant1811

vedant1811 Jan 18, 2017

Adding the keepalive config helped me. Solved the issue so far.

vedant1811 commented Jan 18, 2017

Adding the keepalive config helped me. Solved the issue so far.

@rubinsh

This comment has been minimized.

Show comment
Hide comment
@rubinsh

rubinsh Feb 20, 2017

@lukaszsliwa - Thanks - your solution of adding
config.assets.logger = Logger.new $stdout to production.rb
worked for me!

rubinsh commented Feb 20, 2017

@lukaszsliwa - Thanks - your solution of adding
config.assets.logger = Logger.new $stdout to production.rb
worked for me!

@will-in-wi

This comment has been minimized.

Show comment
Hide comment
@will-in-wi

will-in-wi Oct 1, 2017

Contributor

It sounds like there is a workaround for these situations. Closing.

Contributor

will-in-wi commented Oct 1, 2017

It sounds like there is a workaround for these situations. Closing.

@will-in-wi will-in-wi closed this Oct 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment