Skip to content
This repository

Conditions for child processes #90

Open
jbgo opened this Issue March 15, 2012 · 3 comments

3 participants

Jordan Bach dnewhall Arno Dirlam
Jordan Bach
jbgo commented March 15, 2012

I needed support for monitoring the memory usage of child processes CPU/memory usage in a hurry, so I created this proof of concept branch: jbgo@918bc27

Is there any interest in refining this approach and getting it merged into god one day?

I see that there has already been interest and attempts at this same problem previously:
#30
#7

Here is an example that I'm currently using. Basically, I wanted to get a notification when a child of one of my resque workers exceeded a CPU or memory usage limit, so I can go back and debug the process. I transition to unmonitored because I want to leave the process running, but I don't need to get hundreds of notifications if I'm away from my computer.

w.transition(:up, :unmonitored) do |on|
  on.condition(:child_memory_usage) do |c|
    c.above =  200.megabytes
    c.times = 3
    c.notify = 'developers'
  end

  on.condition(:child_cpu_usage) do |c|
    c.above = 95.percent
    c.times = 3
    c.notify = 'developers'
  end
end
dnewhall

Hello,

I'm trying to use your patch on god, but I get the following error when running god:
E [2012-10-10 00:41:09] ERROR: Unhandled exception in ChildCpuUsage condition - (TypeError): wrong argument type Symbol (expected Proc)
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:37:in sum_ps_output'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:33:in
ps_float'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:19:in percent_cpu'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/conditions/child_cpu_usage.rb:35:in
test'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:367:in handle_poll'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:64:in
handle_event'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:181:in initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:in
loop'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:in initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:in
new'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:in initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:in
new'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:in initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/watch.rb:39:in
initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:in new'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:in
task'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:281:in watch'
/etc/god/httpd.god:1
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:in
load'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:in load'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:in
each'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:in load'
/etc/god/god.conf:12
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:in
load'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:in load_god_file'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:144:in
load_config'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:in each'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:in
load_config'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:72:in default_run'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:80:in
run_in_front'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:23:in dispatch'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:8:in
initialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124:in new'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124
/usr/bin/god:19:in
load'
/usr/bin/god:19

Seems like this doesn't work. Any ideas?

-david

dnewhall

The problem is my version of ruby. It's 1.8.5, and this patch needs 1.8.7. I can't upgrade all of my servers, so this patch wont work for me. :(

Arno Dirlam

Sorry for re-opening an old issue, but this is of critical importance for us as well.
Numerous components use sub-processes, most recently puma in clustered mode (for zero-downtime deployment) and it's impossible to monitor them using god.
Any chance that this could be implemented?

In any case, thanks a lot for making such a great tool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.