Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Conditions for child processes #90

Open
jbgo opened this Issue Mar 15, 2012 · 3 comments

Comments

Projects
None yet
3 participants

jbgo commented Mar 15, 2012

I needed support for monitoring the memory usage of child processes CPU/memory usage in a hurry, so I created this proof of concept branch: jbgo/god@918bc27

Is there any interest in refining this approach and getting it merged into god one day?

I see that there has already been interest and attempts at this same problem previously:
#30
#7

Here is an example that I'm currently using. Basically, I wanted to get a notification when a child of one of my resque workers exceeded a CPU or memory usage limit, so I can go back and debug the process. I transition to unmonitored because I want to leave the process running, but I don't need to get hundreds of notifications if I'm away from my computer.

w.transition(:up, :unmonitored) do |on|
  on.condition(:child_memory_usage) do |c|
    c.above =  200.megabytes
    c.times = 3
    c.notify = 'developers'
  end

  on.condition(:child_cpu_usage) do |c|
    c.above = 95.percent
    c.times = 3
    c.notify = 'developers'
  end
end

Hello,

I'm trying to use your patch on god, but I get the following error when running god:
E [2012-10-10 00:41:09] ERROR: Unhandled exception in ChildCpuUsage condition - (TypeError): wrong argument type Symbol (expected Proc)
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:37:in sum_ps_output' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:33:inps_float'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/system/portable_child_poller.rb:19:in percent_cpu' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/conditions/child_cpu_usage.rb:35:intest'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:367:in handle_poll' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:64:inhandle_event'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:181:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:inloop'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:179:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:innew'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/driver.rb:178:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:innew'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/task.rb:51:in initialize' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/watch.rb:39:ininitialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:in new' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:293:intask'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:281:in watch' /etc/god/httpd.god:1 /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:inload'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:639:in load' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:ineach'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god.rb:638:in load' /etc/god/god.conf:12 /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:inload'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:152:in load_god_file' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:144:inload_config'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:in each' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:143:inload_config'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:72:in default_run' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:80:inrun_in_front'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:23:in dispatch' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/../lib/god/cli/run.rb:8:ininitialize'
/usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124:in new' /usr/lib/ruby/gems/1.8/gems/god-0.13.1/bin/god:124 /usr/bin/god:19:inload'
/usr/bin/god:19

Seems like this doesn't work. Any ideas?

-david

The problem is my version of ruby. It's 1.8.5, and this patch needs 1.8.7. I can't upgrade all of my servers, so this patch wont work for me. :(

Sorry for re-opening an old issue, but this is of critical importance for us as well.
Numerous components use sub-processes, most recently puma in clustered mode (for zero-downtime deployment) and it's impossible to monitor them using god.
Any chance that this could be implemented?

In any case, thanks a lot for making such a great tool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment