Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Including child processes in CPU/memory usage #30

Open
wants to merge 2 commits into from

3 participants

@jstepien

This changeset allows God to watch total CPU/memory usage of a process family. I use God to monitor a deamon which forks a lot and monitoring a single PID isn't sufficient. Now I can simply add a following restart rule in order to keep the whole family under control.

w.restart_if do |restart|
  restart.condition(:memory_usage) do |c| 
    c.above = 210.megabytes
    c.times = 5 
    c.family = true
  end 

  restart.condition(:cpu_usage) do |c| 
    c.above = 80.percent
    c.times = 5 
    c.family = true
  end 
end 

rake test doesn't report any regressions.

@mojombo
Owner

This looks really useful. Any chance you could work up a test for this, to make sure it's working properly?

@jstepien

Sure. Here's a memory usage test. I guess something similar can be written to test CPU usage. Looking forward for your opinion.

@mojombo
Owner

I'm getting the following error on my system (OSX):

  1) Error:
test_family_memory_usage(TestSystemProcessFamily):
TypeError: #<Mocha::Mock:0x1033f6bd8> is not a class/module
    ./test/test_system_process_family.rb:36:in `test_family_memory_usage'
    /Users/tom/.rvm/gems/ruby-1.8.7-p248/gems/mocha-0.9.12/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:22:in `__send__'
    /Users/tom/.rvm/gems/ruby-1.8.7-p248/gems/mocha-0.9.12/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:22:in `run'

Are you seeing this at all?

@jstepien

No, I'm not, but I also had some problems. I'm on Linux and Ruby 1.9.2. I copied the test file structure from test/test_system_process.rb. I wasn't able to run test/test_system_process.rb and the test I've added using Rake. I've managed to run them using

$ ruby -I path/to/mocha-0.9.12/lib/ -I. test/test_system_process.rb
$ ruby -I path/to/mocha-0.9.12/lib/ -I. test/test_system_process_family.rb

Without adding mocha to the load path I get a following error.

$ ruby -I. test/test_system_process.rb 
E [2011-05-26 16:44:09] ERROR: => You need the Mocha gem to run these tests.
@eric eric commented on the diff
lib/god/system/process.rb
((5 lines not shown))
+ # Returns an array of PIDs of the polled process and all its children.
+ def family
@eric Collaborator
eric added a note

This method should be moved into the PortablePoller and an implementation should be written for the SlashProcPoller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 10, 2010
  1. @jstepien
Commits on May 25, 2011
  1. @jstepien
This page is out of date. Refresh to see the latest.
View
14 lib/god/conditions/cpu_usage.rb
@@ -13,6 +13,9 @@ module Conditions
# populated for Watches.
# +above+ is the percent CPU above which to trigger the condition. You
# may use #percent to clarify this amount (see examples).
+ # Optional
+ # +family+ should be true in order to include the CPU usage of all child
+ # processes.
#
# Examples
#
@@ -29,12 +32,13 @@ module Conditions
# c.pid_file = "/var/run/mongrel.3000.pid"
# end
class CpuUsage < PollCondition
- attr_accessor :above, :times, :pid_file
+ attr_accessor :above, :times, :pid_file, :family
def initialize
super
self.above = nil
self.times = [1, 1]
+ self.family = false
end
def prepare
@@ -62,7 +66,11 @@ def valid?
def test
process = System::Process.new(self.pid)
- @timeline.push(process.percent_cpu)
+ if self.family
+ @timeline.push(process.family_percent_cpu)
+ else
+ @timeline.push(process.percent_cpu)
+ end
history = "[" + @timeline.map { |x| "#{x > self.above ? '*' : ''}#{x}%%" }.join(", ") + "]"
@@ -77,4 +85,4 @@ def test
end
end
-end
+end
View
14 lib/god/conditions/memory_usage.rb
@@ -14,6 +14,9 @@ module Conditions
# the condition should trigger. You can also use the sugar
# methods #kilobytes, #megabytes, and #gigabytes to clarify
# this amount (see examples).
+ # Optional
+ # +family+ should be true in order to include the memory usage of all
+ # child processes.
#
# Examples
#
@@ -31,12 +34,13 @@ module Conditions
# c.pid_file = "/var/run/mongrel.3000.pid"
# end
class MemoryUsage < PollCondition
- attr_accessor :above, :times, :pid_file
+ attr_accessor :above, :times, :pid_file, :family
def initialize
super
self.above = nil
self.times = [1, 1]
+ self.family = false
end
def prepare
@@ -64,7 +68,11 @@ def valid?
def test
process = System::Process.new(self.pid)
- @timeline.push(process.memory)
+ if self.family
+ @timeline.push(process.family_memory)
+ else
+ @timeline.push(process.memory)
+ end
history = "[" + @timeline.map { |x| "#{x > self.above ? '*' : ''}#{x}kb" }.join(", ") + "]"
@@ -79,4 +87,4 @@ def test
end
end
-end
+end
View
26 lib/god/system/process.rb
@@ -24,6 +24,12 @@ def exists?
def memory
@poller.memory
end
+
+ # Memory usage in kilobytes (resident set size), including all child
+ # processes
+ def family_memory
+ family.map { |pid| fetch_system_poller.new(pid).memory } .reduce(:+)
+ end
# Percentage memory usage
def percent_memory
@@ -34,6 +40,11 @@ def percent_memory
def percent_cpu
@poller.percent_cpu
end
+
+ # Percentage CPU usage including all child processes
+ def family_percent_cpu
+ family.map { |pid| fetch_system_poller.new(pid).percent_cpu } .reduce(:+)
+ end
private
@@ -44,7 +55,20 @@ def fetch_system_poller
PortablePoller
end
end
- end
+ # Returns an array of PIDs of the polled process and all its children.
+ def family
@eric Collaborator
eric added a note

This method should be moved into the PortablePoller and an implementation should be written for the SlashProcPoller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ stack = [@pid]
+ n = 0
+ while stack.count > n
+ pid = stack[n]
+ output = `ps -e -oppid=,pid= | grep '^#{pid}'`
+ children = output.split("\n").map { |x| x.sub(/^\d+\s/, '').to_i }
+ stack += children
+ n += 1
+ end
+ stack
+ end
+ end
end
end
View
9 test/test_system_process.rb
@@ -19,6 +19,11 @@ def test_memory
assert @process.memory > 0
end
+ def test_family_memory
+ assert_kind_of Integer, @process.family_memory
+ assert @process.family_memory > 0
+ end
+
def test_percent_memory
assert_kind_of Float, @process.percent_memory
end
@@ -26,5 +31,9 @@ def test_percent_memory
def test_percent_cpu
assert_kind_of Float, @process.percent_cpu
end
+
+ def test_family_percent_cpu
+ assert_kind_of Float, @process.family_percent_cpu
+ end
end
View
44 test/test_system_process_family.rb
@@ -0,0 +1,44 @@
+require 'drb/drb'
+require 'ostruct'
+require File.dirname(__FILE__) + '/helper'
+
+class TestSystemProcessFamily < Test::Unit::TestCase
+ DRUBY_URI = 'druby://localhost:38753'
+
+ def setup
+ pid = Process.pid
+ @process = System::Process.new(pid)
+ end
+
+ def test_family_memory_usage
+ pid = fork
+ if pid.nil?
+ my_process = System::Process.new(Process.pid)
+ mem_before = my_process.memory
+ # Allocate some memory.
+ data = (0..10_000).reduce('') { |acc, int| acc + int.to_s }
+ # Create an object which will be used to communicate with the parent
+ # process.
+ obj = OpenStruct.new :done => false,
+ :used_memory => my_process.memory - mem_before
+ DRb.start_service DRUBY_URI, obj
+ at_exit { DRb.stop_service }
+ sleep 0.1 while not obj.done
+ exit 0
+ else
+ begin
+ obj = DRbObject.new_with_uri DRUBY_URI
+ allocated_by_child = obj.used_memory
+ # Do the actual memory assertion.
+ assert (@process.family_memory - @process.memory) > allocated_by_child
+ # Ask the child process to die.
+ obj.done = true
+ rescue DRb::DRbConnError => e
+ # Child's DRb service isn't running yet. Retry.
+ sleep 0.1
+ retry
+ end
+ end
+ end
+end
+
Something went wrong with that request. Please try again.