Skip to content
This repository

Including child processes in CPU/memory usage #30

Open
wants to merge 2 commits into from

3 participants

Jan Stępień Tom Preston-Werner Eric Lindvall
Jan Stępień

This changeset allows God to watch total CPU/memory usage of a process family. I use God to monitor a deamon which forks a lot and monitoring a single PID isn't sufficient. Now I can simply add a following restart rule in order to keep the whole family under control.

w.restart_if do |restart|
  restart.condition(:memory_usage) do |c| 
    c.above = 210.megabytes
    c.times = 5 
    c.family = true
  end 

  restart.condition(:cpu_usage) do |c| 
    c.above = 80.percent
    c.times = 5 
    c.family = true
  end 
end 

rake test doesn't report any regressions.

Tom Preston-Werner
Owner
mojombo commented May 24, 2011

This looks really useful. Any chance you could work up a test for this, to make sure it's working properly?

Jan Stępień

Sure. Here's a memory usage test. I guess something similar can be written to test CPU usage. Looking forward for your opinion.

Tom Preston-Werner
Owner
mojombo commented May 25, 2011

I'm getting the following error on my system (OSX):

  1) Error:
test_family_memory_usage(TestSystemProcessFamily):
TypeError: #<Mocha::Mock:0x1033f6bd8> is not a class/module
    ./test/test_system_process_family.rb:36:in `test_family_memory_usage'
    /Users/tom/.rvm/gems/ruby-1.8.7-p248/gems/mocha-0.9.12/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:22:in `__send__'
    /Users/tom/.rvm/gems/ruby-1.8.7-p248/gems/mocha-0.9.12/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:22:in `run'

Are you seeing this at all?

Jan Stępień

No, I'm not, but I also had some problems. I'm on Linux and Ruby 1.9.2. I copied the test file structure from test/test_system_process.rb. I wasn't able to run test/test_system_process.rb and the test I've added using Rake. I've managed to run them using

$ ruby -I path/to/mocha-0.9.12/lib/ -I. test/test_system_process.rb
$ ruby -I path/to/mocha-0.9.12/lib/ -I. test/test_system_process_family.rb

Without adding mocha to the load path I get a following error.

$ ruby -I. test/test_system_process.rb 
E [2011-05-26 16:44:09] ERROR: => You need the Mocha gem to run these tests.
Eric Lindvall eric commented on the diff December 16, 2011
lib/god/system/process.rb
((5 lines not shown))
48 58
   
  59
+      # Returns an array of PIDs of the polled process and all its children.
  60
+      def family
1
Eric Lindvall Collaborator
eric added a note December 16, 2011

This method should be moved into the PortablePoller and an implementation should be written for the SlashProcPoller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
14  lib/god/conditions/cpu_usage.rb
@@ -13,6 +13,9 @@ module Conditions
13 13
     #                populated for Watches.
14 14
     #     +above+ is the percent CPU above which to trigger the condition. You 
15 15
     #             may use #percent to clarify this amount (see examples).
  16
+    #   Optional
  17
+    #     +family+ should be true in order to include the CPU usage of all child
  18
+    #              processes.
16 19
     #
17 20
     # Examples
18 21
     #
@@ -29,12 +32,13 @@ module Conditions
29 32
     #     c.pid_file = "/var/run/mongrel.3000.pid"
30 33
     #   end
31 34
     class CpuUsage < PollCondition
32  
-      attr_accessor :above, :times, :pid_file
  35
+      attr_accessor :above, :times, :pid_file, :family
33 36
     
34 37
       def initialize
35 38
         super
36 39
         self.above = nil
37 40
         self.times = [1, 1]
  41
+        self.family = false
38 42
       end
39 43
       
40 44
       def prepare
@@ -62,7 +66,11 @@ def valid?
62 66
       
63 67
       def test
64 68
         process = System::Process.new(self.pid)
65  
-        @timeline.push(process.percent_cpu)
  69
+        if self.family
  70
+          @timeline.push(process.family_percent_cpu)
  71
+        else
  72
+          @timeline.push(process.percent_cpu)
  73
+        end
66 74
         
67 75
         history = "[" + @timeline.map { |x| "#{x > self.above ? '*' : ''}#{x}%%" }.join(", ") + "]"
68 76
         
@@ -77,4 +85,4 @@ def test
77 85
     end
78 86
     
79 87
   end
80  
-end
  88
+end
14  lib/god/conditions/memory_usage.rb
@@ -14,6 +14,9 @@ module Conditions
14 14
     #             the condition should trigger. You can also use the sugar
15 15
     #             methods #kilobytes, #megabytes, and #gigabytes to clarify
16 16
     #             this amount (see examples).
  17
+    #   Optional
  18
+    #     +family+ should be true in order to include the memory usage of all
  19
+    #              child processes.
17 20
     #
18 21
     # Examples
19 22
     #
@@ -31,12 +34,13 @@ module Conditions
31 34
     #     c.pid_file = "/var/run/mongrel.3000.pid"
32 35
     #   end
33 36
     class MemoryUsage < PollCondition
34  
-      attr_accessor :above, :times, :pid_file
  37
+      attr_accessor :above, :times, :pid_file, :family
35 38
       
36 39
       def initialize
37 40
         super
38 41
         self.above = nil
39 42
         self.times = [1, 1]
  43
+        self.family = false
40 44
       end
41 45
       
42 46
       def prepare
@@ -64,7 +68,11 @@ def valid?
64 68
       
65 69
       def test
66 70
         process = System::Process.new(self.pid)
67  
-        @timeline.push(process.memory)
  71
+        if self.family
  72
+          @timeline.push(process.family_memory)
  73
+        else
  74
+          @timeline.push(process.memory)
  75
+        end
68 76
         
69 77
         history = "[" + @timeline.map { |x| "#{x > self.above ? '*' : ''}#{x}kb" }.join(", ") + "]"
70 78
         
@@ -79,4 +87,4 @@ def test
79 87
     end
80 88
     
81 89
   end
82  
-end
  90
+end
26  lib/god/system/process.rb
@@ -24,6 +24,12 @@ def exists?
24 24
       def memory
25 25
         @poller.memory
26 26
       end
  27
+
  28
+      # Memory usage in kilobytes (resident set size), including all child
  29
+      # processes
  30
+      def family_memory
  31
+        family.map { |pid| fetch_system_poller.new(pid).memory } .reduce(:+)
  32
+      end
27 33
       
28 34
       # Percentage memory usage
29 35
       def percent_memory
@@ -34,6 +40,11 @@ def percent_memory
34 40
       def percent_cpu
35 41
         @poller.percent_cpu
36 42
       end
  43
+
  44
+      # Percentage CPU usage including all child processes
  45
+      def family_percent_cpu
  46
+        family.map { |pid| fetch_system_poller.new(pid).percent_cpu } .reduce(:+)
  47
+      end
37 48
       
38 49
       private
39 50
       
@@ -44,7 +55,20 @@ def fetch_system_poller
44 55
           PortablePoller
45 56
         end
46 57
       end
47  
-    end
48 58
   
  59
+      # Returns an array of PIDs of the polled process and all its children.
  60
+      def family
  61
+        stack = [@pid]
  62
+        n = 0
  63
+        while stack.count > n
  64
+          pid = stack[n]
  65
+          output = `ps -e -oppid=,pid= | grep '^#{pid}'`
  66
+          children = output.split("\n").map { |x| x.sub(/^\d+\s/, '').to_i }
  67
+          stack += children
  68
+          n += 1
  69
+        end
  70
+        stack
  71
+      end
  72
+    end
49 73
   end
50 74
 end
9  test/test_system_process.rb
@@ -19,6 +19,11 @@ def test_memory
19 19
     assert @process.memory > 0
20 20
   end
21 21
   
  22
+  def test_family_memory
  23
+    assert_kind_of Integer, @process.family_memory
  24
+    assert @process.family_memory > 0
  25
+  end
  26
+
22 27
   def test_percent_memory
23 28
     assert_kind_of Float, @process.percent_memory
24 29
   end
@@ -26,5 +31,9 @@ def test_percent_memory
26 31
   def test_percent_cpu
27 32
     assert_kind_of Float, @process.percent_cpu
28 33
   end
  34
+
  35
+  def test_family_percent_cpu
  36
+    assert_kind_of Float, @process.family_percent_cpu
  37
+  end
29 38
 end
30 39
 
44  test/test_system_process_family.rb
... ...
@@ -0,0 +1,44 @@
  1
+require 'drb/drb'
  2
+require 'ostruct'
  3
+require File.dirname(__FILE__) + '/helper'
  4
+
  5
+class TestSystemProcessFamily < Test::Unit::TestCase
  6
+  DRUBY_URI = 'druby://localhost:38753'
  7
+
  8
+  def setup
  9
+    pid = Process.pid
  10
+    @process = System::Process.new(pid)
  11
+  end
  12
+
  13
+  def test_family_memory_usage
  14
+    pid = fork
  15
+    if pid.nil?
  16
+      my_process = System::Process.new(Process.pid)
  17
+      mem_before = my_process.memory
  18
+      # Allocate some memory.
  19
+      data = (0..10_000).reduce('') { |acc, int| acc + int.to_s }
  20
+      # Create an object which will be used to communicate with the parent
  21
+      # process.
  22
+      obj = OpenStruct.new :done => false,
  23
+        :used_memory => my_process.memory - mem_before
  24
+      DRb.start_service DRUBY_URI, obj
  25
+      at_exit { DRb.stop_service }
  26
+      sleep 0.1 while not obj.done
  27
+      exit 0
  28
+    else
  29
+      begin
  30
+        obj = DRbObject.new_with_uri DRUBY_URI
  31
+        allocated_by_child = obj.used_memory
  32
+        # Do the actual memory assertion.
  33
+        assert (@process.family_memory - @process.memory) > allocated_by_child
  34
+        # Ask the child process to die.
  35
+        obj.done = true
  36
+      rescue DRb::DRbConnError => e
  37
+        # Child's DRb service isn't running yet. Retry.
  38
+        sleep 0.1
  39
+        retry
  40
+      end
  41
+    end
  42
+  end
  43
+end
  44
+
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.