Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Missing proc_exit event #89

Open
nearapogee opened this Issue · 1 comment

2 participants

@nearapogee

I have used a different version of this God config file in the past an it worked like a charm. However, now will start unicorn and check memory usage but does not seem to notice when the process exists.

I am using god to monitor several other resque processes on the same server using events and things seem to be working well. I am running Ubuntu 10.04.4 LTS. Linux 2.6.32-38-generic-pae #83-Ubuntu SMP Wed Jan 4 12:11:13 UTC 2012 i686 GNU/Linux

Where am I going wrong? (Config files below.)

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -D --log-level debug                                            
[sudo] password for deploy:                                                                                                             
I [2012-03-05 08:07:20]  INFO: Syslog enabled.                                                                                          
I [2012-03-05 08:07:20]  INFO: Using pid file directory: /var/run/god                                                                   
I [2012-03-05 08:07:20]  INFO: Started on drbunix:///tmp/god.17165.sock                                                                 
I [2012-03-05 08:07:32]  INFO: unicorn-master Loaded config                                                                             
I [2012-03-05 08:07:32]  INFO: unicorn-master move 'unmonitored' to 'init'                                                              
D [2012-03-05 08:07:32] DEBUG: driver schedule #<God::Conditions::ProcessRunning:0xa2b9148> in 0 seconds                                
I [2012-03-05 08:07:32]  INFO: unicorn-master moved 'unmonitored' to 'init'                                                             
I [2012-03-05 08:07:32]  INFO: unicorn-master [trigger] process is not running (ProcessRunning)                                         
D [2012-03-05 08:07:32] DEBUG: unicorn-master ProcessRunning [false] {true=>:up, false=>:start}                                         
I [2012-03-05 08:07:32]  INFO: unicorn-master move 'init' to 'start'                                                                    
I [2012-03-05 08:07:32]  INFO: unicorn-master before_start: no pid file to delete (CleanPidFile)                                        
I [2012-03-05 08:07:32]  INFO: unicorn-master start: /home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb                                                                                                             
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::ProcessRunning:0xa2b8f7c> in 0 seconds                                
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::Tries:0xa2b8f04> in 0 seconds                                         
I [2012-03-05 08:07:35]  INFO: unicorn-master moved 'init' to 'start'                                                                   
I [2012-03-05 08:07:35]  INFO: unicorn-master [trigger] process is running (ProcessRunning)                                             
D [2012-03-05 08:07:35] DEBUG: unicorn-master ProcessRunning [true] {true=>:up}                                                         
I [2012-03-05 08:07:35]  INFO: unicorn-master move 'start' to 'up'                                                                      
I [2012-03-05 08:07:35]  INFO: unicorn-master registered 'proc_exit' event for pid 3155                                                 
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 0 seconds                                   
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 0 seconds                                      
I [2012-03-05 08:07:35]  INFO: unicorn-master moved 'start' to 'up'                                                                     
I [2012-03-05 08:07:35]  INFO: unicorn-master [ok] memory within bounds [7468kb] (MemoryUsage)                                          
D [2012-03-05 08:07:35] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}                                                      
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 30 seconds                                  
I [2012-03-05 08:07:35]  INFO: unicorn-master [ok] cpu within bounds [0.0%%] (CpuUsage)                                                 
D [2012-03-05 08:07:35] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}                                                         
D [2012-03-05 08:07:35] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 30 seconds 

Then kill 3155

I [2012-03-05 08:16:05]  INFO: unicorn-master [ok] memory within bounds [7476kb, 7476kb, 0kb] (MemoryUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule #<God::Conditions::MemoryUsage:0xa2b89b4> in 30 seconds
I [2012-03-05 08:16:05]  INFO: unicorn-master [ok] cpu within bounds [0.0022087244616234127%%, 0.002071465561885034%%, 0%%] (CpuUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule #<God::Conditions::CpuUsage:0xa2b8860> in 30 seconds

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god status
unicorn:
  unicorn-master: up

As you can see god still thinks the process is up, even though it shows 0 for memory usage. The pid file does exist in the specified location when the process is running. Below is the out put from -V and check, along with the God and Unicorn config files.

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -V
Version: 0.12.1
Polls: enabled
Events: netlink

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god check
using event system: netlink
starting event handler
forking off new process
forked process with pid = 3227
killing process
[ok] process exit event received

God config file.

God.watch do |w|
  w.name = "unicorn-master"
  w.group = "unicorn"

  w.dir = "/home/deploy/apps/project/current"
  w.interval = 30.seconds
  w.env = { 'RAILS_ENV' => 'production' }
  w.start = "/home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb"
  w.pid_file = "/home/deploy/apps/project/shared/pids/unicorn.pid"

  w.uid = "deploy"
  w.gid = "deploy"

  w.behavior(:clean_pid_file)

  # With events
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) { |c| c.running = true }
  end
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) { |c| c.running = true }
    on.condition(:tries) { |c| c.times = 5; c.transition = :start }
  end
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) { |c| c.above = 24.megabytes; c.times = [2,3] }
    on.condition(:cpu_usage) { |c| c.above = 8.percent; c.times = [2,3] }
  end

  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minutes
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Unicorn (ver 4.2.0) config file

worker_processes 2
working_directory "/home/deploy/apps/project/current"
listen "/tmp/project.socket", :backlog => 64
timeout 30
user 'deploy', 'deploy'
pid "/home/deploy/apps/project/shared/pids/unicorn.pid"
stderr_path "/home/deploy/apps/project/shared/log/unicorn.stderr.log"
stdout_path "/home/deploy/apps/project/shared/log/unicorn.stdout.log"
before_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
  old_pid = "#{server.config[:pid]}.oldbin"
  if old_pid != server.pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end
  sleep 1 # Prevent signals from being lost when busy
end
after_fork do |server, worker|
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end
@scottbartell

Did you ever figure this out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.