unicorn.god

defunkt 12 Oct 2009

Some people have been asking for our Unicorn god config.

Here it is:

# http://unicorn.bogomips.org/SIGNALS.html

rails_env = ENV[‘RAILS_ENV’] || ‘production’
rails_root = ENV[‘RAILS_ROOT’] || “/data/github/current”

God.watch do |w|
w.name = “unicorn”
w.interval = 30.seconds # default

# unicorn needs to be run from the rails root
w.start = “cd #{rails_root} && /usr/local/bin/unicorn_rails -c #{rails_root}/config/unicorn.rb -E #{rails_env} -D”

# QUIT gracefully shuts down workers
w.stop = “kill -QUIT `cat #{rails_root}/tmp/pids/unicorn.pid`”

# USR2 causes the master to re-create itself and spawn a new worker pool
w.restart = “kill -USR2 `cat #{rails_root}/tmp/pids/unicorn.pid`”

w.start_grace = 10.seconds
w.restart_grace = 10.seconds
w.pid_file = “#{rails_root}/tmp/pids/unicorn.pid”

w.uid = ‘git’
w.gid = ‘git’

w.behavior(:clean_pid_file)

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end

w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 300.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end

restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
end
end

# lifecycle
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
end

That’s for starting and stopping the master. It’s important to note that god only knows about the master – not the workers. The memory limit condition, then, only applies to the master (and is probably never hit).

To watch the workers we use a cute hack mojombo came up with (though he promises first class support in future versions of code): we start a thread and periodically check the memory usage of workers. If a worker is gobbling up more than 300mb of RSS, we send it a QUIT. The QUIT tells it to die once it finishes processing the current request. Once that happens the master will spawn a new worker – we should hardly notice.

# This will ride alongside god and kill any rogue memory-greedy
# processes. Their sacrifice is for the greater good.

unicorn_worker_memory_limit = 300_000

Thread.new do
loop do
begin
# unicorn workers
#
# ps output line format:
# 31580 275444 unicorn_rails worker¹⁵ -c /data/github/current/config/unicorn.rb -E production -D
# pid ram command

lines = `ps -e -www -o pid,rss,command | grep ‘[u]nicorn_rails worker’`.split(“\n”)
lines.each do |line|
parts = line.split(’ ‘)
if parts¹.to_i > unicorn_worker_memory_limit
# tell the worker to die after it finishes serving its request
::Process.kill(’QUIT’, parts⁰.to_i)
end
end
rescue Object
# don’t die ever once we’ve tested this
nil
end

sleep 30
end
end

That’s it! Don’t forget the Unicorn Signals page when working with Unicorn.

unicorn.god

Comments

The GitHub Bloggers