0
@@ -219,11 +219,12 @@ code {
0
<li>Easily write your own custom conditions in Ruby</li>
0
<li>Supports both poll and event based conditions</li>
0
<li>Different poll conditions can have different intervals</li>
0
- <li>Easily control non-daemonized processes</li>
0
+ <li>Integrated notification system (write your own too!)</li>
0
+ <li>Easily control non-daemonizing scripts</li>
0
- <h1>Installation (v 0.4.0)</h1>
0
+ <h1>Installation (v 0.5)</h1>
0
<p>The best way to get god is via rubygems:</p>
0
<pre>$ sudo gem install god</pre>
0
<p>You can also peruse or clone the code from <a href="http://repo.or.cz/w/god.git">http://repo.or.cz/w/god.git</a></p>
0
@@ -245,23 +246,23 @@ code {
0
<h1>Finally, a Config File that Makes Sense</h1>
0
<p>The easiest way to understand how god will make your life better is by looking at a sample config file. The following configuration file is what I use at <a href="http://site.gravatar.com/">gravatar.com</a> to keep the mongrels running:</p>
0
-<pre><code class="ruby"># file: gravatar.god
0
-# run with: god -c /path/to/gravatar.god
0
+<pre><code class="ruby"># run with: god -c /path/to/gravatar.god
0
# This is the actual config file used to keep the mongrels of
0
# gravatar.com running.
0
-RAILS_ROOT = "/var/www/gravatar2/current"
0
+RAILS_ROOT = "/Users/tom/dev/gravatar2"
0
%w{8200 8201 8202}.each do |port|
0
w.name = "gravatar2-mongrel-#{port}"
0
- w.interval = 30.seconds # default
0
- w.start = "mongrel_rails cluster::start --only #{port} \
0
- -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
0
- w.stop = "mongrel_rails cluster::stop --only #{port} \
0
- -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
0
+ w.interval = 30.seconds # default
0
+ w.start = "mongrel_rails start -c #{RAILS_ROOT} -p #{port} \
0
+ -P #{RAILS_ROOT}/log/mongrel.#{port}.pid -d"
0
+ w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
0
+ w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
0
+ w.start_grace = 10.seconds
0
+ w.restart_grace = 10.seconds
0
w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
0
w.behavior(:clean_pid_file)
0
@@ -284,6 +285,19 @@ RAILS_ROOT = "/var/www/gravatar2/current"
0
+ on.condition(:flapping) do |c|
0
+ c.to_state = [:start, :restart]
0
+ c.transition = :unmonitored
0
+ c.retry_in = 10.minutes
0
+ c.retry_within = 2.hours
0
@@ -301,18 +315,19 @@ end</code></pre>
0
<pre><code class="ruby"> God.watch do |w|
0
w.name = "gravatar2-mongrel-#{port}"
0
- w.interval = 30.seconds # default
0
- w.start = "mongrel_rails cluster::start --only #{port} \
0
- -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
0
- w.stop = "mongrel_rails cluster::stop --only #{port} \
0
- -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
0
- pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
0
+ w.interval = 30.seconds # default
0
+ w.start = "mongrel_rails start -c #{RAILS_ROOT} -p #{port} \
0
+ -P #{RAILS_ROOT}/log/mongrel.#{port}.pid -d"
0
+ w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
0
+ w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
0
+ w.start_grace = 10.seconds
0
+ w.restart_grace = 10.seconds
0
+ w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
0
-<p>A <code>watch</code> represents a single process that has concrete start, stop, and/or restart operations. You can define as many watches as you like. In the example above, I've got a Rails instance running in a Mongrel that I need to keep alive. Every watch must have a unique <code>name</code> so that it can be identified later on. The <code>interval</code> option sets the default poll interval (this can be overridden in each condition). The <code>start</code> and <code>stop</code> attributes specify the commands to start and stop the process. If no <code>restart</code> attribute is set, restart will be represented by a call to stop followed by a call to start. The optional <code>grace</code> attribute sets the amount of time following a start/stop/restart command to wait before resuming normal monitoring operations. If the process you're watches runs as a daemon, you'll need to set the <code>pid_file</code> attribute in order for the behaviors and conditions to function.</p>
0
+<p>A <code>watch</code> represents a single process that has concrete start, stop, and/or restart operations. You can define as many watches as you like. In the example above, I've got some Rails instances running in Mongrels that I need to keep alive. Every watch must have a unique <code>name</code> so that it can be identified later on. The <code>interval</code> option sets the default poll interval (this can be overridden in each condition). The <code>start</code> and <code>stop</code> attributes specify the commands to start and stop the process. If no <code>restart</code> attribute is set, restart will be represented by a call to stop followed by a call to start. The optional <code>grace</code> attribute sets the amount of time following a start/stop/restart command to wait before resuming normal monitoring operations. To be more specific, I can set just <code>start_grace</code>, <code>stop_grace</code>, and/or <code>restart_grace</code>. If the process you're watching runs as a daemon (as mine does), you'll need to set the <code>pid_file</code> attribute.</p>
0
<pre><code class="ruby"> w.behavior(:clean_pid_file)</code></pre>
0
@@ -326,7 +341,7 @@ end</code></pre>
0
<p>Watches contain conditions grouped by the action to execute should they return <code>true</code>. I start with a <code>start_if</code> block that contains a single condition. Conditions are specified by calling <code>condition</code> with an identifier, in this case
0
-<code>:process_running</code>. Each condition can specify a poll interval that will override the default watch interval. In this case, I want to check that the process is still running every 5 seconds instead of the 30 second interval that other conditions will inherit. The ability to set condition specific poll intervals makes it possible to run costly tests less often then cheap tests.</p>
0
+<code>:process_running</code>. Each condition can specify a poll interval that will override the default watch interval. In this case, I want to check that the process is still running every 5 seconds instead of the 30 second interval that other conditions will inherit. The ability to set condition specific poll intervals makes it possible to run critical tests (such as :process_running) more often than less critical tests (such as :memory_usage and :cpu_usage).</p>
0
<pre><code class="ruby"> w.restart_if do |restart|
0
restart.condition(:memory_usage) do |c|
0
@@ -350,6 +365,22 @@ end</code></pre>
0
<p>To keep an eye on CPU usage, I've employed the <code>cpu_usage</code> condition. When CPU usage for a Mongrel process is over 50% for 5 consecutive intervals, it will be restarted.</p>
0
+<pre><code class="ruby"> w.lifecycle do |on|
0
+ on.condition(:flapping) do |c|
0
+ c.to_state = [:start, :restart]
0
+ c.transition = :unmonitored
0
+ c.retry_in = 10.minutes
0
+ c.retry_within = 2.hours
0
+<p>Conditions inside a <code>lifecycle</code> section are active as long as the process is being monitored (they live across state changes).
0
+<p>The <code>:flapping</code> condition guards against the edge case wherein god rapidly starts or restarts your application. Things like server configuration changes or the unavailability of external services could make it impossible for my process to start. In that case, god will try to start my process over and over to no avail. The <code>:flapping</code> condition provides two levels of giving up on flapping processes. If I were to translate the options of the code above, it would be something like: If this watch is started or restarted five times withing 5 minutes, then unmonitor it...then after ten minutes, monitor it again to see if it was just a temporary problem; if the process is seen to be flapping five times within two hours, then give up completely.</p>
0
<p>That's it! Simple, huh?</p>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -398,7 +429,7 @@ end</code></pre>
0
<p>You can start/restart/stop/monitor/unmonitor your Watches with the same utility like so:</p>
0
-<pre>$ sudo god stop 'gravatar2-mongrel-8200'</pre>
0
+<pre>$ sudo god stop gravatar2-mongrel-8200</pre>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -416,7 +447,7 @@ end</code></pre>
0
<p>The above configuration now allows you to control the watch (and any others that are in the group) with a single command:</p>
0
-<pre>$ sudo god stop 'mongrels'</pre>
0
+<pre>$ sudo god stop mongrels</pre>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -426,13 +457,14 @@ end</code></pre>
0
<p>While the configuration syntax you saw in the previous example is very simple, it lacks the power that we need to deal with event based monitoring. In fact, the <code>start_if</code> and <code>restart_if</code> methods are really just calling out to a lower-level API. If we use the low-level API directly, we can harness the full power of god's event based lifecycle system. Let's look at another example config file.</p>
0
-<pre><code class="ruby">RAILS_ROOT = "/Users/tom/dev/git/helloworld"
0
+<pre><code class="ruby">RAILS_ROOT = "/Users/tom/dev/gravatar2"
0
w.interval = 5.seconds # default
0
- w.start = "mongrel_rails start -P ./log/mongrel.pid -c #{RAILS_ROOT}"
0
- w.stop = "mongrel_rails stop -P ./log/mongrel.pid -c #{RAILS_ROOT}"
0
+ w.start = "mongrel_rails start -c #{RAILS_ROOT} -P #{RAILS_ROOT}/log/mongrel.pid -p 3000 -d"
0
+ w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.pid"
0
+ w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.pid"
0
w.pid_file = File.join(RAILS_ROOT, "log/mongrel.pid")
0
# clean pid files before start if necessary
0
@@ -453,7 +485,7 @@ God.watch do |w|
0
on.condition(:tries) do |c|
0
@@ -477,6 +509,19 @@ God.watch do |w|
0
+ on.condition(:flapping) do |c|
0
+ c.to_state = [:start, :restart]
0
+ c.transition = :unmonitored
0
+ c.retry_in = 10.minutes
0
+ c.retry_within = 2.hours
0
@@ -514,12 +559,12 @@ end
0
on.condition(:tries) do |c|
0
-<p>The other half of this transition uses the <code>tries</code> condition to ensure that god doesn't get stuck in this state. It's possible that the process could go down while the transition is being made, in which case god would end up polling forever to see if the process is up. Here I've specified that if this condition is called three times, god should override the normal transition destination and move to the <code>start</code> state instead. If you specify a <code>transition</code> attribute on any condition, that state will be transferred to instead of the normal transfer destination.</p>
0
+<p>The other half of this transition uses the <code>tries</code> condition to ensure that god doesn't get stuck in this state. It's possible that the process could go down while the transition is being made, in which case god would end up polling forever to see if the process is up. Here I've specified that if this condition is called five times, god should override the normal transition destination and move to the <code>start</code> state instead. If you specify a <code>transition</code> attribute on any condition, that state will be transferred to instead of the normal transfer destination.</p>
0
<pre><code class="ruby"> # start if process is not running
0
w.transition(:up, :start) do |on|
0
@@ -549,11 +594,9 @@ end
0
<h1>Watching Non-Daemon Processes</h1>
0
-<p>Need to watch a script that doesn't have built in daemonization? No problem! God will daemonize and keep track of your process for you. If you don't specify a <code>pid_file</code> attribute for a watch, it will be auto-daemonized and a PID file will be stored for it in <code>/var/run/god</code>. If you'd rather have the PID file stored in a different location, you can set it in a <code>God.init</code> block at the top of your config:</p>
0
+<p>Need to watch a script that doesn't have built in daemonization? No problem! God will daemonize and keep track of your process for you. If you don't specify a <code>pid_file</code> attribute for a watch, it will be auto-daemonized and a PID file will be stored for it in <code>/var/run/god</code>. If you'd rather have the PID file stored in a different location, you can set it at the top of your config:</p>
0
-<pre><code class="ruby">God.init do |god|
0
- god.pid_file_directory = '/home/tom/pids'
0
+<pre><code class="ruby">God.pid_file_directory = '/home/tom/pids'
0
# watch with no pid_file attribute set
0
@@ -570,7 +613,7 @@ end</code></pre>
0
<pre><code class="ruby"># load in all god configs
0
God.load "/usr/local/conf/*.god"</code></pre>
0
-<p>God won't start it's monitoring operations until all configurations have been loaded.</p>
0
+<p>God won't start its monitoring operations until all configurations have been loaded.</p>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -587,7 +630,7 @@ God.load "/usr/local/conf/*.god"</code></pre>
0
<pre>$ sudo god load path/to/config.god</pre>
0
-<p>Config files that are loaded dynamically can contain anything that a normal config file contains, however, <code>God.init</code> blocks will be ignored.</p>
0
+<p>Config files that are loaded dynamically can contain anything that a normal config file contains, however, global options such as <code>God.pid_file_directory</code> blocks will be ignored (and produce a warning in the logs).</p>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -595,9 +638,78 @@ God.load "/usr/local/conf/*.god"</code></pre>
0
<p>Sifting through the god logs for statements specific to a specific Watch can be frustrating when you have many of them. You can get the realtime logs for a single Watch via the command line:</p>
0
-<pre>$ sudo god log 'local-3000'</pre>
0
+<pre>$ sudo god log local-3000</pre>
0
+<p>This will display the last 1000 lines of log for the 'local-3000' Watch and update every second with new log messages.</p>
0
+<!-- ------------------------------------------------------------------------- -->
0
+<p>God has an extensible notification framework built in that makes it easy to have notifications sent when conditions are triggered. There are three steps to enabling notifications.</p>
0
+<p>Step 1: Set the options for the notification subsystem(s) that you'll be using. Let's look at how to setup email notifications.</p>
0
+<pre><code class="ruby">God::Contacts::Email.message_settings = {
0
+ :from => 'god@example.com'
0
+God::Contacts::Email.server_settings = {
0
+ :address => "smtp.example.com",
0
+ :domain => "example.com",
0
+ :authentication => :plain,
0
+ :password => "s3kr3ts"
0
+<p>Step 2: Configure some contacts.</p>
0
+<pre><code class="ruby">God.contact(:email) do |c|
0
+ c.email = 'tom@example.com'
0
+God.contact(:email) do |c|
0
+ c.email = 'vanpelt@example.com'
0
+ c.group = 'developers'
0
+God.contact(:email) do |c|
0
+ c.email = 'kevin@example.com'
0
+ c.group = 'developers'
0
+<p>Step 3: Attach to a condition:</p>
0
+<pre><code class="ruby"> w.transition(:up, :start) do |on|
0
+ on.condition(:process_exits) do |c|
0
+<p>There are two ways to specify that a notification should be sent. The first, easier way is shown above. Every condition can take an optional <code>notify</code> attribute that specifies which contacts should be notified when the condition is triggered. The value can be a contact name or contact group *or* an array of contact names and/or contact groups.</p>
0
+<pre><code class="ruby"> w.transition(:up, :start) do |on|
0
+ on.condition(:process_exits) do |c|
0
+ c.notify = {:contacts => ['tom', 'developers'], :priority => 1, :category => 'product'}
0
+</p>The second way allows you to specify the <code>priority</code> and <code>category</code> in addition to the contacts. The extra attributes can be arbitrary integers or strings and will be passed as-is to the notification subsystem.</p>
0
+<p>The above notification will arrive as an email similar to the following.</p>
0
-<p>This will display the last 100 lines of log for the 'local-3000' Watch and update every second with new log messages.</p>
0
+<pre><code>From: god <god@example.com>
0
+To: tom <tom@example.com>
0
+Subject: [god] mongrel-8600 [trigger] process exited (ProcessExits)
0
+Message: mongre-8600 [trigger] process exited (ProcessExits)
0
+Host: candymountain.example.com
0
+Category: product</code></pre>
0
<!-- ------------------------------------------------------------------------- -->
0
@@ -605,34 +717,6 @@ God.load "/usr/local/conf/*.god"</code></pre>
0
<p>God was designed from the start to allow you to easily write your own custom conditions, making it simple to add tests that are application specific.</p>
0
-<pre><code class="ruby">module God
0
- class ProcessRunning < PollCondition
0
- attr_accessor :running
0
- valid &= complain("You must specify the 'pid_file' attribute \
0
- on the Watch for :process_running") if self.watch.pid_file.nil?
0
- valid &= complain("You must specify the 'running' attribute \
0
- for :process_running") if self.running.nil?
0
- return !self.running unless File.exist?(self.watch.pid_file)
0
- pid = File.read(self.watch.pid_file).strip
0
- active = System::Process.new(pid).exists?
0
- (self.running && active) || (!self.running && !active)
0
<p>Brought to you by <a href="http://rubyisawesome.com/">Ruby is Awesome</a></p>
Comments
No one has commented yet.