Permalink
Browse files

check reporting works

  • Loading branch information...
1 parent a1dff7e commit 7143268dc3b0677544a31c15b192c328d0a9576b @danryan committed Aug 10, 2012
View
4 Gemfile
@@ -9,12 +9,16 @@ gem 'simple_form'
gem 'cabin'
# gem 'faraday'
gem 'rest-client'
+gem 'state_machine'
# Sidekiq and related dependencies
gem 'sidekiq'
gem 'slim', :require => nil
gem 'sinatra', :require => nil
+# Handler dependencies
+gem 'tinder'
+gem 'redphone'
group :assets do
gem 'sass-rails', '~> 3.2.3'
View
30 Gemfile.lock
@@ -59,13 +59,18 @@ GEM
mail (~> 2.2)
rspec (~> 2.0)
erubis (2.7.0)
+ eventmachine (0.12.10)
execjs (1.4.0)
multi_json (~> 1.0)
factory_girl (4.0.0)
activesupport (>= 3.0.0)
factory_girl_rails (4.0.0)
factory_girl (~> 4.0.0)
railties (>= 3.0.0)
+ faraday (0.8.1)
+ multipart-post (~> 1.1)
+ faraday_middleware (0.8.8)
+ faraday (>= 0.7.4, < 0.9)
ffi (1.1.5)
forgery (0.5.0)
formatador (0.2.3)
@@ -89,7 +94,9 @@ GEM
activesupport (~> 3.0)
haml (~> 3.0)
railties (~> 3.0)
+ hashie (1.2.0)
hike (1.2.1)
+ http_parser.rb (0.5.3)
i18n (0.6.0)
journey (1.0.4)
jquery-rails (2.0.2)
@@ -112,6 +119,7 @@ GEM
treetop (~> 1.4.8)
mime-types (1.19)
multi_json (1.3.6)
+ multipart-post (1.1.5)
nokogiri (1.5.5)
pg (0.14.0)
polyglot (0.3.3)
@@ -151,6 +159,8 @@ GEM
redis (3.0.1)
redis-namespace (1.2.1)
redis (~> 3.0.0)
+ redphone (0.0.6)
+ json
rest-client (1.6.7)
mime-types (>= 1.16)
rspec (2.11.0)
@@ -188,6 +198,7 @@ GEM
simple_form (2.0.2)
actionpack (~> 3.0)
activemodel (~> 3.0)
+ simple_oauth (0.1.9)
sinatra (1.3.2)
rack (~> 1.3, >= 1.3.6)
rack-protection (~> 1.2)
@@ -200,16 +211,32 @@ GEM
hike (~> 1.2)
rack (~> 1.0)
tilt (~> 1.1, != 1.3.0)
+ state_machine (1.1.2)
tach (0.0.8)
formatador (>= 0.0.16)
temple (0.4.0)
thor (0.15.4)
tilt (1.3.3)
timecop (0.4.5)
timers (1.0.1)
+ tinder (1.9.0)
+ activesupport (>= 2.3, < 4)
+ eventmachine (>= 0.12.0, < 2)
+ faraday (~> 0.8)
+ faraday_middleware (~> 0.8)
+ hashie (~> 1.0)
+ json (~> 1.6)
+ mime-types (~> 1.16)
+ multi_json (~> 1.0)
+ multipart-post (~> 1.1)
+ twitter-stream (~> 0.1)
treetop (1.4.10)
polyglot
polyglot (>= 0.3.1)
+ twitter-stream (0.1.16)
+ eventmachine (>= 0.12.8)
+ http_parser.rb (~> 0.5.1)
+ simple_oauth (~> 0.1.4)
tzinfo (0.3.33)
uglifier (1.2.7)
execjs (>= 0.3.0)
@@ -246,6 +273,7 @@ DEPENDENCIES
rack-test (>= 0.6.0)
rails (= 3.2.7)
rb-fsevent (>= 0.9.1)
+ redphone
rest-client
rspec-rails (>= 2.10.0)
sass-rails (~> 3.2.3)
@@ -255,7 +283,9 @@ DEPENDENCIES
sinatra
slim
spork (>= 1.0.0rc2)
+ state_machine
tach (>= 0.0.8)
timecop (>= 0.3.5)
+ tinder
uglifier (>= 1.0.3)
unicorn
View
3 README.md
@@ -0,0 +1,3 @@
+## TODO
+
+Schedule all checks on startup
View
261 README.rdoc
@@ -1,261 +0,0 @@
-== Welcome to Rails
-
-Rails is a web-application framework that includes everything needed to create
-database-backed web applications according to the Model-View-Control pattern.
-
-This pattern splits the view (also called the presentation) into "dumb"
-templates that are primarily responsible for inserting pre-built data in between
-HTML tags. The model contains the "smart" domain objects (such as Account,
-Product, Person, Post) that holds all the business logic and knows how to
-persist themselves to a database. The controller handles the incoming requests
-(such as Save New Account, Update Product, Show Post) by manipulating the model
-and directing data to the view.
-
-In Rails, the model is handled by what's called an object-relational mapping
-layer entitled Active Record. This layer allows you to present the data from
-database rows as objects and embellish these data objects with business logic
-methods. You can read more about Active Record in
-link:files/vendor/rails/activerecord/README.html.
-
-The controller and view are handled by the Action Pack, which handles both
-layers by its two parts: Action View and Action Controller. These two layers
-are bundled in a single package due to their heavy interdependence. This is
-unlike the relationship between the Active Record and Action Pack that is much
-more separate. Each of these packages can be used independently outside of
-Rails. You can read more about Action Pack in
-link:files/vendor/rails/actionpack/README.html.
-
-
-== Getting Started
-
-1. At the command prompt, create a new Rails application:
- <tt>rails new myapp</tt> (where <tt>myapp</tt> is the application name)
-
-2. Change directory to <tt>myapp</tt> and start the web server:
- <tt>cd myapp; rails server</tt> (run with --help for options)
-
-3. Go to http://localhost:3000/ and you'll see:
- "Welcome aboard: You're riding Ruby on Rails!"
-
-4. Follow the guidelines to start developing your application. You can find
-the following resources handy:
-
-* The Getting Started Guide: http://guides.rubyonrails.org/getting_started.html
-* Ruby on Rails Tutorial Book: http://www.railstutorial.org/
-
-
-== Debugging Rails
-
-Sometimes your application goes wrong. Fortunately there are a lot of tools that
-will help you debug it and get it back on the rails.
-
-First area to check is the application log files. Have "tail -f" commands
-running on the server.log and development.log. Rails will automatically display
-debugging and runtime information to these files. Debugging info will also be
-shown in the browser on requests from 127.0.0.1.
-
-You can also log your own messages directly into the log file from your code
-using the Ruby logger class from inside your controllers. Example:
-
- class WeblogController < ActionController::Base
- def destroy
- @weblog = Weblog.find(params[:id])
- @weblog.destroy
- logger.info("#{Time.now} Destroyed Weblog ID ##{@weblog.id}!")
- end
- end
-
-The result will be a message in your log file along the lines of:
-
- Mon Oct 08 14:22:29 +1000 2007 Destroyed Weblog ID #1!
-
-More information on how to use the logger is at http://www.ruby-doc.org/core/
-
-Also, Ruby documentation can be found at http://www.ruby-lang.org/. There are
-several books available online as well:
-
-* Programming Ruby: http://www.ruby-doc.org/docs/ProgrammingRuby/ (Pickaxe)
-* Learn to Program: http://pine.fm/LearnToProgram/ (a beginners guide)
-
-These two books will bring you up to speed on the Ruby language and also on
-programming in general.
-
-
-== Debugger
-
-Debugger support is available through the debugger command when you start your
-Mongrel or WEBrick server with --debugger. This means that you can break out of
-execution at any point in the code, investigate and change the model, and then,
-resume execution! You need to install ruby-debug to run the server in debugging
-mode. With gems, use <tt>sudo gem install ruby-debug</tt>. Example:
-
- class WeblogController < ActionController::Base
- def index
- @posts = Post.all
- debugger
- end
- end
-
-So the controller will accept the action, run the first line, then present you
-with a IRB prompt in the server window. Here you can do things like:
-
- >> @posts.inspect
- => "[#<Post:0x14a6be8
- @attributes={"title"=>nil, "body"=>nil, "id"=>"1"}>,
- #<Post:0x14a6620
- @attributes={"title"=>"Rails", "body"=>"Only ten..", "id"=>"2"}>]"
- >> @posts.first.title = "hello from a debugger"
- => "hello from a debugger"
-
-...and even better, you can examine how your runtime objects actually work:
-
- >> f = @posts.first
- => #<Post:0x13630c4 @attributes={"title"=>nil, "body"=>nil, "id"=>"1"}>
- >> f.
- Display all 152 possibilities? (y or n)
-
-Finally, when you're ready to resume execution, you can enter "cont".
-
-
-== Console
-
-The console is a Ruby shell, which allows you to interact with your
-application's domain model. Here you'll have all parts of the application
-configured, just like it is when the application is running. You can inspect
-domain models, change values, and save to the database. Starting the script
-without arguments will launch it in the development environment.
-
-To start the console, run <tt>rails console</tt> from the application
-directory.
-
-Options:
-
-* Passing the <tt>-s, --sandbox</tt> argument will rollback any modifications
- made to the database.
-* Passing an environment name as an argument will load the corresponding
- environment. Example: <tt>rails console production</tt>.
-
-To reload your controllers and models after launching the console run
-<tt>reload!</tt>
-
-More information about irb can be found at:
-link:http://www.rubycentral.org/pickaxe/irb.html
-
-
-== dbconsole
-
-You can go to the command line of your database directly through <tt>rails
-dbconsole</tt>. You would be connected to the database with the credentials
-defined in database.yml. Starting the script without arguments will connect you
-to the development database. Passing an argument will connect you to a different
-database, like <tt>rails dbconsole production</tt>. Currently works for MySQL,
-PostgreSQL and SQLite 3.
-
-== Description of Contents
-
-The default directory structure of a generated Ruby on Rails application:
-
- |-- app
- | |-- assets
- | |-- images
- | |-- javascripts
- | `-- stylesheets
- | |-- controllers
- | |-- helpers
- | |-- mailers
- | |-- models
- | `-- views
- | `-- layouts
- |-- config
- | |-- environments
- | |-- initializers
- | `-- locales
- |-- db
- |-- doc
- |-- lib
- | `-- tasks
- |-- log
- |-- public
- |-- script
- |-- test
- | |-- fixtures
- | |-- functional
- | |-- integration
- | |-- performance
- | `-- unit
- |-- tmp
- | |-- cache
- | |-- pids
- | |-- sessions
- | `-- sockets
- `-- vendor
- |-- assets
- `-- stylesheets
- `-- plugins
-
-app
- Holds all the code that's specific to this particular application.
-
-app/assets
- Contains subdirectories for images, stylesheets, and JavaScript files.
-
-app/controllers
- Holds controllers that should be named like weblogs_controller.rb for
- automated URL mapping. All controllers should descend from
- ApplicationController which itself descends from ActionController::Base.
-
-app/models
- Holds models that should be named like post.rb. Models descend from
- ActiveRecord::Base by default.
-
-app/views
- Holds the template files for the view that should be named like
- weblogs/index.html.erb for the WeblogsController#index action. All views use
- eRuby syntax by default.
-
-app/views/layouts
- Holds the template files for layouts to be used with views. This models the
- common header/footer method of wrapping views. In your views, define a layout
- using the <tt>layout :default</tt> and create a file named default.html.erb.
- Inside default.html.erb, call <% yield %> to render the view using this
- layout.
-
-app/helpers
- Holds view helpers that should be named like weblogs_helper.rb. These are
- generated for you automatically when using generators for controllers.
- Helpers can be used to wrap functionality for your views into methods.
-
-config
- Configuration files for the Rails environment, the routing map, the database,
- and other dependencies.
-
-db
- Contains the database schema in schema.rb. db/migrate contains all the
- sequence of Migrations for your schema.
-
-doc
- This directory is where your application documentation will be stored when
- generated using <tt>rake doc:app</tt>
-
-lib
- Application specific libraries. Basically, any kind of custom code that
- doesn't belong under controllers, models, or helpers. This directory is in
- the load path.
-
-public
- The directory available for the web server. Also contains the dispatchers and the
- default HTML files. This should be set as the DOCUMENT_ROOT of your web
- server.
-
-script
- Helper scripts for automation and generation.
-
-test
- Unit and functional tests along with fixtures. When using the rails generate
- command, template test files will be generated for you and placed in this
- directory.
-
-vendor
- External libraries that the application depends on. Also includes the plugins
- subdirectory. If the app has frozen rails, those gems also go here, under
- vendor/rails/. This directory is in the load path.
View
62 app/models/check.rb
@@ -1,30 +1,62 @@
class Check < ActiveRecord::Base
attr_accessible :critical, :interval, :metric, :repeat,
- :resolve, :warning, :duration
-
+ :resolve, :warning, :duration, :handlers
+ attr_accessor :value, :previous_state, :current_state
+
+ serialize :handlers
+
validates :metric, :presence => true
+
+ after_create :schedule_first_check
-
- def metric_url
- "#{Rorschach.graphite_url}/render/?target=#{metric}&format=json&from=-#{duration}s"
- end
-
- def headers
+ def get_value
+ metric_url = "#{Rorschach::Config.graphite_url}/render/?target=#{metric}&format=json&from=-#{duration}s"
headers = {
'Accept' => 'application/json',
'Content-Type' => 'application/json'
}
- if Rorschach.graphite_auth
- headers['Authorization'] = "Basic #{Base64.encode64(Rorschach.graphite_auth)}"
+ if Rorschach::Config.graphite_auth
+ headers['Authorization'] = "Basic #{Base64.encode64(Rorschach::Config.graphite_auth)}"
end
- headers
- end
-
- def get_value
raw_data = RestClient.get(metric_url, headers)
data = JSON.parse(raw_data)
points = data.first["datapoints"].map{ |v,_| v }.compact
- value = ((points.reduce { |a, b| a + b }) / points.size.to_f)
+ self.value = ((points.reduce { |a, b| a + b }) / points.size.to_f).to_f
+ end
+
+ def schedule_first_check
+ CheckWorker.perform_async(id)
end
+ def status_changed?
+ last_status != current_status
+ end
+
+ def notify
+ results = { check_id: id, metric: metric, value: last_value, state: state }
+ PagerDutyWorker.perform_async(results) if handlers.include?('pagerduty')
+ CampfireWorker.perform_async(results) if handlers.include?('campfire')
+ EmailWorker.perform_async(results) if handlers.include?('email')
+ end
+
+ state_machine :initial => :ok do
+ after_transition any => any, :do => :notify
+
+ state :ok
+ state :warning
+ state :critical
+
+ event :set_warning do
+ transition [ :ok, :critical ] => :warning
+ end
+
+ event :set_critical do
+ transition [ :ok, :warning ] => :critical
+ end
+
+ event :set_ok do
+ transition [ :warning, :critical ] => :ok
+ end
+
+ end
end
View
9 app/models/check_result.rb
@@ -1,9 +0,0 @@
-class CheckResult
-
- attr_accessor :status
-
- def initialize(status)
- @status = status
- end
-
-end
View
3 app/models/handler/campfire.rb
@@ -0,0 +1,3 @@
+class Handler::Campfire
+
+end
View
3 app/models/handler/email.rb
@@ -0,0 +1,3 @@
+class Handler::Email
+
+end
View
3 app/models/handler/http.rb
@@ -0,0 +1,3 @@
+class Handler::HTTP
+
+end
View
3 app/models/handler/pager_duty.rb
@@ -0,0 +1,3 @@
+class Handler::PagerDuty
+
+end
View
2 app/views/checks/_form.html.haml
@@ -3,8 +3,6 @@
= f.input :metric
= f.input :warning
= f.input :critical
- = f.input :resolve
- = f.input :repeat
= f.input :interval
.form-actions
View
20 app/views/checks/index.html.haml
@@ -9,10 +9,8 @@
%th Metric
%th Warning
%th Critical
- %th Resolve
- %th Repeat
%th Interval
- %th
+ %th State
%th
%th
@@ -21,12 +19,18 @@
%td= check.metric
%td= check.warning
%td= check.critical
- %td= check.resolve
- %td= check.repeat
%td= check.interval
- %td= link_to 'Show', check
- %td= link_to 'Edit', edit_check_path(check)
- %td= link_to 'Destroy', check, :confirm => 'Are you sure?', :method => :delete
+ %td
+ - case check.state
+ - when 'ok'
+ %span.label= check.state.upcase
+ - when 'warning'
+ %span.label.label-warning= check.state.upcase
+ - when 'critical'
+ %span.label.label-important= check.state.upcase
+
+ %td= link_to 'Edit', edit_check_path(check), class: 'btn btn-info'
+ %td= link_to 'Delete', check, confirm: 'Are you sure?', method: :delete, class: 'btn btn-danger'
%br
View
6 app/views/checks/show.html.haml
@@ -10,12 +10,6 @@
%b Critical:
= @check.critical
%p
- %b Resolve:
- = @check.resolve
-%p
- %b Repeat:
- = @check.repeat
-%p
%b Interval:
= @check.interval
View
19 app/workers/campfire_worker.rb
@@ -0,0 +1,19 @@
+require 'tinder'
+
+class CampfireWorker
+ include Sidekiq::Worker
+
+ def perform(results)
+ check = Check.find(results['check_id'])
+ incident_key = results['metric']
+ state = results['state'].to_s.upcase
+ value = results['value']
+
+ campfire = Tinder::Campfire.new(Rorschach::Config.campfire_account, ssl: true, token: Rorschach::Config.campfire_token)
+
+ room = campfire.find_room_by_id(Rorschach::Config.campfire_room.to_i)
+
+ room.speak("#{incident_key} - #{state.to_s.upcase}: value of #{value}")
+ end
+
+end
View
55 app/workers/check_worker.rb
@@ -1,31 +1,48 @@
class CheckWorker
include Sidekiq::Worker
- def perform(options={})
- metric = options['metric']
- value = options['value'].to_f
- warning = options['warning'].to_f
- critical = options['critical'].to_f
+ def perform(id)
+ check = Check.find(id)
+
+ metric = check.metric
+ value = check.get_value
+ warning = check.warning
+ critical = check.critical
+ last_state = check.state
if critical > warning
- if value >= critical
- Rails.logger.error "CRITICAL", :metric => metric, :status => "CRITICAL", :value => value
- elsif value >= warning
- Rails.logger.warn "WARNING", :metric => metric, :status => "WARNING", :value => value
- else
- Rails.logger.info "OK", :metric => metric, :status => "OK", :value => value
+ current_state = case
+ when value >= critical
+ 'critical'
+ when value >= warning
+ 'warning'
+ else
+ 'ok'
end
else
- if value <= critical
- Rails.logger.error "CRITICAL", :metric => metric, :status => "CRITICAL", :value => value
-
- elsif value <= warning
- Rails.logger.warn "WARNING", :metric => metric, :status => "WARNING", :value => value
- else
- Rails.logger.info "OK", :metric => metric, :status => "OK", :value => value
+ current_state = case
+ when value <= critical
+ 'critical'
+ when value <= warning
+ 'warning'
+ else
+ 'ok'
end
end
- self.class.perform_in(options[:interval], options)
+ check.last_value = value
+ check.save
+
+ if current_state != last_state
+ Rails.logger.info current_state: current_state, last_state: last_state
+ check.send("set_#{current_state}")
+ else
+ Rails.logger.info "nothing to see here"
+ end
+
+ Rails.logger.debug check_id: check.id, metric: check.metric, value: value, state: current_state
+
+ self.class.perform_in(check.interval, check.id)
+
end
end
View
7 app/workers/email_worker.rb
@@ -0,0 +1,7 @@
+class EmailWorker
+ include Sidekiq::Worker
+
+ def perform(id)
+
+ end
+end
View
7 app/workers/pager_duty_worker.rb
@@ -0,0 +1,7 @@
+class PagerDutyWorker
+ include Sidekiq::Worker
+
+ def perform(id)
+
+ end
+end
View
8 config/application.rb
@@ -16,15 +16,7 @@
end
module Rorschach
- def self.env!(k)
- ENV[k] || raise("missing key #{k}")
- end
- # def self.deploy; env!("DEPLOY"); end
- def self.graphite_url; env!("GRAPHITE_URL"); end
- def self.graphite_auth; ENV['GRAPHITE_AUTH']; end
- # def self.force_https?; env!("FORCE_HTTPS") == "true"; end
- # def self.api_key; env!("API_KEY"); end
class Application < Rails::Application
config.generators do |g|
View
11 db/migrate/20120809200840_create_checks.rb
@@ -4,11 +4,12 @@ def change
t.string :metric
t.decimal :warning
t.decimal :critical
- t.integer :duration
- t.boolean :resolve
- t.boolean :repeat
- t.integer :interval
-
+ t.integer :duration, default: 60
+ t.boolean :repeat, default: false
+ t.integer :interval, default: 60
+ t.text :handlers, default: ['campfire'].to_json
+ t.string :state
+ t.string :last_value
t.timestamps
end
end
View
14 db/schema.rb
@@ -17,12 +17,14 @@
t.string "metric"
t.decimal "warning"
t.decimal "critical"
- t.integer "duration"
- t.boolean "resolve"
- t.boolean "repeat"
- t.integer "interval"
- t.datetime "created_at", :null => false
- t.datetime "updated_at", :null => false
+ t.integer "duration", :default => 60
+ t.boolean "repeat", :default => false
+ t.integer "interval", :default => 60
+ t.text "handlers", :default => "[\"campfire\"]"
+ t.string "state"
+ t.string "last_value"
+ t.datetime "created_at", :null => false
+ t.datetime "updated_at", :null => false
end
end
View
13 lib/rorschach/config.rb
@@ -1,5 +1,4 @@
-ENV["TZ"] = "UTC"
-
+# blatantly stolen with love from https://github.com/heroku/umpire
module Rorschach
module Config
def self.env!(k)
@@ -9,8 +8,16 @@ def self.env!(k)
# def self.deploy; env!("DEPLOY"); end
def self.graphite_url; env!("GRAPHITE_URL"); end
def self.graphite_auth; ENV['GRAPHITE_AUTH']; end
+
+ # Campfire variables
+ def self.campfire_account; env!('CAMPFIRE_ACCOUNT'); end
+ def self.campfire_token; env!('CAMPFIRE_TOKEN'); end
+ def self.campfire_room; env!('CAMPFIRE_ROOM'); end
+
+ # PagerDuty variables
+ def self.pagerduty_api_key; env!('PAGERDUTY_API_KEY'); end
+
# def self.force_https?; env!("FORCE_HTTPS") == "true"; end
# def self.api_key; env!("API_KEY"); end
-
end
end

0 comments on commit 7143268

Please sign in to comment.