V0.2.6 - added response tracking and mole_excludes

HISTORY

@@ -22,4 +22,15 @@
     validation problems in mongoDB. Thanks Rafael!
 
 0.2.4
-* Breaking off action_mailer dependencies. Now leveraging Pony for sending out email alerts.
+* Breaking off action_mailer dependencies. Now leveraging Pony for sending out email alerts.
+
+0.2.5
+* Added support for excluding certain params from request or session.
+  See the params_excludes and session_excludes options.
+
+0.2.6
+* Added logging for response information such as status, headers and body. Body is excluded by default
+* Added options for excluding certain mole information for being logged see mole_excludes option.
+* Eliminated unwise defaults. From now, the option :app_name must be specified when initializing.
+  The option :db_name must also be present if the mongo store is used.
+  Keep in mind for rackamole to work with wackamole your db name must follow: mole_{app_name}_{environment}_mdb.

README.rdoc

@@ -1,53 +1,53 @@
-== RackAMole
+== RackAmole
 
   Observe your web applications in the wild!
 
-== DESCRIPTION:
+== DESCRIPTION
 
-The MOle is a rack application that monitors user interactions with your web site. We are not 
-talking about counting page hits here. The MOle tracks all the information available to capture
-the essence of a user interaction with your application. Using the MOle, you are able to see
-which feature is a hit or a bust. As an added bonus, the MOle also track performance and exceptions 
-that might have escaped your test suites or alpha env. To boot your managers will love you for it! 
+  The MOle is a rack application that monitors user interactions with your web site. We are not 
+  talking about counting page hits here. The MOle tracks all the information available to capture
+  the essence of a user interaction with your application. Using the MOle, you are able to see
+  which feature is a hit or a bust. As an added bonus, the MOle also track performance and exceptions 
+  that might have escaped your test suites or alpha env. To boot your managers will love you for it! 
 
-Whether you are releasing a new application or improving on an old one, it is always a good thing 
-to know if anyone is using your application and if they are, how they are using it. 
-What features are your users most fond of and which features find their way into the abyss? 
-You will be able to rapidly assess whether or not your application is a hit and if
-your coolest features are thought as such by your users. You will be able to elegantly record user
-interactions and leverage these findings for the next iteration of your application. 
+  Whether you are releasing a new application or improving on an old one, it is always a good thing 
+  to know if anyone is using your application and if they are, how they are using it. 
+  What features are your users most fond of and which features find their way into the abyss? 
+  You will be able to rapidly assess whether or not your application is a hit and if
+  your coolest features are thought as such by your users. You will be able to elegantly record user
+  interactions and leverage these findings for the next iteration of your application. 
 
 == PROJECT INFORMATION
 
-* Developer:  Fernand Galiana
-* Blog:       http://www.liquidrail.com
-* Site:       http://rackamole.com
-* Twitter:    http://twitter.com/rackamole
-* Forum:      http://groups.google.com/group/rackamole
-* Git:        git://github.com/derailed/rackamole.git
+  Developer:  Fernand Galiana
+  Blog:       http://www.liquidrail.com
+  Site:       http://rackamole.com
+  Twitter:    http://twitter.com/rackamole
+  Forum:      http://groups.google.com/group/rackamole
+  Git:        git://github.com/derailed/rackamole.git
 
-== FEATURES:
+== FEATURES
 
-* Monitors any rack based framework such as Rails and Sinatra
-* Captures the essence of the request as well as user information
-* Tracks performance issues based on your latency threshold
-* Tracks exceptions that might occurred during a request
+  Monitors any rack based framework such as Rails and Sinatra
+  Captures the essence of the request as well as user information
+  Tracks performance issues based on your latency threshold
+  Tracks exceptions that might occurred during a request
 
-== REQUIREMENTS:
+== REQUIREMENTS
 
-* Logging
-* Hitimes
-* MongoDb + mongo ruby adapter
-* Twitter4r
-* Chronic
-* Erubis
-* Pony
+  Logging
+  Hitimes
+  mongo + mongo_ext
+  Twitter4r
+  Chronic
+  Erubis
+  Pony
 
-== INSTALL:
+== INSTALL
 
-* sudo gem install rackamole
+  sudo gem install rackamole
 
-== USAGE:
+== USAGE
 
 === Rails applications
 

lib/rackamole.rb

@@ -1,7 +1,7 @@
 module Rackamole
 
   # :stopdoc:
-  VERSION = '0.2.5'
+  VERSION = '0.2.6'
   LIBPATH = ::File.expand_path(::File.dirname(__FILE__)) + ::File::SEPARATOR
   PATH = ::File.dirname(LIBPATH) + ::File::SEPARATOR
   # :startdoc:

lib/rackamole/mole.rb

@@ -3,6 +3,8 @@
 require 'mongo'
 require 'yaml'
 
+# TODO - remove default app and db name
+# TODO - add recording for response
 module Rack
   class Mole
 
@@ -43,6 +45,9 @@ class Mole
     #    :user_key => { :session_key => :user_id, :extractor => lambda{ |id| User.find( id ).name} }
     # ==
     #
+    # :mole_excludes :: Exclude some of the paramaters that the mole would normaly include. The list of 
+    #                   customizable paramaters is: software, ip, browser type, url, path, status, headers and body.
+    #                   This options takes an array of symbols. Defaults to [:body].    
     # :excluded_paths:: Exclude paths that you do not wish to mole by specifying an array of regular expresssions.
     # :param_excludes:: Exempt params from being logged to the mole. Specify an array of keys as 
     #                   as symbols ie [:password, :card_number].
@@ -85,12 +90,12 @@ def call( env )
       elapsed = Hitimes::Interval.measure do
         begin
           status, headers, body = @app.call( env )
-        rescue => boom
+        rescue => boom    
           env['mole.exception'] = boom
           mole_feature( env, elapsed, status, headers, body )
           raise boom
         end
-      end
+      end      
       mole_feature( env, elapsed, status, headers, body )
       return status, headers, body
     end 
@@ -119,10 +124,10 @@ def default_options
       { 
         :moleable        =>  true,    
         :log_level       =>  :info,
-        :expiration      =>  60*60, # 1 hour    
-        :app_name        =>  "Moled App",
+        :expiration      =>  60*60, # 1 hour
         :environment     =>  'development',
         :excluded_paths  =>  [/.?\.ico/, /.?\.png/],
+        :mole_excludes   =>  [:body],
         :perf_threshold  =>  10.0,
         :store           =>  Rackamole::Store::Log.new
       }
@@ -227,12 +232,19 @@ def mole_request?( request )
     end
 
     # Extract interesting information from the request
-    def mole_info( env, elapsed, status, headers, body )      
-      request = Rack::Request.new( env )
-      info    = OrderedHash.new
+    def mole_info( env, elapsed, status, headers, body )
+      request  = Rack::Request.new( env )
+      response = nil
+      begin
+        response = Rack::Response.new( body, status, headers )
+      rescue
+        ;
+      end
+
+      info     = OrderedHash.new
 
       return info unless mole_request?( request )
-
+                                                
       session     = env['rack.session']
       route       = get_route( request )
 
@@ -257,29 +269,32 @@ def mole_info( env, elapsed, status, headers, body )
       info[:type]         = (elapsed and elapsed > options[:perf_threshold] ? Rackamole.perf : Rackamole.feature)
       info[:app_name]     = options[:app_name]
       info[:environment]  = options[:environment] || "Unknown"
-      info[:user_id]      = user_id      if user_id
+      info[:user_id]      = user_id if user_id
       info[:user_name]    = user_name || "Unknown"
-      info[:ip]           = ip
-      info[:browser]      = id_browser( user_agent )
       info[:host]         = env['SERVER_NAME']
-      info[:software]     = env['SERVER_SOFTWARE']
       info[:request_time] = elapsed if elapsed
-      info[:url]          = request.url
       info[:method]       = env['REQUEST_METHOD']
-      info[:path]         = request.path
-      info[:route_info]   = route if route
+      info[:route_info]   = route if route      
       info[:created_at]   = Time.now.utc
+
+      # Optional elements...
+      mole?( :software  , info, env['SERVER_SOFTWARE'] )
+      mole?( :ip        , info, ip )
+      mole?( :browser   , info, id_browser( user_agent ) ) 
+      mole?( :url       , info, request.url )
+      mole?( :path      , info, request.path )
+      mole?( :status    , info, status )
+      mole?( :headers   , info, headers )
+      mole?( :body      , info, response.body ) if response
 
       # Dump request params
       unless request.params.empty?
         info[:params] = filter_params( request.params, options[:param_excludes] || [] )
-        # request.params.keys.sort.each { |k| info[:params][k.to_sym] = request.params[k].to_json }
       end
 
       # Dump session var
       if session and !session.empty?
         info[:session] = filter_params( session, options[:session_excludes] || [] )
-        # session.keys.sort{ |a,b| a.to_s <=> b.to_s }.each { |k| info[:session][k.to_sym] = session[k].to_json }
       end
 
       # Check if an exception was raised. If so consume it and clear state
@@ -293,7 +308,14 @@ def mole_info( env, elapsed, status, headers, body )
       end      
       info
     end
-
+
+    # check exclusion to see if the information should be moled
+    def mole?( key, stash, value )
+# puts "Checking #{key} -- #{options[:mole_excludes].inspect}"      
+      return stash[key] = value if !options[:mole_excludes] or options[:mole_excludes].empty?
+      stash[key] = (options[:mole_excludes].include?( key ) ? nil : value)
+    end
+
     # filters out params hash and convert values to json
     def filter_params( source, excludes )
       results = OrderedHash.new

lib/rackamole/store/mongo_db.rb

@@ -98,8 +98,7 @@ def validate_options( opts )
         def default_options
           {
              :host     => 'localhost',
-             :port     => Mongo::Connection::DEFAULT_PORT,
-             :db_name  => 'mole_mdb'
+             :port     => Mongo::Connection::DEFAULT_PORT
           }
         end
 
@@ -210,7 +209,10 @@ def field_map
             :ruby_version => :ver,
             :fault        => :msg,
             :stack        => :sta,
-            :created_at   => :cro
+            :created_at   => :cro,
+            :status       => :sts,
+            :headers      => :hdr,
+            :body         => :bod
           }
         end      
     end

spec/rackamole/mole_spec.rb

@@ -161,7 +161,39 @@ def slow_app( opts={} )
       fault.count.should == 1
     end
   end
+
+  # ---------------------------------------------------------------------------
+  describe "exclusions" do    
+    before( :each)  do
+      opts = @opts.clone
+      opts[:mole_excludes] = [:headers, :body, :browser, :ip, :url]      
+      app( opts )
+    end
+
+    it "should exclude some mole attributes correctly" do      
+      get "/fred/blee", nil, @test_env
 
+      @test_store.mole_result[:app_name].should     == "Test App"
+      @test_store.mole_result[:environment].should  == :test
+      @test_store.mole_result[:user_id].should      be_nil
+      @test_store.mole_result[:user_name].should    == 'fernand'
+      @test_store.mole_result[:method].should       == 'GET'
+      @test_store.mole_result[:path].should         == '/fred/blee'
+      @test_store.mole_result[:type].should         == Rackamole.feature
+      @test_store.mole_result[:params].should       be_nil
+      @test_store.mole_result[:session].should_not  be_nil
+      @test_store.mole_result[:session].keys.should have(2).items
+      @test_store.mole_result[:status].should       == 200
+
+      # Excluded
+      @test_store.mole_result[:headers].should      be_nil
+      @test_store.mole_result[:body].should         be_nil      
+      @test_store.mole_result[:browser].should      be_nil      
+      @test_store.mole_result[:ip].should           be_nil  
+      @test_store.mole_result[:url].should          be_nil          
+    end
+  end
+
   # ---------------------------------------------------------------------------    
   describe 'moling a request' do
     before :each do
@@ -170,6 +202,7 @@ def slow_app( opts={} )
 
     it "should set the mole meta correctly" do
       get "/fred/blee", nil, @test_env
+
       @test_store.mole_result[:app_name].should     == "Test App"
       @test_store.mole_result[:environment].should  == :test
       @test_store.mole_result[:user_id].should      be_nil
@@ -183,6 +216,9 @@ def slow_app( opts={} )
       @test_store.mole_result[:params].should       be_nil
       @test_store.mole_result[:session].should_not  be_nil
       @test_store.mole_result[:session].keys.should have(2).items
+      @test_store.mole_result[:status].should       == 200
+      @test_store.mole_result[:headers].should      == { "Content-Type" => "text/plain" }
+      @test_store.mole_result[:body].should         be_nil
     end
 
     it "mole an exception correctly" do
@@ -195,7 +231,7 @@ def slow_app( opts={} )
         @test_store.mole_result[:stack].should have(4).items
         @test_store.mole_result[:fault].should == 'Oh snap!'
         last_request.env['mole.stash'].should_not be_nil
-        fault = last_request.env['mole.stash'].send( :find_fault, "/", "./spec/rackamole/mole_spec.rb:190" )
+        fault = last_request.env['mole.stash'].send( :find_fault, "/", "./spec/rackamole/mole_spec.rb:226" )
         fault.should_not be_nil
         fault.count.should == 1
       end
@@ -234,7 +270,7 @@ def slow_app( opts={} )
     it "should find route info correctly" do
       RAILS_ENV = true
       ActionController::Routing::Routes.stub!( :recognize_path ).and_return( { :controller => 'fred', :action => 'blee' } )
-      rack = Rack::Mole.new( nil )
+      rack = Rack::Mole.new( nil, :app_name => "test app" )
 
       # routes.should_receive( 'recognize_path' ).with( 'fred', { :method => 'blee' } ).and_return(  )
       res = rack.send( :get_route, OpenStruct.new( :path => "/", :request_method => "GET") )
@@ -288,12 +324,13 @@ def slow_app( opts={} )
   describe '#alertable?' do
     before( :each ) do
       @rack = Rack::Mole.new( nil, 
-        :twitter => { 
+        :app_name => "test app",
+        :twitter  => { 
           :username => 'fred', 
           :password => 'blee', 
           :alert_on => [Rackamole.perf, Rackamole.fault] 
         },
-        :email => { 
+        :email    => { 
           :from     => 'fred', 
           :to       => ['blee'], 
           :alert_on => [Rackamole.perf, Rackamole.fault, Rackamole.feature] 
@@ -321,6 +358,7 @@ def slow_app( opts={} )
   describe '#configured?' do
     before( :each ) do
       options = {
+        :app_name     => "test app",
         :blee         => [1,2,3],
         :twitter      => { :username => 'Fernand', :password => "Blee", :alert_on => [Rackamole.perf, Rackamole.fault] },
       }
@@ -354,7 +392,7 @@ def slow_app( opts={} )
   # ---------------------------------------------------------------------------
   describe '#id_browser' do
     before :all do
-      @rack = Rack::Mole.new( nil )
+      @rack = Rack::Mole.new( nil, :app_name => "test app" )
     end
 
     it "should detect a browser type correctly" do
@@ -442,4 +480,13 @@ def slow_app( opts={} )
 
   end
 
+  # ---------------------------------------------------------------------------    
+  describe 'required params' do
+    it "should crap out if a required param is omitted" do
+      lambda {
+        Rack::Mole.new( app )
+      }.should raise_error( /app_name/ )
+    end
+  end
+
 end