Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 2853d9adb9
Fetching contributors…

Cannot retrieve contributors at this time

72 lines (51 sloc) 3.072 kb

Ruby Reduce

Ruby Reduce is a small library designed to run reduce commands on a Rails 3.x log without needing to setup a hadoop cluster to do it. So it is basically designed to be a small scale reduce function. The original idea was to make the gem plugable so that you could plugin different inputers to chunk non Rails 3 log files. Also there was supposed to be a similar plugin design for the output but that has not happened yet. So currently it only accepts rails 3 files for input and writes the results to MongoDB.


gem install ruby_reduce


The usage is also very straight forward and demostraited by this code snippet.

require 'date' #only required because I parse a date
require 'ruby_reduce' # 1) Require ruby reduce

module RubyReduce
  #return host and port and mongo_db (or nothing for default host and port)
  def self.mongo_connection

  #actual mongo db to connect to
  def self.mongo_db
    return 'test_db'

#actually running the reduce function
r =  RubyReduce.reduce({
    :input => '../production.log', #the location of the log file to reduce
    # The map function can be anything that response to call and accepts 2 paramaters.  
    # The first paramater is the log file name and the second are all the log statements 
    # for a single rails request.  The only thing required here is that you **emit** a 
    # key value pair to reduce later.  simple call the `emit(key, value)` function to accomplish this.
    :map => do |key, value|
      processed_by = /Processing by (\w*#\w*)/.match(value)

        unless processed_by.nil?
          emit processed_by[1].gsub!("#", "_"), value
    # The reduce function can be anything that responds to call and takes one argument.  
    # This argument will be the value emited in the map function (so reduce is called once 
    # for every key value pair emited.  Also in the reduce function you need to **emit** 
    # the result of the reduce.  Simply call `emit(result)` to tell RubyReduce what the result is.
    :reduce => do |log_statement|
        date = /Started [GET|POST|DELETE|PUT|HEAD].* at (.*)/.match(log_statement) 
        unless date.nil? 
          processing_time = /Completed \d* .* in (\d*)/.match(log_statement)[1]
          emit({'date' => Time.parse(date[1]), 'processed_time' => processing_time})
    #MongoDB collection name to write results to
    :output => 'graph_data'

Once all of the key value pairs have been reduced the results will be collected by key and written into mongodb with one document for each key emited in the map function. The id (_id) of this document is the emited key


Right now this is at best alpha software. There are no tests and while I use it for my on projects it is not really been on a wide range of problems. Your feedback is welcome.


Contact me at with question, comments, or just that you are using the library and want me to continue work on it.

Jump to Line
Something went wrong with that request. Please try again.