Home

Apache Crunch is a log analysis tool

Check out our Tutorial if you want to get started right away.

With Apache Crunch, you write code like

filter! do
    status == "404" and reqheader_accept =~ /^image\//
end

count_by do
    reqheader_accept =~ /^image\/(\S+)/
    $1
end

puts "404s by image type:"
count_by.to_yaml

To get results like

404s by image type:
---
"png": 219
"jpeg": 16
"gif": 1

Running

You run ApacheCrunch like so:

apachecrunch.rb <PROCEDURE> <LOG> [--format <FORMAT>] [--progress <METER TYPE>]

where PROCEDURE is the path to a file written in our domain-specific language describing the analysis to be performed and LOG is the path to the log file. If you have configured additional log formats, you can specify one as FORMAT -- otherwise, the NCSA format will be assumed. To add your own custom log formats, see Parsing custom log formats

--progress gives you a progress meter as the log file is parsed. METER TYPE can be "entry", which prints out how many entries have been parsed so far, or "time", which prints out the time of the last entry parsed.

DSL (How To Write Analysis Procedures)

The file that describes how you want your log analyzed is called a procedure. For example, this procedure:

c = count_where do
    status == "301"
end
puts c

Would print the number of requests in the log with HTTP status 301. The status there is an attribute of the log entry, and you can get a list of those in the next section. count_where is a routine, and those are detailed further down.

Log entry attributes

The following attributes can be referenced in any block passed to a routine in your procedure. Each is listed with its symbol from the mod_log_config documentation:

• remote_host (%h)
• log_name (%l)
• remote_user (%u)
• time (%t)
• req_firstline (%r)
• status (%s)
• bytes_sent (%b or %B)
• serve_time_micro (%D)
• req_method (%m): May be derived from req_firstline
• url_path (%U): May be derived from req_firstline
• query_string (%q): May be derived from req_firstline
• protocol (%H): May be derived from req_firstline
• reqheader_foo_bar (%{Foo-Bar}i)
• regex_foobar (%{foobar:[A-Za-z0-9]+}r) This is not actually an Apache format string. It matches an arbitrary regex (which can't contain the character '}'... sorry), in this example an alphanumeric string.

Derived attributes

Some of the attributes listed above may be derived from others. For example, if your format string contains "%r" (req_firstline), you will be able to use method, url_path, query_string, and protocol in your procedures. They are derived from req_firstline.

Routines

These are the routines available.

each

Executes the given block once for each entry parsed.

# Print the User-Agent header for each line
each do
    puts reqheader_user_agent
end

count_where

Determines the number of entries for which the block evaluates to true

# Count the number of redirects on requests for images
c = count_where do
    status =~ /^3/ and reqheader_accept =~ /\bimage\//
end
puts c

filter

Filters the log file by the given block. If you pass an argument, it's the path to a file into which the filtered results will be written. Otherwise, the filtering will be done in a temporary location. Either way, any subsequent analysis will be performed on the results of this filtering.

# Filter down to only requests from localhost in a file called "/tmp/localhost.log"
filter "/tmp/localhost.log" do
    remote_host == "127.0.0.1"
end

filter!

Filters the log file in place. This alters the file! Any subsequent routines in the procedure will operate on the new, filtered log.

# Filter down to only requests from localhost
filter! do
    remote_host == "127.0.0.1"
end

count_by

Counts the number of entries for which the given block evaluates to each possible value.

# Print the top 10 most requested paths
results = count_by do
    req_method == "GET"
    path
end

sorted_paths = results.keys.sort do |a, b|
    results[a] <=> results[b]
end[:10]
sorted_paths.each do |path|
    puts "#{results[path]}\t#{path}"
end

distribution

Finds the distribution of the block's value into buckets (as you would use to make a histogram). You pass the size of the bucket as a parameter.

# Print the counts of outgoing response sizes in bytes, in buckets of width 100
d = distribution 100 do
    bytes_sent
end
d.each_pair do |bucket, count|
    puts "%8d%s" % [bucket, count]
end

log_distribution

Finds the distribution of the block's value into buckets of exponentially-increasing width. You pass the base of the exponential sequence.

# The counts of response times, in microseconds, in buckets of width 2, 4, 8, 16, etc.
log_distribution 2 do
    serve_time_micro
end

confidence_interval

Returns the bounds of a confidence interval for the values returned by the block. You pass the percentage confidence as an argument.

# Print the bounds of a 95% confidence interval for length of request bodies
ci = confidence_interval 95 do
    reqheader_content_length.nil? nil : reqheader_content_length.to_i
end
ci.each do |bound|
    puts bound
end

Configuring Additional Log Formats

• If you need a log format that's not included by default (i.e. anything other than NCSA or Ubuntu's out-of-the-box format), see Parsing custom log formats
• Not particularly fast
• Doesn't (yet) support any of the inclusion ("...") _config documentation](http://httpd.apache.org/docs/2.0/mod/mod_log_config.html)
• Ignores any log entries that it can't parse and prints a warning. This includes anything where a field contains a quotation mark. Yes, sometimes the first line of the request or the User-Agent header contains quotation marks. It's horrible.

License

Creative Commons Share-Alike