Home

Apache Crunch is a log analysis tool

You write code like

filter! do
    status == "404" and reqheader_accept =~ /^image\//
end

count_by do
    reqheader_accept =~ /^image\/(\S+)/
    $1
end

puts "404s by image type:"
count_by.to_yaml

To get results like

404s by image type:
---
"png": 219
"jpeg": 16
"gif": 1

Running

You run ApacheCrunch like so:

apachecrunch.rb <PROCEDURE> <LOG> [--format <FORMAT>]

where PROCEDURE is the path to a file written in our domain-specific language describing the analysis to be performed and LOG is the path to the log file. If you have configured additional log formats, you can specify one as FORMAT -- otherwise, the NCSA format will be assumed.

DSL (How To Write Analysis Procedures)

The file that describes how you want your log analyzed is called a procedure. For example, this procedure:

c = count_where do
    status == "301"
end
puts c

Would print the number of requests in the log with HTTP status 301. The status there is an attribute of the log entry, and you can get a list of those in the next section. count_where is a routine, and those are detailed further down.

Log entry attributes

The following attributes can be referenced in any block passed to a routine in your procedure. Each is listed with its symbol from the mod_log_config documentation:

• remote_host (%h)
• log_name (%l)
• remote_user (%u)
• time (%t)
• req_firstline (%r)
• status (%s)
• bytes_sent (%b or %B)
• serve_time_ms (%D)
• reqheader_foo_bar (%{Foo-Bar}i)
• regex_foobar (%{foobar:[A-Za-z0-9]+}r) This is not actually an Apache format string. It matches an arbitrary regex (which can't contain the character '}'... sorry), in this example an alphanumeric string.

Routines

These are the routines available.

each

Executes the given block once for each entry parsed.

# Print the User-Agent header for each line
each do
    puts reqheader_user_agent
end

count_where

Determines the number of entries for which the block evaluates to true

# Count the number of redirects on requests for images
c = count_where do
    status =~ /^3/ and reqheader_accept =~ /\bimage\//
end
puts c

filter!

Filters the log file in place. This alters the file! Any subsequent routines in the procedure will operate on the new, filtered log.

# Filter down to only requests from localhost
filter! do
    remote_host == "127.0.0.1"
end

count_by

Counts the number of entries for which the given block evaluates to each possible value.

# Print the top 10 most requested paths
results = count_by do
    req_firstline =~ /^GET ([^\S?#]+)/
    $1
end

sorted_paths = results.keys.sort do |a, b|
    results[a] <=> results[b]
end[:10]
sorted_paths.each do |path|
    puts "#{results[path]}\t#{path}"
end

distribution

Finds the distribution of the block's value into buckets. You pass the size of the bucket as a parameter.

# Print the distribution of request times-to-serve in microseconds, in 10000-microsecond buckets.
d = distribution 10**4 do
    serve_time_ms
end
d.each_pair do |bucket, count|
    puts "#{bucket}\t#{count}"
end

confidence_interval

Returns the bounds of a confidence interval for the values returned by the block. You pass the percentage confidence as an argument.

# Print the bounds of a 95% confidence interval for value of the request's Content-Length header
ci = confidence_interval 95 do
    reqheader_content_length
end
ci.each do |bound|
    puts bound
end

Configuring Additional Log Formats

If you need a log format that's not included by default (i.e. anything other than NCSA), add it to log_formats.rb.

Caveats

• Not particularly fast
• Doesn't (yet) support any of the inclusion ("...") or request specifier (">" and "<") stuff from the mod_log_config documentation
• Ignores any log entries that it can't parse and prints a warning. This includes anything where a field contains a quotation mark. Yes, sometimes the first line of the request or the User-Agent header contains quotation marks. It's horrible.

License

Creative Commons Share-Alike