Home

Apache Crunch is a log analysis tool

Check out our Tutorial if you want to get started right away.

With Apache Crunch, you write code like

filter! do
    status == "404" and reqheader_accept =~ /^image\//
end

count_by do
    reqheader_accept =~ /^image\/(\S+)/
    $1
end

puts "404s by image type:"
count_by.to_yaml

To get results like

404s by image type:
---
"png": 219
"jpeg": 16
"gif": 1

Running

You run ApacheCrunch like so:

apachecrunch.rb <PROCEDURE> <LOG> [--format <FORMAT>] [--progress <METER TYPE>]

where PROCEDURE is the path to a file written in our domain-specific language describing the analysis to be performed and LOG is the path to the log file. If you have configured additional log formats, you can specify one as FORMAT -- otherwise, the NCSA format will be assumed. To add your own custom log formats, see Parsing custom log formats

--progress gives you a progress meter as the log file is parsed. METER TYPE can be "entry", which prints out how many entries have been parsed so far, or "time", which prints out the time of the last entry parsed.

DSL (How To Write Analysis Procedures)

The file that describes how you want your log analyzed is called a procedure. For example, this procedure:

c = count_where do
    status == "301"
end
puts c

Would print the number of requests in the log with HTTP status 301. The status there is an attribute of the log entry, and you can get a list of those in the next section. count_where is a routine, and those are detailed further down.

Log entry attributes

The following attributes can be referenced in any block passed to a routine in your procedure. Each is listed with its symbol from the mod_log_config documentation:

remote_host (%h)
log_name (%l)
remote_user (%u)
time (%t): Makes year, month, day, hour, minute, and second available as derived attributes
req_firstline (%r)
status (%s)
bytes_sent (%b or %B)
serve_time_micro (%D)
req_method (%m): May be derived from req_firstline
url_path (%U): May be derived from req_firstline
query_string (%q): May be derived from req_firstline
protocol (%H): May be derived from req_firstline
reqheader_foo_bar (%{Foo-Bar}i)
regex_foobar (%{foobar:[A-Za-z0-9]+}r) This is not actually an Apache format string. It matches an arbitrary regex (which can't contain the character '}'... sorry), in this example an alphanumeric string.

Derived attributes

Some of the attributes listed above may be derived from others. For example, if your format string contains "%r" (req_firstline), you will be able to use method, url_path, query_string, and protocol in your procedures. They are derived from req_firstline.

Derived attributes tend to be somewhat slower than regular ones.

Routines

These are the routines available.

each

Executes the given block once for each entry parsed.

# Print the User-Agent header for each line
each do
    puts reqheader_user_agent
end

sum

Determines the sum of the values produced by the block. Every value must be able to be interpreted as a float (i.e. must have a to_f method), and the return value of this routine is a float.

# Determine how many bytes were sent to hosts on the 10.0.0.0/8 network
filter {remote_host =~ /^10\./}
puts (sum {bytes_sent}).to_i

count_where

Determines the number of entries for which the block evaluates to true

# Count the number of redirects on requests for images
c = count_where do
    status =~ /^3/ and reqheader_accept =~ /\bimage\//
end
puts c

most_common

Finds the n most common values for the given quantity. Returns a list of [value, count] pairs, sorted descending by count.

# Print the 10 most common user-agents hitting your website
mc = most_common 10 do
    reqheader_user_agent
end
mc.each do |user_agent,count|
    puts "%-8d: %s" % [count, user_agent]
end

filter

Filters the log file by the given block. If you pass an argument, it's the path to a file into which the filtered results will be written. Otherwise, the filtering will be done in a temporary location. Either way, any subsequent analysis will be performed on the results of this filtering.

# Filter down to only requests from localhost in a file called "/tmp/localhost.log"
filter "/tmp/localhost.log" do
    remote_host == "127.0.0.1"
end

filter!

Filters the log file in place. This alters the file! Any subsequent routines in the procedure will operate on the new, filtered log.

# Filter down to only requests from localhost
filter! do
    remote_host == "127.0.0.1"
end

count_by

Counts the number of entries for which the given block evaluates to each possible value.

# Get the number of requests with each status code
results = count_by do
    status
end

distribution

Finds the distribution of the block's value into buckets (as you would use to make a histogram). You pass the size of the bucket as a parameter.

# Print the counts of outgoing response sizes in bytes, in buckets of width 100
d = distribution 100 do
    bytes_sent
end
d.each_pair do |bucket, count|
    puts "%8d%s" % [bucket, count]
end

log_distribution

Finds the distribution of the block's value into buckets of exponentially-increasing width. You pass the base of the exponential sequence.

# The counts of response times, in microseconds, in buckets of width 2, 4, 8, 16, etc.
log_distribution 2 do
    serve_time_micro
end

confidence_interval

Returns the bounds of a confidence interval for the values returned by the block. You pass the percentage confidence as an argument.

All values produced by the block must be able to be cast as a float (i.e. they must have a to_f method), and the return value of this routine is a float.

# Print the bounds of a 95% confidence interval for length of request bodies
ci = confidence_interval 95 do
    reqheader_content_length.nil? nil : reqheader_content_length.to_i
end
ci.each do |bound|
    puts bound.to_i
end

percentile

Returns the nth percentile for the values produced by the block.