Tutorial

Installing

Let's get Apache Crunch installed:

gem install apachecrunch

That should be it. You can also download a tarball or the latest source here.

If you get a "command not found" error when you try to run apachecrunch, you may need to add the path where gem installs its executables to your PATH environment variable. Use gem environment to find out where this is.

A simple procedure to filter log data

In Apache Crunch, a procedure is a simple script that describes what you want to do to your log file. As a simple example, let's write a procedure to filter your log file down to only POST requests. Open a file called my_procedure.rb and enter this code:

filter "only-posts.log" do
    req_method == "POST"
end

filter is what we call a routine. We're passing the path to a file, only-posts.log, that will contain the results of this filter routine. And then we tell it we want to filter down to results where req_method, a log entry attribute, is POST.

This is all Apache Crunch needs. Let's run it. If you're using the out-of-the-box Apache configuration for your distribution, this should be as simple as

# RedHat:
apachecrunch my_procedure.rb /var/log/httpd/access_log
# Ubuntu:
apachecrunch --format ubuntu my_procedure.rb /var/log/apache2/access_log

When this finishes, check out the new file only-posts.log. It should contain all the POST requests from your access log.

If you got a "Log line did not match expected format" warning for every line in the log, you need to tell ApacheCrunch about your log format. You're in luck, because we have a wiki page on Parsing custom log formats, and it's super easy. However, if you only got a few of these warnings, it's probably because there's a User-Agent or something in your log file that has a double-quote in it, such as gosospider "Mozilla 5.0.

A cooler procedure for finding the 10 most commonly requested paths

Let's try our hand at something a little trickier. We're often interested in what's going on with the most commonly requested pages in our application. To find out what pages these are, we can use the most_common routine, which returns a list of [value, count] pairs for the most common values in the given block. This code calculates the most common 10 paths requested in your log:

paths_and_counts = most_common 10 do
    url_path
end
paths_and_counts.each do |pair|
    puts "%-10d%s" % pair.reverse
end

Progress meters

You may find yourself running an Apache Crunch procedure and wishing you knew how long it was going to take to finish. You can solve this problem with the --progress <METER TYPE> flag. For example, to see how many log entries have been processed so far:

apachecrunch --progress entry my_procedure.rb /var/log/apache2/access_log

or to see the date and time of the most recent log entry processed,

apachecrunch --progress time my_procedure.rb /var/log/apache2/access_log