Parsing custom log formats

Apache Crunch allows you to parse pretty much any arbitrary log format. You tell it what format to expect by passing the --format argument and a format name. The formats available by default are ncsa, which RedHat's Apache package uses by default, and ubuntu, which is what's used by Ubuntu's.

If you're using a log format that Apache Crunch doesn't know about, you just need to add it to a configuration file, either ~/.apachecrunch/log_formats.rb or /etc/apachecrunch/log_formats.rb. For example, you might add something like this:

formats[:my_format] = %q!%h %l %u %t \"%r\" %s %b %D %{Content-length}i \"%{Referer}i\"!

You can copy the format string straight from the LogFormat line in your Apache config, but I recommend using Ruby's %q operator as shown to make sure everything stays literal.

When you want to parse a log in this format, you just call Apache Crunch like so:

apachecrunch --format my_format my_procedure.rb /var/log/apache2/access_log

Accounting for unsupported log formats with arbitrary regexes

If your log format contains an interpolated value that Apache Crunch doesn't know about, or if there's just some extra data added by another application, you can use regular expression matching in the format definition. For example, suppose your logs contain a unique request identifier in base 64, like YWJjZGVmZ2g=. You could account for this with something like

formats[:my_format] = %q!%h %l %u %t \"%r\" %s %b %{req_identifier:[A-Za-z0-9/+=]+}!

You'd then be able to access this identifier from your procedures as regex_req_identifier.