apacheblackbox.pl [options] --output=file Options: --output the output file to create --detail enable the storing of full details for each unique request --skip regular expression for requests that will not count to any stats --show regular expression for requests that will have full detail --fileonly strip parameters from requests for storing detail --logfile use a logfile instead of STDIN --help full documentation
This is a parser for apache log files that is intended to be used in a pipe from within apache. The basic idea is that you define a custom log format that contains performance and size details for each request, this parser will understand this format and produce regular statistics based on the input.
The statistics gets written to a file, see --output, in the format:
variable=value
A sample from an actual run below:
parser.uptime=50 parser.generatedtime=1204722895 apache.requests.get=1359263 apache.requests.post=50980 apache.status.1xx=0 apache.status.2xx=1123841 apache.status.3xx=307707 apache.status.4xx=444 apache.status.5xx=22 apache.status.100=0 apache.status.101=0 apache.status.200=1123740
There are many more variables kept with self explanatory names. In addition to overall stats details for each unique URL can be kept. The list of URLs to be considered for statistics can be manipulated using the --skip and --show options, how to deal with parameters in the requests can be modified using the --fileonly option.
Most of the values are always incrementing counters, they will overflow back to zero when the limits of the data storage is reached, these are best used in a DERIVE data format in a RRD file using tools such as Cacti.
The output file to create
When enabled a detailed block of stats will be kept for each url accessed, 2 other options modify the behaviour of this see --show and --fileonly
This takes a regular expression of pages that stats should not be kept for. You can use this to keep stats for everything except for example images by using something like .jpg|.gif$. Requests matching this regular expression will not count towards any stats, not detailed stats and not the totals for the server.
If --detail is enabled all unique requests will get a block of stats individual to that request, this will often result in a large list of unexpected statistics being kept for no good reason, you can restrict it to only keeping statistics for files ending in .php by using a regular expression like .php$. With this in place you will only get performance stats for your PHP scripts.
If --detail is enabled statistics for each unique request will be kept, determining uniqueness will include the parameters passed to the script as part of the URL, often this is not desirable using this option the parameters gets discarded and all requests - regardless of parameters - will be considered as one requests and thus count only to one detailed stats block.
By default the intention is to use CustomLog in Apache and pipe the log lines to this script, this is the most robust option as Apache will start the parser should it exit etc, in some cases it might be desirable to just parse a log file instead. Supplying the --logfile option with a path to a file will enable this.
--help
This help page.
Installation is usually just a matter of copying the file anywhere on your file system. Documentation can be viewed with perldoc, to use --help you need Pod::Usage. If you want to parse log files rather than pipe to STDIN you will also need the Proc::Daemon and File::Tail Perl modules.
To activate this script you need to have mod_logio enabled in your apache configuration:
LoadModule logio_module modules/mod_logio.so
With this enabled you can now configure a custom log format to write compatible log entries to this script, you can do this inside a VirtualHost, various other places where CustomLog is valid in an apache config file:
<IfModule mod_logio.c>
CustomLog "| /path/to/apacheblackbox.pl --output /var/www/blackbox.txt --detail" "%a %X %t \"%r\" %s/%>s %{pid}P/%{tid}P %T/%D %I/%O/%B"
</IfModule>
As you can see any of the above options simply get passed to the script from the Apache CustomLog lines, you can further select only certain requests to be passed to this log using the normal apache methods such as -Location_ blocks etc:
<Location /cgi-bin>
SetEnv blackboxlog 1
</Location>
<IfModule mod_logio.c>
CustomLog "| /path/to/apacheblackbox.pl --output /var/www/blackbox.txt --detail --show \\'script.pl\\'" "%a %X %t \"%r\" %s/%>s %{pid}P/%{tid}P %T/%D %I/%O/%B" env=blackboxlog
</IfModule>
Combining the Apache control blocks with the options to this script such as --skip and --show gives you very fine grained control over what stats to record and can be used using the same basic script on many virtual hosts and sets of files on the same machine concurrently.
If you need to consider logs from several virtual hosts as one or simply wish to also store the log on your disk you can instruct the script to read the logfile from disk using the --logfile option. You would then make a CustomLog that save to a file and just run in the parser, when called with the --logfile option it will daemonize to the background. While we've not had stability problems with the script you may want to monitor it is running either through process list, via the parser.generatedtime variable or the age of the output file.
This is based on a basic concept found here: http://www.onlamp.com/pub/a/apache/2004/04/22/blackbox_logs.html
Written by R.I.Pienaar, visit http://www.devco.net/ for more info.