Configurable Perl script to analyse and summarise syslog files
Perl Shell JavaScript
Switch branches/tags
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


# Copyright 2003, 2009 Michael Knox,
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    GNU General Public License for more details.
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <>.

This Perl script was developed to analyse and sumamrise logfiles. The original version was hardcoded for syslog formated logfiles.
The new version (this one), has a number of significant changes:
1. Specify the logfile format in the config file
2. Multiple log and config files specified on command line
3. Multiple actions & reports for each rule
4. The arithmetic reporting functions have been moved from ACTION's to REPORT's.

Command line args
./ -l <log file> -c <config file> [-d <debug level>]
<debug level> is 0 -> 9, 0 no debug, 9 extremely verbose.

Config file 
Any content following a # is regarded as a comment

** Format's **

Log file format are described with the following notation ...
FORMAT <format name> {
	FIELDS 2 \t
	FIELD 1 abc
	FIELD 2 def

This format stanza tells that the logfile has 2 fields which are tablimited. Rules can then refer to field 1 as 'abc' and field 2 as 'def'

Subfields can also be defined, for instance:
FORMAT <format name> {
	FIELDS 2 \t
	FIELD 1 abc
	FIELD 2 def
	FIELDS 2-3 /\d+/
	FIELD 2-1 mno
	FIELD 2-2 pqr default
	FIELD 2-3 stu

Field 2 is then broken up into 3 fields using the regex /\d+/, and these fields are then available as 'mno', 'pqr' and 'stu'. The original field 2 is still available as 'def'.
Also if the regex could not be applied to the line, the 'default' parameter on FIELD 2-2 means that the contents of FIELD 2 will also be in FIELD 2-2. If the default parameter is not used, the field content would have been put in FIELD 2-1.

The other parameter shown is 'LASTLINEINDEX'. This parameter is used to key a index of the previous lines based on the field specified by 'LASTLINEINDEX'. A hash of previous lines is stored, and key based on this field.

There's  a final parameter, 'default'. This tells logparse to use this format when processing logifles, unless it's told to use a differnet format.
The ability to use multiple format's in the same run however, is yet to be implemented.

** Rule's **
This is where you can generate your reports and statistics for the log files.

RULE [<sample rule name>] {
	FIELDNAME /regex/

rule name is optional. If it's not specified an incremnting integer is used for the name. It is recommended to use names however as this makes it a lot easier to use the 'APPEND' argument for ACTIONS.

There are 3 sections to a rule command, field filters, ACTION's and REPORT's.
If a line doesn't start with ACTION or REPORT, it is assumed to be a field filter.
The first argument is used as the field label (eg 'abc', or field 1 as defined in the sample format above).
The second argument is used as a regex against the specified field.
If the field doesn't match the regex then no further attempts to match the current rule are attempted for the current line.
Regex's can be prefixed by !, in which case a negative match is made. eg If the command is abc /!456/, then only the lines where field abc do not contain 456 will be processed for ACTIONS or REPORTS.
Field matches are treated as a single series of logical AND's, any fails, and the ACTION's and REPORT's will not be processed for that rule for the specific line.

If the FIELD matches all pass, then all the ACTION's are applied.
After all the logfiles are processed the REPORT commands are then processed.

* ACTION's *
ACTION MSG /(.*): TTY=(.*) ; PWD=(.*); USER=(.*); COMMAND=(.*)/ {HOST, 1, 4, 5}

regex is applied to	FIELDNAME, and the the matching fields are stored according to {FIELD LIST}.
FIELD LIST can include FIELDNAMES (generally HOST), the field list is a numerical list of the matches in the regex which are kept for processing by REPORT commands.
In the above example HOST, the 1st, 4th (USER) and 5th (COMMAND) match are kept for processing.

Data from multiple ACTION's in the same rule is stored seperately, but RULE's for that rule look at the combined data set (data from all ACTION's).

* REPORT's *
	REPORT <title> <summary> [action]
	action could be: count, total, avg etc
	REPORT "Sudo Usage" "{2} ran {4} as {3} on {1}: {x} times" count
This will generate report title "Sudo Usage", and each entry in the report is set by <sumamry>. They are keyed based on the field's in <summary>. The field numbers in <summary> are relative to the data collected in ACTION's.
By default report's are count's, ie the total number of instances which match the key's in <summary>.
{x} in <summary> is the count.

To continue this example ...
	ACTION MSG /(.*): TTY=(.*) ; PWD=(.*); USER=(.*); COMMAND=(.*)/ {HOST, 1, 4, 5}
	REPORT "Sudo Usage" "{2} ran {4} as {3} on {1}: {x} times" count

The REPORT titled "Sudo Usage", contains a count of lines similiar to ...
<USER> ran <command> as <TARGET USER> on <HOST>: <count> times

REPORT's use data combined from all ACTION's, so multiple ACTION's generally replace the APPEND command from the original version.