README

Logster is a utility for reading log files and generating metrics in Graphite or Ganglia. It is ideal for visualizing trends of events that are occurring in your application/system/error logs. For example, you might use logster to graph the number of occurrences of HTTP response code that appears in your web server logs.

Logster maintains a cursor, via logtail, on each log file that it reads so that each successive execution only inspects new log entries. In other words, a 1 minute crontab entry for logster would allow you to generate near real-time trends in Graphite or Ganglia for anything you want to measure from your logs. 

This tool is made up of a framework script, logster, and parsing scripts that are written to accommodate your specific log format. Two sample parsers are included in this distribution. The parser scripts essentially read a log file line by line, apply a regular expression to extract useful data from the lines you are interested in, and then aggregate that data into metrics that will be submitted to either Ganglia or Graphite. Take a look through the sample parsers, which should give you some idea of how to get started writing your own.


History

The logster project was created at Etsy as a fork of ganglia-logtailer (https://bitbucket.org/maplebed/ganglia-logtailer). We made the decision to fork ganglia-logtailer because we were removing daemon-mode from the original framework. We only make use of cron-mode, and supporting both cron- and daemon-modes makes for more work when creating parsing scripts. We care strongly about simplicity in writing parsing scripts -- which enables more of our engineers to write log parsers quickly.


Installation

Logster depends on the "logtail" utility that can be obtained from the logcheck package, either from a Debian package manager or from source:

  http://packages.debian.org/source/sid/logcheck.

An RPM for logtail can be found here:

  http://rpmfind.net/linux/RPM/epel/testing/5/x86_64/logcheck-1.3.13-3.el5.noarch.html

Once you have logtail installed, then the only other thing you need to do is run the installation commands in the Makefile:

  $ sudo make install


Usage

You can test logster from the command line. There are two sample parsers: SampleLogster, which generates stats from an Apache access log; and Log4jLogster, which generates stats from a log4j log. The --dry-run option will allow you to see the metrics being generated on stdout rather than sending them to either Ganglia or Graphite.

  $ sudo /usr/sbin/logster --dry-run --output=ganglia SampleLogster /var/log/httpd/access_log

  $ sudo /usr/sbin/logster --dry-run --output=graphite --graphite-host=graphite.example.com:2003 SampleLogster /var/log/httpd/access_log

Additional usage details can be found with the -h option:

$ ./logster -h
usage: logster [options] parser logfile

Tail a log file and filter each line to generate metrics that can be sent to
common monitoring packages.

Usage: logster [options] parser logfile

Tail a log file and filter each line to generate metrics that can be sent to
common monitoring packages.

Options:
  -h, --help            show this help message and exit
  -p METRIC_PREFIX, --metric-prefix=METRIC_PREFIX
                        Add prefix to all published metrics. This is for
                        people that may multiple instances of same service on
                        same host.
  --parser-help         Print usage and options for the selected parser
  --parser-options=PARSER_OPTIONS
                        Options to pass to the logster parser such as "-o
                        VALUE --option2 VALUE". These are parser-specific and
                        passed directly to the parser.
  --gmetric-options=GMETRIC_OPTIONS
                        Options to pass to gmetric such as "-d 180 -c
                        /etc/ganglia/gmond.conf" (default). These are passed
                        directly to gmetric.
  --graphite-host=GRAPHITE_HOST
                        Hostname and port for Graphite collector, e.g.
                        graphite.example.com:2003
  -s STATE_DIR, --state-dir=STATE_DIR
                        Where to store the logtail state file.  Default
                        location /var/run
  -o OUTPUT, --output=OUTPUT
                        Where to send metrics (can specify multiple times).
                        Choices are 'graphite', 'ganglia', or 'stdout'.
  -d, --dry-run         Parse the log file but send stats to standard output.
  -D, --debug           Provide more verbose logging for debugging.