Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Logster is a utility for reading log files and generating metrics in Graphite or Ganglia. It is ideal for visualizing trends of events that are occurring in your application/system/error logs. For example, you might use logster to graph the number of occurrences of HTTP response code that appears in your web server logs. Logster maintains a cursor, via logtail, on each log file that it reads so that each successive execution only inspects new log entries. In other words, a 1 minute crontab entry for logster would allow you to generate near real-time trends in Graphite or Ganglia for anything you want to measure from your logs. This tool is made up of a framework script, logster, and parsing scripts that are written to accommodate your specific log format. Two sample parsers are included in this distribution. The parser scripts essentially read a log file line by line, apply a regular expression to extract useful data from the lines you are interested in, and then aggregate that data into metrics that will be submitted to either Ganglia or Graphite. Take a look through the sample parsers, which should give you some idea of how to get started writing your own. History The logster project was created at Etsy as a fork of ganglia-logtailer (https://bitbucket.org/maplebed/ganglia-logtailer). We made the decision to fork ganglia-logtailer because we were removing daemon-mode from the original framework. We only make use of cron-mode, and supporting both cron- and daemon-modes makes for more work when creating parsing scripts. We care strongly about simplicity in writing parsing scripts -- which enables more of our engineers to write log parsers quickly. Installation Logster depends on the "logtail" utility that can be obtained from the logcheck package, either from a Debian package manager or from source: http://packages.debian.org/source/sid/logcheck. An RPM for logtail can be found here: http://rpmfind.net/linux/RPM/epel/testing/5/x86_64/logcheck-1.3.13-3.el5.noarch.html Once you have logtail installed, then the only other thing you need to do is run the installation commands in the Makefile: $ sudo make install Usage You can test logster from the command line. There are two sample parsers: SampleLogster, which generates stats from an Apache access log; and Log4jLogster, which generates stats from a log4j log. The --dry-run option will allow you to see the metrics being generated on stdout rather than sending them to either Ganglia or Graphite. $ sudo /usr/sbin/logster --dry-run --output=ganglia SampleLogster /var/log/httpd/access_log $ sudo /usr/sbin/logster --dry-run --output=graphite --graphite-host=graphite.example.com:2003 SampleLogster /var/log/httpd/access_log Additional usage details can be found with the -h option: $ ./logster -h usage: logster [options] parser logfile Tail a log file and filter each line to generate metrics that can be sent to common monitoring packages. Usage: logster [options] parser logfile Tail a log file and filter each line to generate metrics that can be sent to common monitoring packages. Options: -h, --help show this help message and exit -p METRIC_PREFIX, --metric-prefix=METRIC_PREFIX Add prefix to all published metrics. This is for people that may multiple instances of same service on same host. --parser-help Print usage and options for the selected parser --parser-options=PARSER_OPTIONS Options to pass to the logster parser such as "-o VALUE --option2 VALUE". These are parser-specific and passed directly to the parser. --gmetric-options=GMETRIC_OPTIONS Options to pass to gmetric such as "-d 180 -c /etc/ganglia/gmond.conf" (default). These are passed directly to gmetric. --graphite-host=GRAPHITE_HOST Hostname and port for Graphite collector, e.g. graphite.example.com:2003 -s STATE_DIR, --state-dir=STATE_DIR Where to store the logtail state file. Default location /var/run -o OUTPUT, --output=OUTPUT Where to send metrics (can specify multiple times). Choices are 'graphite', 'ganglia', or 'stdout'. -d, --dry-run Parse the log file but send stats to standard output. -D, --debug Provide more verbose logging for debugging.