Permalink
Browse files

First release.

  • Loading branch information...
0 parents commit 07e3b6a13a196e48ed9304e8c6b1a071da61244a @mikebrittain mikebrittain committed Mar 12, 2011
Showing with 617 additions and 0 deletions.
  1. +7 −0 Makefile
  2. +61 −0 README
  3. +91 −0 SampleGangliaLogster.py
  4. +91 −0 SampleGraphiteLogster.py
  5. +312 −0 logster
  6. +55 −0 logster_helper.py
@@ -0,0 +1,7 @@
+install:
+ /bin/mkdir -p /usr/share/logster
+ /bin/mkdir -p /var/log/logster
+ /usr/bin/install -m 0755 -t /usr/sbin logster
+ /usr/bin/install -m 0644 -t /usr/share/logster logster_helper.py
+ /usr/bin/install -m 0644 -t /usr/share/logster SampleGraphiteLogster.py
+ /usr/bin/install -m 0644 -t /usr/share/logster SampleGangliaLogster.py
61 README
@@ -0,0 +1,61 @@
+Logster is a utility for reading log files and generating metrics in Graphite or Ganglia. It is ideal for visualizing trends of events that are occurring in your application/system/error logs. For example, you might use logster to graph the number of occurrences of HTTP response code that appears in your web server logs.
+
+Logster maintains a cursor, via logtail, on each log file that it reads so that each successive execution only inspects new log entries. In other words, a 1 minute crontab entry for logster would allow you to generate near real-time trends in Graphite or Ganglia for anything you want to measure from your logs.
+
+This tool is made up of a framework script, logster, and parsing scripts that are written to accommodate your specific log format. Two sample parsers are included in this distribution. The parser scripts essentially read a log file line by line, apply a regular expression to extract useful data from the lines you are interested in, and then aggregate that data into metrics that will be submitted to either Ganglia or Graphite. Take a look through the sample parsers, which should give you some idea of how to get started writing your own.
+
+
+History
+
+The logster project was created at Etsy as a fork of ganglia-logtailer (https://bitbucket.org/maplebed/ganglia-logtailer). We made the decision to fork ganglia-logtailer because we were removing daemon-mode from the original framework. We only make use of cron-mode, and supporting both cron- and daemon-modes makes for more work when creating parsing scripts. We care strongly about simplicity in writing parsing scripts -- which enables more of our engineers to write log parsers quickly.
+
+
+Installation
+
+Logster depends on the "logtail" utility that can be obtained from the logcheck package, either from a Debian package manager or from source:
+
+ http://packages.debian.org/source/sid/logcheck.
+
+An RPM for logtail can be found here:
+
+ http://rpmfind.net/linux/RPM/epel/testing/5/x86_64/logcheck-1.3.13-3.el5.noarch.html
+
+Once you have logtail installed, then the only other thing you need to do is run the installation commands in the Makefile:
+
+ $ sudo make install
+
+
+Usage
+
+You can test logster from the command line. There are two sample parsers (SampleGangliaLogster and SampleGraphiteLogster) that can be used to generate stats from an Apache access log. The --dry-run option will allow you to see the metrics being generated on stdout rather than sending them to either Ganglia or Graphite.
+
+ $ sudo /usr/sbin/logster --dry-run SampleGangliaLogster /var/log/httpd/access_log
+
+ $ sudo /usr/sbin/logster --dry-run SampleGraphiteLogster /var/log/httpd/access_log
+
+Additional usage details can be found with the -h option:
+
+$ ./logster -h
+usage: logster [options] parser logfile
+
+Tail a log file and filter each line to generate metrics that can be sent to
+common monitoring packages.
+
+options:
+ -h, --help show this help message and exit
+ -p METRIC_PREFIX, --metric-prefix=METRIC_PREFIX
+ Add prefix to all published metrics. This is for
+ people that may multiple instances of same service on
+ same host.
+ --gmetric-options=GMETRIC_OPTIONS
+ Options to pass to gmetric such as -d 180 -c
+ /etc/ganglia/gmond.conf (default). These are passed
+ directly to gmetric.
+ --graphite-host=GRAPHITE_HOST
+ Hostname and port for Graphite collector, e.g.
+ graphite.example.com:2003
+ -s STATE_DIR, --state-dir=STATE_DIR
+ Where to store the logtail state file. Default
+ location /var/run
+ -d, --dry-run Parse the log file but send stats to standard output.
+ -D, --debug Provide more verbose logging for debugging.
@@ -0,0 +1,91 @@
+### A sample logster parser file that can be used to count the number
+### of response codes found in an Apache access log.
+###
+### For example:
+### $ sudo ./logster --dry-run SampleGangliaLogster /var/log/httpd/access_log
+###
+###
+### Copyright 2011, Etsy, Inc.
+###
+### This file is part of Logster.
+###
+### Logster is free software: you can redistribute it and/or modify
+### it under the terms of the GNU General Public License as published by
+### the Free Software Foundation, either version 3 of the License, or
+### (at your option) any later version.
+###
+### Logster is distributed in the hope that it will be useful,
+### but WITHOUT ANY WARRANTY; without even the implied warranty of
+### MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+### GNU General Public License for more details.
+###
+### You should have received a copy of the GNU General Public License
+### along with Logster. If not, see <http://www.gnu.org/licenses/>.
+###
+
+import time
+import re
+
+from logster_helper import GangliaMetricObject, GangliaLogster
+from logster_helper import LogsterParsingException
+
+class SampleGangliaLogster(GangliaLogster):
+
+ def __init__(self):
+ '''Initialize any data structures or variables needed for keeping track
+ of the tasty bits we find in the log we are parsing.'''
+ self.http_1xx = 0;
+ self.http_2xx = 0;
+ self.http_3xx = 0;
+ self.http_4xx = 0;
+ self.http_5xx = 0;
+
+ # Regular expression for matching lines we are interested in, and capturing
+ # fields from the line (in this case, http_status_code).
+ self.reg = re.compile('.*HTTP/1.\d\" (?P<http_status_code>\d{3}) .*')
+
+
+ def parse_line(self, line):
+ '''This function should digest the contents of one line at a time, updating
+ object's state variables. Takes a single argument, the line to be parsed.'''
+
+ try:
+ # Apply regular expression to each line and extract interesting bits.
+ regMatch = self.reg.match(line)
+
+ if regMatch:
+ linebits = regMatch.groupdict()
+ status = int(linebits['http_status_code'])
+
+ if (status < 200):
+ self.http_1xx += 1
+ elif (status < 300):
+ self.http_2xx += 1
+ elif (status < 400):
+ self.http_3xx += 1
+ elif (status < 500):
+ self.http_4xx += 1
+ else:
+ self.http_5xx += 1
+
+ else:
+ raise LogsterParsingException, "regmatch failed to match"
+
+ except Exception, e:
+ raise LogsterParsingException, "regmatch or contents failed with %s" % e
+
+
+ def get_state(self, duration):
+ '''Run any necessary calculations on the data collected from the logs
+ and return a list of metric objects.'''
+ self.duration = duration
+
+ # Return a list of metrics objects
+ return [
+ GangliaMetricObject("http_1xx", (self.http_1xx / self.duration), units="Responses per sec"),
+ GangliaMetricObject("http_2xx", (self.http_2xx / self.duration), units="Responses per sec"),
+ GangliaMetricObject("http_3xx", (self.http_3xx / self.duration), units="Responses per sec"),
+ GangliaMetricObject("http_4xx", (self.http_4xx / self.duration), units="Responses per sec"),
+ GangliaMetricObject("http_5xx", (self.http_5xx / self.duration), units="Responses per sec"),
+ ]
+
@@ -0,0 +1,91 @@
+### A sample logster parser file that can be used to count the number
+### of response codes found in an Apache access log.
+###
+### For example:
+### sudo ./logster --dry-run SampleGraphiteLogster /var/log/httpd/access_log
+###
+###
+### Copyright 2011, Etsy, Inc.
+###
+### This file is part of Logster.
+###
+### Logster is free software: you can redistribute it and/or modify
+### it under the terms of the GNU General Public License as published by
+### the Free Software Foundation, either version 3 of the License, or
+### (at your option) any later version.
+###
+### Logster is distributed in the hope that it will be useful,
+### but WITHOUT ANY WARRANTY; without even the implied warranty of
+### MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+### GNU General Public License for more details.
+###
+### You should have received a copy of the GNU General Public License
+### along with Logster. If not, see <http://www.gnu.org/licenses/>.
+###
+
+import time
+import re
+
+from logster_helper import GraphiteMetricObject, GraphiteLogster
+from logster_helper import LogsterParsingException
+
+class SampleGraphiteLogster(GraphiteLogster):
+
+ def __init__(self):
+ '''Initialize any data structures or variables needed for keeping track
+ of the tasty bits we find in the log we are parsing.'''
+ self.http_1xx = 0;
+ self.http_2xx = 0;
+ self.http_3xx = 0;
+ self.http_4xx = 0;
+ self.http_5xx = 0;
+
+ # Regular expression for matching lines we are interested in, and capturing
+ # fields from the line (in this case, http_status_code).
+ self.reg = re.compile('.*HTTP/1.\d\" (?P<http_status_code>\d{3}) .*')
+
+
+ def parse_line(self, line):
+ '''This function should digest the contents of one line at a time, updating
+ object's state variables. Takes a single argument, the line to be parsed.'''
+
+ try:
+ # Apply regular expression to each line and extract interesting bits.
+ regMatch = self.reg.match(line)
+
+ if regMatch:
+ linebits = regMatch.groupdict()
+ status = int(linebits['http_status_code'])
+
+ if (status < 200):
+ self.http_1xx += 1
+ elif (status < 300):
+ self.http_2xx += 1
+ elif (status < 400):
+ self.http_3xx += 1
+ elif (status < 500):
+ self.http_4xx += 1
+ else:
+ self.http_5xx += 1
+
+ else:
+ raise LogsterParsingException, "regmatch failed to match"
+
+ except Exception, e:
+ raise LogsterParsingException, "regmatch or contents failed with %s" % e
+
+
+ def get_state(self, duration):
+ '''Run any necessary calculations on the data collected from the logs
+ and return a list of metric objects.'''
+ self.duration = duration
+
+ # Return a list of metrics objects
+ return [
+ GraphiteMetricObject("http_1xx", (self.http_1xx / self.duration)),
+ GraphiteMetricObject("http_2xx", (self.http_2xx / self.duration)),
+ GraphiteMetricObject("http_3xx", (self.http_3xx / self.duration)),
+ GraphiteMetricObject("http_4xx", (self.http_4xx / self.duration)),
+ GraphiteMetricObject("http_5xx", (self.http_5xx / self.duration)),
+ ]
+
Oops, something went wrong.

0 comments on commit 07e3b6a

Please sign in to comment.