zbx-statsd is a fork of pistolero's zbx-statsd (which is a clone of Etsy's statsd and Steve Ivy's py-statsd), designed to feed stats from logstash's statsd to Zabbix (http://zabbix.com)
Clone or download
Pull request Compare This branch is 17 commits ahead of pistolero:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
README.md
__init__.py
client.py
configfile.py.example
daemon.py
server.py

README.md

Introduction

ls-zbxstatsd is a fork of pistolero's zbx-statsd, which is a clone of Etsy's statsd and Steve Ivy's py-statsd. It is designed to accept statsd output from logstash and output to Zabbix for stats collection and graphing.

Dependencies:

Features:

  • Automatic item creation

    • Will use API calls to automatically create items with:
      • Application Name
      • History
      • Per key/trunk Units (e.g. s for seconds, B for bytes, etc).
      • Per key/trunk Value multiplier (e.g. 0.001 for milliseconds to seconds)
  • Running table of host/key pairings

    • A python dictionary of all known host/key pairings is compared when new values come in. No unnecessary API calls.
    • Stored values from db are added at start-up time.
  • SQLite3 database stores host/key pairings

    • Stored values are loaded into running table at start-up time.
    • As new values are added to the running table, they are also written to the db.
  • Configuration file directives

    • API login credentials
    • SQLite3 Host/Key pairing database file path. This prevents attempted re-add of already created items on startup.
    • Zabbix application name
    • ls-zbxstatsd commandline options (so you don't have to!)
  • Works with Zabbix proxy for stats collection

    • May need to wait for proxy to pick up new item configuration, based on zabbix_proxy.conf settings, before new values will show up.
    • Still need access to the API to create items, so this won't work in a completely firewalled/isolated environment

Attribution:

Usage:

Note: The default value for PORT is 8126, rather than the normal statsd value of 8125.

$ python server.py --help
usage: server.py [-h] [-d] [-n NAME] [-p PORT] [--zabbix-port ZABBIX_PORT]
                 [--zabbix-host ZABBIX_HOST] [-l LOG_FILE] [-f FLUSH_INTERVAL]
                 [-t PCT] [-D] [--pidfile PIDFILE] [--restart] [--stop]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           debug mode
  -n NAME, --name NAME  hostname to run on
  -p PORT, --port PORT  port to run on
  --zabbix-port ZABBIX_PORT
                        port to connect to zabbix on
  --zabbix-host ZABBIX_HOST
                        host to connect to zabbix on
  -l LOG_FILE           log file
  -f FLUSH_INTERVAL, --flush-interval FLUSH_INTERVAL
                        interval between flushes
  -t PCT, --pct PCT     stats pct threshold
  -D, --daemon          daemonize
  --pidfile PIDFILE     pid file
  --restart             restart a running daemon
  --stop                stop a running daemon

Typical launch command (if you've put the options in configfile.py):

$ python server.py -D

Launch command (with command-line overrides):

$ python server.py -D -n ls-zbxstatsd.example.com --zabbix-host zabbix-server.example.com -l /path/to/logfile.log

Logstash Configuration:

Logstash statsd information can be found here: http://logstash.net/docs/1.1.9/outputs/statsd

Note: Do not alter the namespace! This script expects the default "logstash." The code could be hacked to be namespace independent, but isn't at present.

Instructions:

sender: In this example a pre-existing field called zabbix_host is used. This can be a hard-coded string or a field value. In any case, it MUST be an existing host in Zabbix, and it MUST have the double semicolon post-pended. The reason for this is that period delimiting doesn't work if your Zabbix host names are FQDNs. How will the script know? Double semicolons. Miss this detail and the script will not work.

DOUBLE SEMICOLONS. 'nuff said.

The Zabbix key names will be the fields you specify here, e.g. apache.bytes, apache.status[200] (or any other valid HTTP response code), apache.duration

Example:

  statsd {
    type => "apache"
    count => [ "apache.bytes", "%{bytes}" ]
    increment => "apache.status[%{status}]"
    timing => [ "apache.duration", "%{duration}" ]
    sender => "%{zabbix_host};;" # DOUBLE SEMICOLONS.  You have been warned :)
    host => "statsd.example.com" # The host where ls-zbxstatsd will be running
    port => 8126
  }

ls-zbxstatsd Configuration:

configfile.py is imported like a standard python module, rendering all of its values as regular variables in the context they were imported into.

### Zabbix API
url = "http://www.example.com/zabbix"
username = "username"
password = "password"

### Host/Key pairing database
dbfile = '/tmp/zbxstatsd.db'

### Zabbix Item Constants
applications = ["Log Metrics" ]
history = 30
units = { "apache.bytes" : "B", "apache.duration" : "s" }
formulas = { "apache.duration" : "0.000001" }

### Program Constants
itemrefresh = 86400
zabbix_host = "localhost"
zabbix_port = 10051
daemon_host = "localhost"
daemon_port = 8126
flush_interval = 10000
percentage = 90
debug = False
logfile = '/tmp/ls-zbxstatsd.log'
pidfile = '/tmp/ls-zbxstatsd.pid'

itemrefresh is the time in seconds between checks back to Zabbix to verify the item exists. Defaults to 1 day (86400 seconds). Can be increased here as desired.

zabbix_host is actually the Zabbix server where the statistics will be sent via the zabbix_sender protocol.

daemon_host sets the IP on which ls-zbxstatsd will listen for statsd traffic, and daemon_port the port.

The units and formulas dictionaries allow for per zabbix-key item differentiation, e.g. duration is in microseconds in the apache output, so I need to set the multiplier to 0.000001 and set units to s for seconds. Have more keys? Add them here.

flush_interval is in milliseconds, so 10000 = 10 seconds.