varnishkafka - varnish log collector with Apache Kafka integration
Copyright (c) 2013 Wikimedia Foundation
Copyright (c) 2013 Magnus Edenhill
varnishkafka is a varnish log collector with an integrated Apache Kafka producer. It was written from scratch with performance and modularity in mind, varnishkafka consumes about a third of the CPU that varnishncsa does and has a far more frugal memory approach.
Supported outputs and formats
Currently supported outputs:
- kafka - produce messages to one or more Apache Kafka brokers
- stdout - write log messages to stdout (as varnishncsa does)
Currently supported output formats:
- string - arbitrary text output according to the configured formatting.
- json - output as JSON with configurable field, field types and names.
New formats and outputs can easily be added.
varnishkafka is configured with a configuration file and is designed to operate as a system daemon.
Please see the configuration file example, varnishkafka.conf.example, for a full description of configuration properties.
The standard Varnish VSL command line arguments are supported, both through the command line and configuration file.
varnishkafka is licensed under the 2-clause BSD license.
The Apache Kafka support is provided by librdkafka
libvarnishapi librdkafka libyajl pthreads zlib
make sudo make install # or to install in another location than /usr/local, set DESTDIR env # to the filesystem root of your choice. sudo make DESTDIR=/usr make install
# If /etc/varnishkafka.conf exists varnishkafka # Alternative configuration path varnishkafka -S /path/to/varnishkafka.conf # Read offline log file varnishkafka -r varnishlog.vsl # Usage description varnishkafka -h