Varnish log collector with Apache Kafka integration. This is a mirror from https://gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing.
C Makefile Emacs Lisp
Latest commit e16e908 Sep 19, 2016 @elukey elukey Improve resilience during varnish (re)starts
This patch introduces a trick already used in Varnish's VUT library,
namely forcing a reconnect to the shm log if it is detected as
abandoned or not opened. This can happen if varnishkafka starts before varnish,
or if varnish restarts/stops while varnishkafka is running. Varnishkafka is now
going to retry every 100ms endlessly until a log is aquired.

Background reading:
- https://github.com/varnishcache/varnish-cache/blob/4.1/lib/libvarnishtools/vut.c#L366
- https://github.com/varnishcache/varnish-cache/blob/master/bin/varnishlog/varnishlog.c

To keep things consistent with the current behavior, the sequence number is restarted to its
starting value when a log overrun/abandon is detected.

Bug: T138747
Change-Id: I87048786b36de325aee663a873b12874588664fb

README.md

varnishkafka - varnish log collector with Apache Kafka integration

Copyright (c) 2013 Wikimedia Foundation

Copyright (c) 2013 Magnus Edenhill

https://github.com/wikimedia/varnishkafka

Description

varnishkafka is a varnish log collector with an integrated Apache Kafka producer. It was written from scratch with performance and modularity in mind.

Supported outputs and formats

Currently supported outputs:

  • kafka - produce messages to one or more Apache Kafka brokers
  • stdout - write log messages to stdout (as varnishncsa does)

Currently supported output formats:

  • string - arbitrary text output according to the configured formatting.
  • json - output as JSON with configurable field, field types and names.

New formats and outputs can easily be added.

Varnish 4 compatibility

Varnishkafka is fully compatible with the new Varnish 4 API, but since they changed a lot from the previous version we had to break compatibility with Varnish 3. If you want to use Varnishkafka with Varnish 3, please check the related branch. New features will be added only to this branch from now on.

Configuration

varnishkafka is configured with a configuration file and is designed to operate as a system daemon.

Please see the configuration file example, varnishkafka.conf.example, for a full description of configuration properties.

The standard Varnish VSL command line arguments are supported, both through the command line and configuration file.

License

varnishkafka is licensed under the 2-clause BSD license.

The Apache Kafka support is provided by librdkafka

Usage

Requirements

libvarnishapi
librdkafka
libyajl
pthreads
zlib
libm

Instructions

Building

  make
  sudo make install
  # or to install in another location than /usr/local, set DESTDIR env
  # to the filesystem root of your choice.
  sudo make DESTDIR=/usr make install

Run

# If /etc/varnishkafka.conf exists
varnishkafka

# Alternative configuration path
varnishkafka -S /path/to/varnishkafka.conf

# Read a specific log file
varnishkafka -S /path/to/varnishkafka.conf -N /path/to/_.vsm

# Read a specific Varnish instance log file
varnishkafka -S /path/to/varnishkafka.conf -n varnish_instance_name

# Usage description
varnishkafka -h