Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Varnish log collector with Apache Kafka integration.
C Emacs Lisp
Branch: master

allow zero-length scratch adds as well

(and fix -Werror failing out for overflow issues
with other ways of writing that assert...)

Change-Id: I796844f82fef1f475a134741c3267ea1227bf936
latest commit ddcdacebab
@blblack blblack authored
Failed to load latest commit information.
.dir-locals.el
.gitignore
.gitreview Add .gitreview
LICENSE
Makefile
README.md
base64.c
base64.h
config.c
varnishkafka.c
varnishkafka.conf.example Added %D and %T formatters (request time)
varnishkafka.h

README.md

varnishkafka - varnish log collector with Apache Kafka integration

Copyright (c) 2013 Wikimedia Foundation

Copyright (c) 2013 Magnus Edenhill

https://github.com/wikimedia/varnishkafka

Description

varnishkafka is a varnish log collector with an integrated Apache Kafka producer. It was written from scratch with performance and modularity in mind, varnishkafka consumes about a third of the CPU that varnishncsa does and has a far more frugal memory approach.

Supported outputs and formats

Currently supported outputs:

  • kafka - produce messages to one or more Apache Kafka brokers
  • stdout - write log messages to stdout (as varnishncsa does)

Currently supported output formats:

  • string - arbitrary text output according to the configured formatting.
  • json - output as JSON with configurable field, field types and names.

New formats and outputs can easily be added.

Configuration

varnishkafka is configured with a configuration file and is designed to operate as a system daemon.

Please see the configuration file example, varnishkafka.conf.example, for a full description of configuration properties.

The standard Varnish VSL command line arguments are supported, both through the command line and configuration file.

License

varnishkafka is licensed under the 2-clause BSD license.

The Apache Kafka support is provided by librdkafka

Usage

Requirements

libvarnishapi
librdkafka
libyajl
pthreads
zlib

Instructions

Building

  make
  sudo make install
  # or to install in another location than /usr/local, set DESTDIR env
  # to the filesystem root of your choice.
  sudo make DESTDIR=/usr make install

Run

# If /etc/varnishkafka.conf exists
varnishkafka

# Alternative configuration path
varnishkafka -S /path/to/varnishkafka.conf

# Read offline log file
varnishkafka -r varnishlog.vsl

# Usage description
varnishkafka -h
Something went wrong with that request. Please try again.