A syslog daemon for producing messages to Apache Kafka.
This code is built with the following assumptions. You may get mixed results if you deviate from these versions.
- Protocol Buffers
- Zookeeper (for Kafka)
You will need to setup Google's Protocol Buffers in order for this project to compile and run. Since this is fairly well documented and straightforward I'll leave that up to the user. Most platforms that have a descent packaging system will probably have a protobuf package already. If you're on a Mac using Homebrew this is as easy as:
brew install protobuf
You will also need Apache Kafka. As of this writing the latest is Kafka 0.7.2 but the project is still in the incubation stage. Unfortunately this means they can't push an official Apache Kafka jar to Maven repositories. So you'll need to add it to your environment manually. This can be done in the following steps:
curl https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz -o kafka-0.7.2-incubating-src.tgz tar xvzf kafka-0.7.2-incubating-src.tgz cd kafka-0.7.2-incubating-src ./sbt update ./sbt package cd ./core/target/scala_2.8.0/ mvn install:install-file -Dfile=kafka-0.7.2.jar -DgroupId=org.apache.kafka -DartifactId=kafka-core -Dversion=0.7.2-incubating -Dpackaging=jar
If everything went okay you should now have a kafka-0.7.2.jar in your local Maven repository.
To make a jar you can do:
The jar file is then located under
Running an instance
Make sure your Kafka and Zookeeper servers are running first (see Kafka documentation)
In order to run syslog-kafka on another machine you will probably want to use the dist assembly like so:
The zip file now under the
target directory should be deployed to
SYSLOGKAFKA_HOME on the remote server.
** MORE DOCUMENTATION TO FOLLOW. THIS IS A WORK IN PROGRESS IN MY SPARE TIME. **
Example Kafka Producer Configuration (conf/kafka.producer.properties)
# comma delimited list of ZK servers zk.connect=127.0.0.1:2181 # use syslog message encoder serializer.class=kafka.serializer.SyslogMessageEncoder # asynchronous producer producer.type=async # compression.code (0=uncompressed,1=gzip,2=snappy) compression.codec=2 # batch size (one of many knobs to turn in kafka depending on expected data size and request rate) batch.size=100
All aspects of this software are distributed under Apache Software License 2.0. See LICENSE file for full license text.
- Xavier Stevens (@xstevens)