Permalink
Browse files

Initial code check-in

  • Loading branch information...
xstevens committed Mar 10, 2013
1 parent 8e59e52 commit a1db85a44b3009ee97beacbe77a8d97b13d38cfe
View
@@ -0,0 +1,52 @@
+build
+target
+.project
+.classpath
+.settings
+
+# Compiled source #
+###################
+*.com
+*.class
+*.dll
+*.exe
+*.o
+*.so
+
+# Packages #
+############
+# it's better to unpack these files and commit the raw source
+# git has its own built in compression methods
+*.7z
+*.dmg
+*.gz
+*.iso
+*.rar
+*.tar
+*.zip
+
+# Logs and databases #
+######################
+logs
+*.log
+*.sql
+*.sqlite
+
+# OS generated files #
+######################
+.DS_Store?
+ehthumbs.db
+Icon?
+Thumbs.db
+
+# IntelliJ Idea project files
+.idea
+*.iml
+
+# Clover
+.clover
+report
+
+# Editor files
+.*.sw*
+*~
View
@@ -1,4 +1,72 @@
-syslog-kafka
-============
+# syslog-kafka #
-A daemon to transfer syslog messages to Apache Kafka.
+Version: 0.1-SNAPSHOT
+
+#### A syslog daemon for producing messages to Apache Kafka. ####
+
+
+### Version Compatability ###
+This code is built with the following assumptions. You may get mixed results if you deviate from these versions.
+
+* [Syslog4j](http://www.syslog4j.org/) 0.9.30
+* [Kafka](http://incubator.apache.org/kafka) 0.7.2
+* [Protocol Buffers](https://developers.google.com/protocol-buffers) 2.4.1
+
+### Prerequisites ###
+* Protocol Buffers
+* Zookeeper (for Kafka)
+* Kafka
+
+### Setup ###
+You will need to setup Google's Protocol Buffers in order for this project to compile and run. Since this is fairly well documented and straightforward I'll leave that up to the user. Most platforms that have a descent packaging system will probably have a protobuf package already. If you're on a Mac using Homebrew this is as easy as:
+
+`brew install protobuf`
+
+You will also need Apache Kafka. As of this writing the latest is Kafka 0.7.2 but the project is still in the incubation stage. Unfortunately this means they can't push an official Apache Kafka jar to Maven repositories. So you'll need to add it to your environment manually. This can be done in the following steps:
+
+ curl https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz -o kafka-0.7.2-incubating-src.tgz
+ tar xvzf kafka-0.7.2-incubating-src.tgz
+ cd kafka-0.7.2-incubating-src
+ ./sbt update
+ ./sbt package
+ cd ./core/target/scala_2.8.0/
+ mvn install:install-file -Dfile=kafka-0.7.2.jar -DgroupId=org.apache.kafka -DartifactId=kafka-core -Dversion=0.7.2-incubating -Dpackaging=jar
+
+If everything went okay you should now have a kafka-0.7.2.jar in your local Maven repository.
+
+### Building ###
+To make a jar you can do:
+
+`mvn package`
+
+The jar file is then located under `target`.
+
+### Running an instance ###
+**Make sure your Kafka and Zookeeper servers are running first (see Kafka documentation)**
+
+In order to run syslog-kafka on another machine you will probably want to use the _dist_ assembly like so:
+
+`mvn assembly:assembly`
+
+The zip file now under the `target` directory should be deployed to `SYSLOGKAFKA_HOME` on the remote server.
+
+** MORE DOCUMENTATION TO FOLLOW. THIS IS A WORK IN PROGRESS IN MY SPARE TIME. **
+
+### Example Kafka Producer Configuration (conf/kafka.producer.properties) ###
+ # comma delimited list of ZK servers
+ zk.connect=127.0.0.1:2181
+ # use syslog message encoder
+ serializer.class=kafka.serializer.SyslogMessageEncoder
+ # asynchronous producer
+ producer.type=async
+ # compression.code (0=uncompressed,1=gzip,2=snappy)
+ compression.codec=2
+ # batch size (one of many knobs to turn in kafka depending on expected data size and request rate)
+ batch.size=100
+
+### License ###
+All aspects of this software are distributed under Apache Software License 2.0. See LICENSE file for full license text.
+
+### Contributors ###
+
+* Xavier Stevens ([@xstevens](http://twitter.com/xstevens))
View
@@ -0,0 +1,73 @@
+#!/bin/bash
+
+function usage() {
+ echo "Usage: $0 <server-port>"
+ exit 1
+}
+
+# Print usage if incorrect number of args
+[[ $# -ne 1 ]] && usage
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+SERVER_PORT=$1
+SERVER_CLASS_NAME="kafka.syslog.SyslogKafkaServer"
+NOW=`date "+%Y%m%d%H%M%S"`
+SYSLOGKAFKA_OPTS="-Dserver.port=$SERVER_PORT -Dsyslogkafka.log.dir=$bin/../logs"
+GC_OPTS="-XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:$bin/../logs/gc.log.$NOW"
+JAVA_OPTS="$SYSLOGKAFKA_OPTS -Xmx64m -XX:+UseCompressedOops $GC_OPTS"
+
+export MALLOC_ARENA_MAX=2
+
+if [ "$SYSLOGKAFKA_USER" = "" ]; then
+ SYSLOGKAFKA_USER="$USER"
+fi
+
+if [ -d "/var/run/syslogkafka" ]; then
+ PIDFILE="/var/run/syslogkafka/syslogkafka.pid"
+else
+ PIDFILE="$bin/../syslogkafka.pid"
+fi
+
+# if this is a developer then use the main jar in the build directory
+if [ -d $bin/../target ]; then
+ MAIN_JAR_PATH="$bin/../target/syslog-kafka-*.jar"
+ if [ "$DAEMON_DETACHED" = "" ]; then
+ DAEMON_DETACHED=false
+ fi
+else
+ MAIN_JAR_PATH="$bin/../syslog-kafka-*.jar"
+ if [ "$DAEMON_DETACHED" = "" ]; then
+ DAEMON_DETACHED=true
+ fi
+fi
+
+CLASSPATH="$bin/../conf"
+# add main jar
+for lib in `ls $MAIN_JAR_PATH`; do
+ CLASSPATH=${CLASSPATH}:$lib
+done
+
+# add dependency libs
+for lib in `ls $bin/../lib/*.jar`; do
+ CLASSPATH=${CLASSPATH}:$lib
+done
+
+# create logs dir if it doesn't exist
+if [ ! -d $bin/../logs ]; then
+ mkdir -p $bin/../logs
+fi
+
+if [ "$DAEMON_DETACHED" = false ]; then
+ java $JAVA_OPTS -cp $CLASSPATH $SERVER_CLASS_NAME
+ RETVAL=$?
+else
+ nohup java $JAVA_OPTS -cp $CLASSPATH $SERVER_CLASS_NAME > $bin/../logs/syslogkafka.out 2>&1 < /dev/null &
+ PID=$!
+ RETVAL=$?
+
+ echo $PID > $PIDFILE
+fi
+
+exit $RETVAL
@@ -0,0 +1,6 @@
+#zk.connect=localhost:2181
+broker.list=0:localhost:9092
+serializer.class=kafka.serializer.SyslogMessageEncoder
+producer.type=async
+compression.codec=2
+batch.size=100
View
@@ -0,0 +1,21 @@
+#log4j.debug=true
+
+log4j.rootCategory=INFO, stdout, DRFA
+
+# Stdout
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging pattern
+#log4j.appender.stdout.layout.ConversionPattern=[%d{ISO8601}]%5p%6.6r[%t]%x - %C.%M(%F:%L) - %m%n
+
+# File
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.DRFA.File=${syslogkafka.log.dir}/syslogkafka.log
+log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+log4j.appender.DRFA.Append=true
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging Pattern format
+#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
+
Oops, something went wrong.

0 comments on commit a1db85a

Please sign in to comment.