Flume syslog source 2
This project defines a plugin for Flume containing an improved Syslog source.
It parses RFC 3164 (BSD Syslog) as well as the newer RFC 5424 (Syslog.) The source goes to a greater length to create a Flume event than the built-in version. All fields possible are parsed, and the body is just the body of the message. Even the Tag and PID fields are parsed. This should leave clean and normalized log messages, at the expense of more CPU cycles.
The data is interpreted as ASCII octets where possible. The notable exception is the hostname, which Flume stores as a String, and must thus be decoded. Where applicable, decoding to Unicode assumes the octet stream is encoded in UTF-8.
mvn package. The produced Jar should be installed like any Flume
plugin, and the plugin class to be added to
flume.plugin.classes in the
configuration file is called
The source is accessed using the name
syslog2, and allows both TCP and UDP
For TCP, the syntax is
syslog2(tcp[, host[, port]])
and for UDP
syslog2(udp[, host[, port[, bufferSize]]])
The default port is 514, the well-known Syslog port. The buffer size (which is the maximum size of an incoming UDP packet) is 64 kB unless overridden.
Note that the structured data of RFC 5424 is parsed, but not split into multiple fields. This is for space effiency. If you need it split, add a sink decorator for it.
The code hasn’t been tested in Flume yet.
SyslogParser class is the main class and implements a recursive decent
parser based on an
InputStream. Most bytes are handled without concern for
encoding, for speed and robustness. The philosophy is; it’s better to get the
data into Flume, than rejecting it. This might not be suitable for you, but
then you have the option of writing a decorator to handle that.
Copyright 2011 Spotify LtdLicensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.