Common Event Format (CEF) Interceptor for Apache Flume (License Apache 2.0)
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
.travis.yml
LICENSE
README.md
pom.xml
settings.xml

README.md

CEF (Common Event Format)

Common Event Format Interceptor for Apache Flume. Use this interceptor to parse and convert CEF payloads in Syslog to Flume Event headers.

Common Event Format commonly known as CEF (pronounced sef), is the defacto format of HP ArcSight Logging and Security monitoring product family (including Loggers and ESM), you can read more about CEF here: https://www.protect724.hpe.com/docs/DOC-1072. It's also widely adopted by a few security products making parsing it critical to interpreting the logs and making sense out of them. CEF is usually transmitted over Syslog protocol as a payload.

This parser is a available in form of an Apache Flume Interceptor (https://flume.apache.org/FlumeUserGuide.html#flume-interceptors) and is best used in conjunction with SyslogSources (TCP / UDP) however you can use it with any other Flume Source as long as the byte[] body of emitted from the Flume Source is CEF only.

Note: This parser currently doesn't validate events beyond CEF Prefix and Invalid extensions. It currently doesn't check dictionary validation or data type validation for dictionary items. This is partly because Flume headers are String valued therefore type conversion has no benefit and as well as for performance reasons.

Compliance

This version of CEF Interceptor is complaint with Apache Flume 1.6.0

How to use?

Please download the published artifact from Maven Central, 0.0.11 or greater is current stable version.

<dependency>
    <groupId>com.srotya.flume</groupId>
    <artifactId>cef</artifactId>
    <version>0.0.11</version>
</dependency>

The parser has no external dependencies so all you need to do is download the latest JAR from Maven Central to the lib folder of your Flume installation and configure the interceptor as shown below and start sending some CEF logs:

agent.sources = syslog
agent.channels = memoryChannel
agent.sinks = loggerSink
agent.sources.syslog.type = syslogtcp
agent.sources.syslog.port = 1514
agent.sources.syslog.interceptors = cef
agent.sources.syslog.interceptors.cef.type = com.srotya.flume.cef.CEFInterceptor$Builder
agent.sources.syslog.channels = memoryChannel
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

Bugs and Issues?

Please report bugs via github issues.