A sink for Apache Flume designed to couple with the existing Spooling Directory Source
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
.travis.yml
LICENSE
README.md
pom.xml

README.md

flume-spooling-directory-sink

Build Status

An Apache Flume sink that spools to a directory and periodically rotates the temporary files into another directory.

Example config

...
agent.sinks.directoryFileSink.type = com.unrulymedia.flume.SpoolingDirectoryFileSink
agent.sinks.directoryFileSink.channel = someChannel
agent.sinks.directoryFileSink.sink.directory = target/
agent.sinks.directoryFileSink.sink.rollInterval = 5
...

Motivation

The built-in flume SpoolingDirectorySource does not have an inverse sink (as the FileSink does not work in this way) so the SpoolingDirectoryFileSink is an implementation of this.

This enables us to easily create Flume topologies with spooling reliability in-between for resiliency.

Installation

Create a jar with mvn package and include in the flume plugins.d directory.

License

MIT