Mirror of Apache Flume
Java PowerShell Shell Python Thrift Protocol Buffer
Latest commit 1e8f265 Aug 25, 2016 @granthenke granthenke committed with mpercy FLUME-2972. Handle offset migration in the new Kafka Channel
Offsets tracking the position in Kafka consumers change from using
Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to
0.9.x.

FLUME-2823 makes the client change in the Kafka Channel but does not
ensure existing offsets get migrated in order to continue consuming
where it left off.

Flume should have some automated logic on startup to check if Kafka
offsets exist, if not and migration is enabled (by default) then copy
the offsets from Zookeeper and commit them to Kafka.

Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy

(Grant Henke via Mike Percy)
Permalink
Failed to load latest commit information.
bin FLUME-2913. Don't strip SLF4J from imported classpaths May 27, 2016
conf FLUME-2148. Windows : Add flume-env.ps1 Apr 14, 2015
dev-support FLUME-1787. Implement Pre-commit testing with Jenkins. Mar 22, 2013
flume-checkstyle FLUME-2959. Fix issues with flume-checkstyle module Jul 27, 2016
flume-ng-auth FLUME-2941. Integrate checkstyle for test classes Jul 8, 2016
flume-ng-channels FLUME-2972. Handle offset migration in the new Kafka Channel Aug 25, 2016
flume-ng-clients Fix various typos Jul 29, 2016
flume-ng-configuration Fix various typos Jul 29, 2016
flume-ng-core FLUME-2974. Fix some broken SpoolDirSource unit tests Aug 22, 2016
flume-ng-dist FLUME-2498. Implement Taildir Source Aug 18, 2015
flume-ng-doc FLUME-2972. Handle offset migration in the new Kafka Channel Aug 25, 2016
flume-ng-embedded-agent FLUME-2941. Integrate checkstyle for test classes Jul 8, 2016
flume-ng-legacy-sources FLUME-2941. Integrate checkstyle for test classes Jul 8, 2016
flume-ng-node Fix various typos Jul 29, 2016
flume-ng-sdk Fix various typos Jul 29, 2016
flume-ng-sinks Fix various typos Jul 29, 2016
flume-ng-sources Fix various typos Jul 29, 2016
flume-ng-tests FLUME-2952. SyslogAgent: Avoid NPE on stop() Jul 14, 2016
flume-tools FLUME-2941. Integrate checkstyle for test classes Jul 8, 2016
.gitignore FLUME-2206. ElasticSearchSink ttl field modification to mimic Elastic… Oct 31, 2013
CHANGELOG Fix various typos Jul 29, 2016
DEVNOTES Add MAVEN_OPTS build workaround to DEVNOTES for 1.2.0 Jul 11, 2012
LICENSE FLUME-2683. Update License file for 1.6 release Apr 20, 2015
NOTICE Updated copyright year in NOTICE file. Jul 8, 2012
README.md Fix broken link in README Jul 19, 2016
RELEASE-NOTES FLUME-2676. Update flume docs and release notes to 1.7-SNAPSHOT in tr… Apr 16, 2015
doap_Flume.rdf FLUME-2365. Add DOAP file for Flume. Jun 13, 2014
pom.xml FLUME-2941. Integrate checkstyle for test classes Jul 8, 2016

README.md

Welcome to Apache Flume!

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic application.

The Apache Flume 1.x (NG) code line is a refactoring of the first generation Flume to solve certain known issues and limitations of the original design.

Apache Flume is open-sourced under the Apache Software Foundation License v2.0.

Documentation

Documentation is included in the binary distribution under the docs directory. In source form, it can be found in the flume-ng-doc directory.

The Flume 1.x guide and FAQ are available here:

Contact us!

Bug and Issue tracker.

Compiling Flume

Compiling Flume requires the following tools:

  • Oracle Java JDK 1.7
  • Apache Maven 3.x

Note: The Apache Flume build requires more memory than the default configuration. We recommend you set the following Maven options:

export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

To compile Flume and build a distribution tarball, run mvn install from the top level directory. The artifacts will be placed under flume-ng-dist/target/.