The default value is false, the hidden file will not be tailed when set true

If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the last block might
not end up in COMPLETE state. In this case block recovery should happen but as the lease is
still held by Flume the NameNode will start the recovery process only after the hard limit of
1 hour expires.

This change adds an explicit recoverLease() call in case of close failure.

This closes #127

Reviewers: Hari Shreedharan

(Denes Arvay via Bessenyei Balázs Donát)

This closes #126

Reviewers: Denes Arvay, Bessenyei Balázs Donát

(Marcell Hegedus via Bessenyei Balázs Donát)

When logging level is set to DEBUG, Kafka Sink and Kafka Channel may throw a NullPointerException.

This patch ensures that `metadata` is not null to avoid the exception.

This closes #125

Reviewers: Denes Arvay, Bessenyei Balázs Donát

(loleek via Bessenyei Balázs Donát)

…r Source

This patch addresses an edge case of the Taildir Source wherein it can miss
reading events written in the same second as the file closing.

This closes #128

Reviewers: Satoshi Iijima, Bessenyei Balázs Donát

(eskrm via Bessenyei Balázs Donát)

…d to data loss

This commit fixes the issue when in HDFSEventSink.process() a BucketWriter.append()
call threw a BucketClosedException then the newly created BucketWriter wasn't
flushed after the processing loop.

This closes #129

Reviewers: Attila Simon, Mike Percy

(Denes Arvay via Mike Percy)

This patch adds the following new metrics to the FileChannel's counters:
- eventPutErrorCount: incremented if an IOException occurs during put operation.
- eventTakeErrorCount: incremented if an IOException or CorruptEventException occurs
  during take operation.
- checkpointWriteErrorCount: incremented if an exception occurs during checkpoint write.
- unhealthy: this flag represents whether the channel has started successfully
  (i.e. the replay ran without any problem), so the channel is capable for normal operation
- closed flag: the numeric representation (1: closed, 0: open) of the negated open flag.

Closes #131.

Reviewers: Attila Simon, Mike Percy

(Denes Arvay via Mike Percy)

…Sink

This patch adds the ability of header substitution n Kafka Sink's
kafka.topic configuration variable.

This closes #137.

Reviewers: Denes Arvay

(Takafumi Saito via Denes Arvay)

- Removed the unsupported PermSize and MaxPermSize settings from .travis.yml
- Updated DEVNOTES, README and Flume User Guide
- Removed the maven-compiler-plugin from the taildir-source subproject
- Changed the sourceJavaVersion and targetJavaVersion to 1.8 in the root pom.xml

(Lior Zeno via Denes Arvay)

Log4jAppender and LoadBalancingLog4jAppender resolve local hosts address at startup and
add it to each event's header with the key "flume.client.log4j.address".

This closes #121.

(Andras Beni via Denes Arvay)

JMSSource has created only nondurable subscriptions which could lead to event loss in case
of topic destination type.

This change enables durable subscription creation and lets user specify client id.
Also removed JMSMessageConsumerFactory which has no additional value.

This closes #120.

Reviewers: Attila Simon, Denes Arvay

(Andras Beni via Denes Arvay)

This commit changes the Travis CI build config to use Java 8.

This closes #142.

Reviewers: Denes Arvay

(Attila Simon via Denes Arvay)

Cleanup after Netty initialisation fails (call this.stop())

- Make sure this.stop() releases the resources and end up the component in
  a LifecycleAware.STOPPED state
- Added junit test to cover the invalid host scenario
- Added junit test to cover the used port scenario

This closes #141.

Reviewers: Denes Arvay

(Attila Simon via Denes Arvay)

This patch fixes the issue in NetcatSource which occurs if there is a problem
while binding the channel's socket to a local address and leads to a file descriptor
(socket) leak.

Reviewers: Attila Simon, Denes Arvay

(Siddharth Ahuja via Denes Arvay)

This patch adds a netcat UDP source.

Reviewers: Lior Zeno, Chris Horrocks, Bessenyei Balázs Donát

(Tristan Stevens via Bessenyei Balázs Donát)

Update Developer Guide with notes on how to upgrade Protocol Buffer
version.

Reviewers: Ashish Paliwal, Attila Simon

(Roshan Naik via Denes Arvay)

- Make avro ip filter tests more reliable by checking whether the
  caught exception is really what the test expected
- Use lambda instead of anonymous classes to make the code shorter

This closes #143.

Reviewers: Denes Arvay

(Attila Simon via Denes Arvay)

After a bad response, connection.getInputStream() returns null.
This patch adds a check for this.

This closes #139

Reviewers: Bessenyei Balázs Donát

(filippovmn via Bessenyei Balázs Donát)

Flume user guide does not specify whether a value in event header could be null or not.
Given an external system generating events which header values can be null and a user configures
Flume with Memory Channel then he will have no trouble.
Later on when the user changes Memory Channel to File Channel then Flume will fail with NPE.
It is because FC is serializing events with protocol buffer and header values are defined as
required in the proto file.
In this patch I have changed the value field to optional. However protocol buffer does not have
a notation for null and setting a field to null raises NPE again. Added a null check before
serialization to prevent this.
There is on caveat: When an optional field is not set, at deserialization it will be set to a
default value: in this case it will be empty string.

Reviewers: Miklos Csanady

(Marcell Hegedus via Denes Arvay)

…eringInterceptor

- Use RegexFilteringInterceptor.class in LoggerFactory.getLogger() call
- Fix the Javadoc of the RegexFilteringInterceptor.Builder class

This closes #148

Reviewers: Attila Simon, Marcell Hegedus

(Peter Chen via Denes Arvay)

This commit extracts the version numbers from the subprojects'
pom.xml to the root pom.xml without introducing any other change
(i.e. the dependency tree didn't change)

This closes #132

Reviewers: Ferenc Szabo, Attila Simon

(Miklos Csanady via Denes Arvay)

This closes #149

Reviewers: Denes Arvay

(Miklos Csanady via Denes Arvay)

…ollection of messages

Log4jAppender treats Collection messages as a special case making it possible to log
Collection of events in one Log4j log call. The appender sends these events to the
receiving Flume instance as one batch with the rpcClient.appendBatch() method.

This closes #151

Reviewers: Ferenc Szabo, Miklos Csanady

(Denes Arvay via Denes Arvay)

This change adds a new metric (channel.file.checkpoint.backup.write.error)
to the File Channel. It gets incremented if an exception happens
during backup checkpoints writes.

This closes #156

Reviewers: Denes Arvay

(Ferenc Szabo via Denes Arvay)

Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on
ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no native
library is found for os.name=Linux and os.arch=ppc64le" are seen
Native libraries for ppc64le were added in snappy-java version 1.1.1.
Hence Flume needs to have a higher version of snappy-java.

This closes #157

Reviewers: Ferenc Szabo, Attila Simon

(Pravin D'silva via Denes Arvay)

The Spring Framework libraries are transitive depencencies through ActiveMQ
thus it's not possible to upgrade.
They are only used is tests so moved ActiveMQ to test scope.

This closes #153

Reviewers: Attila Simon, Denes Arvay

(Ferenc Szabo via Denes Arvay)

Upgrading the version in dependency management and removing unused ones.
Both 1.x and 2.x jackson versions are needed, renamed the jackson.version
property to codehaus.jackson.version and added fasterxml.jackson.version
for the 2.x jackson.

This closes #154

Reviewers: Attila Simon, Miklos Csanady, Denes Arvay

(Ferenc Szabo via Denes Arvay)

Exclude the vulnerable jasper library transitive dependencies from hadoop, hbase
dependencies.

This closes #152

Reviewers: Attila Simon, Miklos Csanady

(Ferenc Szabo via Denes Arvay)

Reduce the output verbosity to fix the "log length has exceeded the limit of
4 MB" Travis CI error.

This closes #161

Reviewers: Ferenc Szabo

(Miklos Csanady via Denes Arvay)

…eSink

The current implementation of HBaseSink and AsyncHbaseSink is not
compatible with the 2.0 version of HBase, which will be released soon.
This change adds a check and makes these sinks fail gracefully if
incompatible HBase jars can be found in the classpath.

This closes #160

Reviewers: Bessenyei Balázs Donát, Ferenc Szabo, Denes Arvay

(Miklos Csanady via Denes Arvay)

…anager private API

This fix removes LeaseManager private API usages from tests in
TestHDFSEventSinkOnMiniCluster as these methods are expected to be
removed/changed in future versions of Hadoop/HDFS.

Test refactored, it has the same result and behaviour.

This closes #164

Reviewers: Denes Arvay

(Ferenc Szabo via Denes Arvay)

Minor change to fix an infinite recursion in
org.apache.flume.conf.source.SourceConfiguration.getSourceConfigurationType()

This closes #166

Reviewers: Denes Arvay

(TeddyBear1314 via Denes Arvay)

Change ki to k1 in Kafka Sink configuration example.
(a1.sinks.ki.kafka.producer.compression.type)

This closes #165

Reviewers: Bessenyei Balázs Donát, Denes Arvay

(Robin Wang via Denes Arvay)

- thrift version changed to 0.9.3
- thrift objects generated with the new version
- deprecated method calls updated in ScribeSource
- cross-compatibility tested with Flume 1.6 and 1.7

This closes #163

Reviewers: Denes Arvay

(Ferenc Szabo via Denes Arvay)

This change adds a configuration parameter to the TimestampInterceptor
for the user to be able to define the name of the timestamp header.

Reviewers: Tristan Stevens, Attila Simon

(Hari Shreedharan, Tristan Stevens, Attila Simon via Denes Arvay)

This closes #169

Reviewers: Marcell Hegedus

(Miklos Csanady via Denes Arvay)

This closes #170

Reviewers: Ferenc Szabo

(Denes Arvay via Denes Arvay)

This patch fixes the infinite loop between Kafka source and Kafka sink
by introducing the following configuration parameters in those components:
- topicHeader in Kafka source to specify the name of the header where it
  stores the topic name where the event comes from.
- setTopicHeader in Kafka source to control whether the topic name is stored
  in the given header.
- topicHeader in Kafka sink to configure the name of the header which
  is used to specify in which topic to send the event.
- allowTopicOverride in Kafka sink to control whether the target topic's name
  can be overridden by the specified header.

This closes #105

Reviewers: Attila Simon

(Tristan Stevens via Denes Arvay)

With Java 8's new doclint feature the javadoc generation breaks due to the
strict checking so adding -Xdoclint:none to the maven-javadoc-plugin's
configuration.

This closes #168

Reviewers: Bessenyei Balázs Donát, Ferenc Szabo

(Denes Arvay via Denes Arvay)

Display "2009-current year" instead of the hardcoded "2009-2012".

This closes #171

Reviewers: Ferenc Szabo

(Denes Arvay via Denes Arvay)

javax.jms has GNU GPL licence which is not compatible with Apache

This closes #172

Reviewers: Miklos Csanady, Denes Arvay

(Ferenc Szabo via Denes Arvay)

Adding geronimo-jms_1.1_spec jar to the Apache licensed list.

…ation files

This closes #69

(Cherry-picked from flume-1.7 branch by Denes Arvay)

flume-checkstyle breaks the assembly because its parent is not the flume-parent
Removing the moduleSets definition from the src assembly solved the issue.
Files are added based on fileSets, the resulting tarball's content equals
to the result of the dev-support/generate-source-release.sh in a clean
working directory.

This closes #175

Reviewers: Miklos Csanady, Ferenc Szabo

(Denes Arvay via Denes Arvay)

This patch changes the JdbcChannel tests to use in-memory derby to make parallel
test run possible.

This closes #177

Reviewers: Denes Arvay

(Ferenc Szabo via Denes Arvay)

…ume config

This patch adds the ability to set the asyncHbaseClient's config parameters via
the Flume configuration.

This closes #178

Reviewers: Ferenc Szabo, Denes Arvay

(Miklos Csanady via Denes Arvay)

Update Jetty version to 9.4.6.
Beside the version upgrade this patch exposes the new Jetty's configuration
variables on the HTTPSource to provide users the possibility of fine-tuning
the HTTPSource.

This closes #158

Reviewers: Ferenc Szabo, Mike Percy, Miklos Csanady, Denes Arvay

(Tristan Stevens via Denes Arvay)

Many unit tests use hardcoded port numbers which leads to flakiness and causes
problems when running builds in parallel.
This patch fixes this issue by searching for available ports instead of the
hardcoded ones.

This closes #124

Reviewers: Miklos Csanady, Ferenc Szabo

(Andras Beni via Denes Arvay)

This closes #181

Reviewers: Ferenc Szabo

(Ralph Goers via Denes Arvay)

KafkaChannelTest had quite a few test methods so during the build sometimes it
caused a timeout. Refactoring it to smaller test classes decrease the chance of
timeouts.

This closes #183

Reviewers: Miklos Csanady, Ferenc Szabo

(Viktor Somogyi via Ferenc Szabo)

Set surefire version to the latest and configure it
to rerun failed tests.
This way the flaky tests will not break the build as often.

This closes #182

Reviewers: Ferenc Szabo

(Miklos Csanady via Ferenc Szabo)

Current derby version contains security vulnerabilities.
This update will upgrade to a later version.

This closes #184

Reviewers: Ferenc Szabo

(Miklos Csanady via Ferenc Szabo)

…ctly

This closes #188

Reviewers: Tristan Stevens, Miklos Csanady

(Ferenc Szabo via Denes Arvay)

This change upgrades the netty library to 3.10.6 and also adds netty-all 4.1.17

This closes #155

Reviewers: Miklos Csanady

(Ferenc Szabo via Denes Arvay)

The only Guava class used in public API is the ImmutableMap in the

org.apache.flume.Context,
org.apache.flume.node.MaterializedConfiguration and
org.apache.flume.node.SimpleMaterializedConfiguration

classes.

This commit replaces these with java.util.Map.

Note: this is a breaking change as it modifies
the return type of a method on a public interface.

This closes #195

Reviewers: Ferenc Szabo, Mike Percy

(Denes Arvay via Ferenc Szabo)

…lume config

Adding an interface with 3 implementations to provide functionality at the
configuration level to replace variables/keys from external sources. This
component is capable of hiding sensitive information or injecting generated
data into the configuration.

The implementation affects only the configuration layer so existing components
(sinks/sources/channels/etc) do not have to change and new components can
already have it through the configuration.

New custom implementations can be easily added even in plugin form.

Each implementation has unit tests in their module and an integration test in
the flume-ng-tests module.

This closes #197

Reviewers: Miklos Csanady, Denes Arvay

(Ferenc Szabo via Denes Arvay)

… way

This closes #199

Reviewers: Denes Arvay

(Ferenc Szabo via Ferenc Szabo)

…topped while handleConfigurationEvent

Adding better locking mechanism to Application class to prevent deadlock.

this closes #108
this closes #144

Revievers: Denes Arvay, Attila Simon, Benedict Jin, Ferenc Szabo

(Andras Beni, Yan Jian via Ferenc Szabo)

…ul shutdown)

This closes #101

Reviewers: Bessenyei Balazs Donat, Ferenc Szabo

(Laxman Ch via Ferenc Szabo)

…erver without authentication

changing the default "" value of the password to null

Reviewers: Bessenyei Balazs Donat, Peter Turcsanyi, Ferenc Szabo

(Denes Arvay via Ferenc Szabo)

…leted

from the TAILDIR source

We fetch file names from a directory and later we fetch inodes.
If there is a delete between these operations this problem occurs.
Reproduced from unit test.
Added exception handling to handle this case.
It is enough to ignore the NoSuchFileException and continue.

this closes #208

reviewers: Ferenc Szabo, Peter Turcsanyi

(Endre Major via Ferenc Szabo)

HBase2Sink is the equivalent of HBaseSink for HBase version 2.
HBaseSink used some API calls which were deprecated in HBase 1.x
and they are not available in HBase 2.x any more.

HBase2Sink has been implemented by copying the existing
flume-ng-hbase-sink module to the new flume-ng-hbase2-sink module,
then adjusting the incompatible API calls to HBase 2.
The package and class names have also been modified to have
the hbase2/HBase2 tag. "Hbase" typos have been fixed too.

The functionality provided by HBase2Sink and the configuration parameters
are the same as in case of HBaseSink (except the hbase2 tag in the sink type
and the package/class names).

HBaseSink has not been modified, so it works with HBase 1.x as before.

This closes #209

Reviewers: Denes Arvay, Endre Major, Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

…n JMSSource

Handling RuntimeExceptions in the same way as JMSExceptions in order to trigger
the reconnecting mechanism in JMSSource.

This closes #210

Reviewers: Endre Major, Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

Currently the StressSource just runs flat out (on modern hardware
can generate 20M events per second with no problems),
relying on backpressure from the channel to regulate itself.

Adding a shaded version of the guava rate limiter and includes unit tests.

This closes #203

Reviewers: Ferenc Szabo, Endre Major, Peter Turcsanyi

(Tristan Stevens via Ferenc Szabo)

Added functionality to track files in the meta directory
rather than renaming them.
Improved tests for checking multilevel directories.

This closes #214

Reviewers: Ferenc Szabo, Peter Turcsanyi

(Endre Major via Ferenc Szabo)

By introducing error counters it will be easier to monitor problems.
Also errors are categorized, hopefully this will help setting up better
monitoring solutions.

Concept: an error is when an Exception is thrown or an ERROR level log is
written during event processing. In case of an error at least 1 error counter
is increased at least once. (Preferably 1 counter once).
Errors during event processing are counted.
Initialization errors are not handled here.
3 types of errors are differentiated.
- Channel read/write errors from the channel when the channel
  throws a ChannelException.
- Event read/write errors. E.g: A source cannot read an event due to
- Generic errors - e.g.: TaildirSource cannot write position file.

This closes #222

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)

Git ignored files are not checked anymore.
Files with licence error are logged to the console which is more accessable
during development than opening the report file

…hsize than

the channel transaction capacity

The loadSources() method seemed like an appropriate place to check this.
Added 2 new interfaces for getting the transaction capacity and the batch size
fields. The check is only done for channels that implement the
TransactioCapacitySupported interface and sources and sinks that implement
the BatchSizeSupported interface.

This closes #212

Reviewers: Ferenc Szabo, Peter Turcsanyi

(Endre Major via Ferenc Szabo)

This closes #224

Reviewers: Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

case of failure

This closes #227

Reviewers: Endre Major, Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

This PR is based on Yan Jian's fix and his test improvements.
Also contains the deadlock reproduction contributed by @adenes.
I have made minimal changes to those contributions.
Denes's test was used for checking the fix.
Yan's fix contains an optimization as it first calls the callback function
that removes the BucketWriter from the cache.
This is useful, should help to avoid some errors.

This closes #226

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major, Yan Jian, Denes Arvay via Ferenc Szabo)

It makes possible to specify global/common SSL keystore parameters (path,
password and type) at Flume agent (process) level for all sources/sinks.
In this way, it is not necessary to define (=copy) the SSL config for each
component in the agent config.

The global SSL parameters can be specified through the standard -D JSSE
system properties or in environment variables.
Component level configuration is still possible.

Priority:
 1. component parameters in agent config
 2. -D system properties
 2. environment variables

This closes #228

Reviewers: Ferenc Szabo, Tristan Stevens, Endre Major

(Peter Turcsanyi via Ferenc Szabo)

This is based on @mcsanady 's original pull request #202
I took the test changes from him but reworked the new feature implementation
since it failed some unit tests.
Previously when a close failed we immediately did a recover lease.
This PR introduces a background retry mechanism. It uses the already
existing "hdfs.closeTries" parameter. Unfortunately it has infinite retries
by default, that seems a bit too long for me.

I also did a minimal code clean up. The most important is that
HDFSWriter writer in BucketWriter became final. This is essential for later use
in inner classes. Only some testing solutions made it not final.
I reworked those to use the constructor.

This closes #229
This closes #202

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)

Adding SSL/TLS support for syslog and multi port syslog sources
This change also contains some refactoring to avoid code duplication in
sources with SSL/TLS support

As SSL/TLS handling is refactored it is convinient to
add the possibility to specify enabled protocol list FLUME-3275 and
add the possibility to specify enabled cipher suite list FLUME-3276

This closes #230

Reviewers: Peter Turcsanyi, Endre Major

(Ferenc Szabo via Ferenc Szabo)

Kafka client does not handle -D keystore parameters directly so Flume has to
pass them explicitly in Kafka properties (like ssl.keystore.location, etc).
Also using the same method for the truststore (in order to handle
keystore/truststore in the same way).

This closes #231

Reviewers: Denes Arvay

(Peter Turcsanyi via Ferenc Szabo)

…code analyzer

Adding spotbugs, pmd to the build.
moving flume-checkstyle to a new build-support module that contains
any future build tool configuration.

Fixed some trivial checkstyle issues.

Updated apache parent version and maven plugin versions.
Added maxAllowedViolations so this commit could contain only the new checks
and not any code change.

Did some cleanup in the pom files.

This closes #232

Reviewers: Peter Turcsanyi, Endre Major

(Ferenc Szabo via Ferenc Szabo)

Fixing the documentation

(Ferenc Szabo via Ferenc Szabo)

when using default "maxIOWorkers" value.

Reviewers: Denes Arvay, Ferenc Szabo

(Takafumi Saito via Ferenc Szabo)

In the newer version of the Syslog message format (RFC-5424) the hostname
is not a mandatory header anymore so the Syslog client might not send it.
On the Flume side it would be a useful information that could be used
in interceptors or for event routing.
To keep this information, two new properties have been added to the Syslog
sources: clientIPHeader and clientHostnameHeader.
Flume users can define custom event header names through these parameters
for storing the IP address / hostname of the Syslog client in the Flume
event as headers.
The IP address / hostname are retrieved from the underlying network sockets,
not from the Syslog message.

This change is based on the patch submitted by Jinjiang Ling which has been
rebased onto the current trunk and the review comments have been implemented.

This closes #234

Reviewers: Ferenc Szabo, Endre Major

(Peter Turcsanyi via Ferenc Szabo)

…n values.

Adding support for getFloat() and getDouble() on context

Reviewers: Mike Percy, Ferenc Szabo

(Hans Uhlig via Ferenc Szabo)

Reason: 2.8.9 has a vulnerability issue, fixed in 2.8.11+

This closes #236

Reviewers: Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

This is based on the contributions for FLUME-2653 regarding a new feature
for the hdfs sink.
Added a new parameter hdfs.emptyInUseSuffix to allow the output file name
to remain unchanged. See the user guide changes for details.
This is desired feature from the community.

I added a new junit test case for testing.
Temporarily modified old test cases in my ide to use the new flag, and
they passed. I did this just as one of test, to be on the safe side.
It is not in this PR.

This closes #237

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)

Changed http sink to log with slf4j instead of log4j.
Changed some test to use slf4j as well.

This closes #233

Reviewers: Peter Turcsanyi, Endre Major

(Ferenc Szabo via Ferenc Szabo)

This has been tested with unit tests. The main difference that caused the most
problems is the consumer.poll(Duration) change. This does not block even when
it fetches meta data whereas the previous poll(long timeout) blocked
indefinitely for meta data fetching.
This has resulted in many test timing issues. I tried to do minimal changes at
the tests, just enough to make them pass.

Kafka 2.0 requires a higher version for slf4j, I had to update it to 1.7.25.

Option migrateZookeeperOffsets is deprecated in this PR.
This will allow us to get rid of Kafka server libraries in Flume.

Compatibility testing.
Modified the TestUtil to be able to use external servers. This way I could test
against a variety of Kafka Server versions using the normal unit tests.
Channel tests using 2.0.1 client:
Kafka_2.11_0.9.0.0 Not compatible
Kafka_2.11_0.10.0.0 Not compatible
Kafka_2.11_0.10.1.0 passed with TestPartition timeouts
(rerunning the single test passes so it is a tes isolation issue)
Kafka_2.11_0.10.2.0 passed with TestPartition timeouts
(rerunning the single test passes so it is a tes isolation issue)
Kafka_2.11-0.11.0.3 - timeouts in TestPartitions when creating topics
Kafka_2.11-1.0.2 - passed
Kafka_2.11-1.1.1 - passed
Kafka_2.11-2.0.1 - passed

This closes #235

Reviewers: Tristan Stevens, Ferenc Szabo, Peter Turcsanyi

(Endre Major via Ferenc Szabo)

Hadoop 1/2 profiles were obsolete and had not been used for a long time,
so they have been deleted.
HBase profile was always active, so its content has been moved to
top level.
Additional clean-ups: some version declarations moved to the parent pom,
redundant version declarations and exclusions deleted.

This closes #239

Reviewers: Ferenc Szabo

(Peter Turcsanyi via Ferenc Szabo)

Adding missing counter to KafkaSink

Reviewers: Denes Arvay, Attila Simon, Ferenc Szabo

(Udai Kiran Potluri via Ferenc Szabo)

It seems when solving https://issues.apache.org/jira/browse/FLUME-2799 ,
an oversight resulted in the message offset not being added to the header.

This change corrects this.

This closes #238

Reviewers: Ferenc Szabo, Peter Turcsanyi

(Jehan Bruggeman via Ferenc Szabo)

If there are multiple files in the path(s) that need to be tailed and there
is a file written by high frequency, then Taildir can read the batchSize size
events from that file every time. This can lead to an endless loop and Taildir
will only read data from the busy file, while other files will not be
processed.
Another problem is that in this case TaildirSource will be unresponsive to
stop requests too.

This commit handles this situation by introducing a new config property called
maxBatchCount. It controls the number of batches being read consecutively
from the same file. After reading maxBatchCount rounds from a file, Taildir
will switch to another file / will have a break in the processing.

This change is based on hunshenshi's patch.

This closes #240

Reviewers: Ferenc Szabo, Endre Major

(Peter Turcsanyi via Ferenc Szabo)

TaildirSource.process() implements the correct polling logic now. It returns
Status.READY / Status.BACKOFF which controls the common backoff sleeping
mechanism implemented in PollableSourceRunner.PollingRunner (instead of
always returning Status.READY and sleeping inside the method which was
an incorrect behaviour).

This closes #241

Reviewers: Endre Major, Denes Arvay

(Peter Turcsanyi via Ferenc Szabo)

The default hdfs.callTimeout used by the HDFS sink was too low only 10 seconds
that can cause problems on a busy system.
The new default is 30 sec.
I think this parameter should be deprecated and some new more error tolerant
solution should be used. To enable the future change I indicated this in the
code and in the Users Guide.
Tested only with the unit tests.

This closes #243

Reviewers: Ferenc Szabo, Peret Turcsanyi

(Endre Major via Ferenc Szabo)

An update to the configuration section of the user guide.

This closes #246

Reviewers: Peret Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)