Permalink
Commits on Nov 30, 2016
  1. Fix Remove Header Interceptor properties table in FlumeUserGuide.rst

    This closes #91
    
    Reviewers: Denes Arvay, Bessenyei Balázs Donát
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Nov 30, 2016
Commits on Nov 23, 2016
  1. Clean up references to the Flume 0.9 TailSource and recommendations t…

    …o use Exec source to tail files
    
    Flume has Spooling Directory Source and Taildir Source now, so there's no need to use workarounds such as the Exec source to tail files.
    
    This closes #88
    
    Reviewers: Denes Arvay, Jeff Holoman
    
    (Peter Ableda via Bessenyei Balázs Donát)
    Peter Ableda committed with bessbd Nov 18, 2016
  2. Fix KafkaSource and KafkaSink example configurations in the user guide

    This closes #86
    
    Reviewers: Jeff Holoman, Denes Arvay
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Nov 23, 2016
Commits on Nov 21, 2016
  1. FLUME-3025. Expose FileChannel.open on JMX

    This patch exposes the FileChannel's open flag on JMX to make it possible to detect when it wasn't able to start up. In those cases the open flag is false.
    
    This closes #85
    
    Reviewers: Attila Simon, Bessenyei Balázs Donát
    
    (Denes Arvay via Bessenyei Balázs Donát)
    adenes committed with bessbd Nov 21, 2016
Commits on Nov 20, 2016
  1. FLUME-2171. Add Interceptor to remove headers from event

    Similar to Flume OG decorators, this introduces an Interceptor to remove headers from an event.
    
    Reviewers: Bessenyei Balázs Donát, Jeff Holoman, Denes Arvay, Attila Simon
    
    (Gabriel Commeau and Bessenyei Balázs Donát via Jeff Holoman)
    gcommeau committed with jholoman Nov 20, 2016
Commits on Nov 19, 2016
  1. Remove hostHeader = hostname property from Host interceptor example

    We are overriding the host header name from `host` to `hostname` in the example usage section. Due to this example users are overriding the header name too but still use the `%{host}` substitution as shown in the HDFS Sink section. This leads to errors.
    
    This change removes the mentioned example config line, thus the default header will be used.
    
    This closes #87
    
    Reviewers: Bessenyei Balázs Donát
    
    (Peter Ableda via Bessenyei Balázs Donát)
    Peter Ableda committed with bessbd Nov 18, 2016
Commits on Nov 9, 2016
  1. FLUME-3020. Improve HDFS Sink escape sequence substitution

    When using escape sequences, the current code will call InetAddress
    for every event which results in a huge impact to performance.
    
    This patch fixes that issue by caching the local host in a static variable.
    We can do this because there is zero chance the local host will change for a life of a JVM.
    
    Reviewers: Hari Shreedharan, Jeff Holoman, Bessenyei Balázs Donát
    
    (Theodore michael Malaska via Bessenyei Balázs Donát)
    TedBear42 committed with bessbd Nov 9, 2016
Commits on Nov 5, 2016
  1. FLUME-2812. Fix semaphore leak causing java.lang.Error: Maximum permi…

    …t count exceeded in MemoryChannel
    
    bytesRemaining.release(putByteCounter) gets called in MemoryTransaction.doRollback while acquire is called only in doCommit. This results in semaphore leak and the number of permits in the semaphore eventually exceeds Integer.MAX_VALUE and Semaphore.release() throws java.lang.Error: Maximum permit count exceeded.
    
    This closes #83
    
    Reviewers: Attila Simon, Bessenyei Balázs Donát
    
    (Denes Arvay via Bessenyei Balázs Donát)
    adenes committed with bessbd Oct 28, 2016
Commits on Oct 28, 2016
  1. Add e-mail templates to HowToRelease.md

    Reviewers: Attila Simon
    
    This closes #78
    bessbd committed Oct 25, 2016
  2. FLUME-2857. Make Kafka Source/Channel/Sink restore default values whe…

    …n live updating config
    
    This commit changes Kafka Channel, Sink and Source to fix an error where
    sub-configurations aren't tolerant of the configure() method being called more
    than once (as happens when a Live Config Update happens).
    
    Reviewers: Denes Arvay, Attila Simon, Bessenyei Balázs Donát
    
    (Tristan Stevens via Bessenyei Balázs Donát)
    tmgstevens committed with bessbd Oct 28, 2016
  3. Add Developer Section / Developers Quick Hack Sheet from cwiki.apache…

    ….org
    
    Reviewers: Attila Simon
    
    This closes #81
    bessbd committed Oct 25, 2016
  4. Add Developer Section / How to Commit from cwiki.apache.org

    Reviewers: Attila Simon
    
    This closes #80
    bessbd committed Oct 28, 2016
Commits on Oct 27, 2016
  1. Add Developer Section / How to Contribute from cwiki.apache.org

    Reviewers: Attila Simon
    
    This closes #79
    bessbd committed Oct 25, 2016
  2. FLUME-2997. Fix flaky test in SpillableMemoryChannel

    When the mock sinks are slower than sources, testParallelSingleSourceAndSink sometimes fails.
    In such situations the channel can get full, thus uncaught ChannelFullException breaks the test.
    
    Since testCapacityWithOverflow was designed to cover such a scenario, we
    can safely fix the test by increasing the channel capacity to make sure it won't get full.
    
    This commit does the previously mentioned change.
    
    Reviewers: Bessenyei Balázs Donát
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Oct 27, 2016
Commits on Oct 25, 2016
  1. Add Developer Section / How to Release from cwiki.apache.org

    Reviewers: Mike Percy
    
    This closes #77
    bessbd committed Oct 24, 2016
Commits on Oct 24, 2016
  1. FLUME-3002. Fix tests in TestBucketWriter

    Some tests are flaky in TestBucketWriter.
    
    This commit fixes the flakiness mentioned before by
    adding a new constructor with an extra Clock parameter.
    
    Reviewers: Attila Simon, Denes Arvay
    bessbd committed Oct 24, 2016
  2. Set MAVEN_OPTS for Travis-CI

    Travis builds occasionally fail with Java heap space error.
    
    This commit sets higher limits to fix the issue mentioned above.
    
    Reviewers: Lior Zeno, Mike Percy
    
    This closes #75
    bessbd committed Oct 24, 2016
Commits on Oct 22, 2016
  1. Fix .travis.yml

    License was missing from .travis.yml, so RAT checks failed.
    
    This commit adds the license header to .travis.yml.
    After this change RAT passes again.
    
    This closes #74
    bessbd committed Oct 22, 2016
  2. Add .travis.yml

    ASF infra has enabled Travis for Apache Flume.
    
    This change adds .travis.yml, because the option "Build only if .travis.yml is present" is turned on.
    
    With this commit, builds will run with -DskipTests.
    bessbd committed Oct 22, 2016
Commits on Oct 18, 2016
  1. Fix startup order for ExecSource

    This patch starts the SourceCounter in the ExecSource before starting
    the runner thread that processes input from the forked command.
    
    Starting the SourceCounter before calling its methods is required to get
    correct metric counts because the metrics are all reset to 0 when
    SourceCounter.start() is called.
    
    A follow-up patch will attempt to enforce that a SourceCounter or any
    other MonitoredCounterGroup is running at the time that calls to modify
    the metrics occur.
    
    After applying this patch, TestExecSource.testMonitoredCounterGroup no
    longer fails.
    
    This closes #72
    
    Reviewers: Attila Simon, Bessenyei Balázs Donát
    
    (Mike Percy via Bessenyei Balázs Donát)
    mpercy committed with bessbd Oct 13, 2016
Commits on Oct 12, 2016
  1. Remove test dependencies from binary release artifact

    This patch removes some test-specific dependencies from the binary
    release artifact. These were introduced by the new
    flume-shared-kafka-test module that is intended for sharing test code.
    Please see the new comment in bin.xml for more information.
    
    Reviewers: Bessenyei Balázs Donát
    
    Closes #70
    mpercy committed Oct 12, 2016
  2. Update LICENSE

    The LICENSE file was outdated and missing some license texts
    
    This closes #68
    
    Reviewers: Mike Percy, Bessenyei Balázs Donát
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Oct 12, 2016
Commits on Oct 11, 2016
  1. Add flume-shared module to assembly xmls

    Fixes the issue that the assembly plugin created the source package without
    the flume-shared project in it so maven compile failed on that.
    
    This closes #67
    
    Reviewers: Mike Percy
    
    (Denes Arvay via Bessenyei Balázs Donát)
    adenes committed with bessbd Oct 11, 2016
Commits on Oct 10, 2016
  1. Update CHANGELOG for 1.7

    bessbd committed Oct 10, 2016
  2. Update RELEASE-NOTES for 1.7.0

    bessbd committed Oct 10, 2016
  3. FLUME-2971. Add secure Kafka Sink/Source/Channel setup to the User Guide

    The User Guide already has details about configuring Kafka channel to work with a kerberized Kafka cluster.
    
    This patch adds similar description for Kafka Sink and Kafka Source.
    
    Reviewers: Tristan Stevens, Mike Percy, Bessenyei Balázs Donát
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Oct 10, 2016
  4. FLUME-2999. Kafka channel and sink should enable statically assigned …

    …partition per event via header
    
    This feature is useful for anyone who needs greater control of which
    partitions are being written to - normally in a situation where multiple
    Flume agents are being deployed in order to horizontally scale, or
    alternatively if there is a scenario where there is a skew in data that
    might lead to one or more partitions hotspotting.
    
    We also have the ability to specify custom partitions on to the Kafka
    Producer itself using the kafka.* configuration properties. The Kafka
    Producer provides the ability to set the partition ID using a particular
    ProducerRecord constructor, this is just a matter of providing the
    option to use this constructor.
    
    Reviewers: Attila Simon, Mike Percy
    
    (Tristan Stevens via Mike Percy)
    tmgstevens committed with mpercy Oct 10, 2016
  5. FLUME-2911. Add include pattern option in SpoolDir source

    * Documented what happens when ignorePattern and includePattern both
      match for a given file.
    * Added two tests to simulate what happens when both ignorePattern and
      includePattern options are specified
    * Refactored of ReliableSpoolingFileEventReader test and fix of code
      style violations
    
    Closes #60
    
    Reviewers: Bessenyei Balázs Donát, Denes Arvay, Attila Simon
    
    (Andrea Rota via Mike Percy)
    andrearota committed with mpercy Aug 4, 2016
Commits on Oct 7, 2016
  1. FLUME-3003. Fix flaky testSourceCounter in TestSyslogUdpSource

    The event processing in SyslogUDPSource is handled on a separate thread by Netty so message delivery, thus the sourceCounter's increment can be delayed resulting in a flaky test.
    
    This patch fixes that issue by adding more delay when necessary.
    
    Reviewers: Lior Zeno, Bessenyei Balázs Donát
    
    (Denes Arvay via Bessenyei Balázs Donát)
    adenes committed with bessbd Oct 7, 2016
Commits on Sep 30, 2016
  1. FLUME-2998. Add a missing configuration parameter to SequenceSource docs

    SequenceSource's totalEvents is configurable, but it is not in the documentation.
    
    This patch fixes that problem by adding the missing configuration parameter to the docs.
    
    Reviewers: Denes Arvay, Bessenyei Balázs Donát
    
    (Attila Simon via Bessenyei Balázs Donát)
    simonati committed with bessbd Sep 30, 2016
Commits on Sep 26, 2016
  1. FLUME-2966. Fix NPE in JMS Source

    JMS Source does not check for null text in a TextMessage. This can lead to NullPointerException.
    This commit fixes that problem by checking for nullity of textMessage.getText().
    
    Reviewers: Denes Arvay, Attila Simon, Mike Percy, Bessenyei Balázs Donát
    
    (Siddharth Ahuja via Bessenyei Balázs Donát)
    c-sahuja committed with bessbd Sep 26, 2016
Commits on Sep 8, 2016
  1. FLUME-2797. Use SourceCounter for SyslogTcpSource

    This patch uses the newer SourceCounter class for the SyslogTcpSource.
    It also marks the SyslogTcpSource as deprecated and improves how the
    unit tests are written so they don't require DNS.
    
    Reviewers: Attila Simon, Lior Zeno, Balázs Donát Bessenyei, Mike Percy
    
    (Denes Arvay via Mike Percy)
    adenes committed with mpercy Sep 8, 2016
Commits on Sep 1, 2016
  1. FLUME-2982. Add localhost escape sequence to HDFS sink

    It would be useful to be able to just use %[localhost] escape sequence
    instead of having to pass in a header or use the host interceptor.
    
    Part of the problem of using an interceptor is the case where Flume only
    consists of a channel and a sink (think KafkaChannel). There is support
    in Flume for a sink-side interceptor at the time of this writing.
    
    Reviewers: Grant Henke, Mike Percy
    
    (Jeff Holoman via Mike Percy)
    jholoman committed with mpercy Sep 1, 2016
  2. FLUME-2983. Handle offset migration in the new Kafka Source

    Similar to FLUME-2972, offsets tracking the position in Kafka consumers
    change from using zookeeper for offset storage to Kafka when moving from
    0.8.x to 0.9.x.
    
    FLUME-2821 makes the client change in the Kafka Source but does not
    ensure existing offsets get migrated in order to continue consuming
    where it left off. Flume should have some automated logic on startup to
    check if Kafka offsets exist, if not and migration is enabled (by
    default) then copy the offsets from Zookeeper and commit them to Kafka.
    
    This change should also fix the backwards incompatibility caused by
    removing the zookeeperConnect property. The bootstrap can be looked up
    if zookeeperConnect is used.
    
    Reviewers: Denes Arvay, Mike Percy
    
    (Grant Henke via Mike Percy)
    granthenke committed with mpercy Sep 1, 2016