Permalink
Commits on Nov 13, 2016
  1. KAFKA-3829: Ensure valid configuration prior to creating connector

    Author: Jason Gustafson <jason@confluent.io>
    
    Reviewers: Shikhar Bhushan <shikhar@schmizz.net>, Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #1911 from hachikuji/KAFKA-3829
    hachikuji committed with ewencp Nov 13, 2016
Commits on Nov 11, 2016
  1. MINOR: Fix export command for additional env vars in connect system t…

    …ests
    
    Author: Konstantine Karantasis <konstantine@confluent.io>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #2126 from kkonstantine/MINOR-Fix-formatting-of-env-vars-in-connect-system-test-template
    kkonstantine committed with ewencp Nov 11, 2016
  2. KAFKA-4081; KafkaConsumer should not allow negative offsets to be com…

    …mitted
    
    Author: Mickael Maison <mickael.maison@gmail.com>
    
    Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
    
    Closes #1827 from mimaison/KAFKA-4081
    mimaison committed with hachikuji Nov 11, 2016
  3. KAFKA-4379: Remove caching of dirty and removed keys from StoreChange…

    …Logger
    
    The `StoreChangeLogger` currently keeps a cache of dirty and removed keys and will batch the changelog records such that we don't send a record for each update. However, with KIP-63 this is unnecessary as the batching and de-duping is done by the caching layer. Further, the `StoreChangeLogger` relies on `context.timestamp()` which is likely to be incorrect when caching is enabled
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Matthias J. Sax, Eno Thereska, Guozhang Wang
    
    Closes #2103 from dguy/store-change-logger
    dguy committed with guozhangwang Nov 11, 2016
  4. MINOR: Extend mirror maker test to include interceptors

    Author: Konstantine Karantasis <konstantine@confluent.io>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #2081 from kkonstantine/MINOR-Extend-mirror-maker-test-to-include-interceptors
    kkonstantine committed with ewencp Nov 11, 2016
  5. HOTFIX: failing to close this iterator causes leaks in rocksdb

    guozhangwang dguy
    
    Author: dan norwood <norwood@confluent.io>
    
    Reviewers: Damian Guy, Michael G. Noll, Guozhang Wang
    
    Closes #2122 from norwood/close-call
    norwood committed with guozhangwang Nov 11, 2016
  6. MINOR: improve exception message for incompatible Serdes to actual ke…

    …y/value data types
    
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Michael G. Noll, Guozhang Wang
    
    Closes #2118 from mjsax/hotfixImproveSerdeTypeMissmatchError
    mjsax committed with guozhangwang Nov 11, 2016
Commits on Nov 10, 2016
  1. revert streams/src/main/java/org/apache/kafka/streams/processor/Consu…

    …merRecordTimestampExtractor.java
    guozhangwang committed Nov 10, 2016
Commits on Nov 9, 2016
  1. MINOR: add upgrade guide for Kafka Streams API

    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Michael G. Noll, Eno Thereska
    
    Closes #2114 from mjsax/updateDocUpgradeSection
    mjsax committed with guozhangwang Nov 9, 2016
  2. KAFKA-4364: Remove secrets from DEBUG logging

    leverage fix from KAFKA-2690 to remove secrets from task logging
    
    Author: rnpridgeon <ryan.n.pridgeon@gmail.com>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #2115 from rnpridgeon/KAFKA-4364
    rnpridgeon committed with ewencp Nov 9, 2016
  3. MINOR: fix incorrect logging in StreamThread

    Fix incorrect logging when unable to create an active task. The output was: Failed to create an active task %s:
    It should have the taskId.
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Ismael Juma, Eno Thereska
    
    Closes #2109 from dguy/minor-logging
    dguy committed with guozhangwang Nov 9, 2016
  4. MINOR: remove unused fields from KTableImpl

    Remove `keySerde`, `valSerde`, `OUTERTHIS_NAME`, `OUTEROTHER_NAME`, `LEFTTHIS_NAME`, `LEFTOTHER_NAME` from `KTableImpl` as they are all unused fields
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Guozhang Wang <wangguoz@gmail.com>
    
    Closes #2119 from dguy/minor-ktable-unused
    dguy committed with guozhangwang Nov 9, 2016
  5. KAFKA-4311: Multi layer cache eviction causes forwarding to incorrect…

    … ProcessorNode
    
    Given a topology like the one below. If a record arriving in `tableOne` causes a cache eviction, it will trigger the `leftJoin` that will do a `get` from `reducer-store`. If the key is not currently cached in `reducer-store`, but is in the backing store, it will be put into the cache, and it may also trigger an eviction. If it does trigger an eviction and the eldest entry is dirty it will flush the dirty keys. It is at this point that a ClassCastException is thrown. This occurs because the ProcessorContext is still set to the context of the `leftJoin` and the next child in the topology is `mapValues`.
    We need to set the correct `ProcessorNode`, on the context, in the `ForwardingCacheFlushListener` prior to calling `context.forward`. We also need to  remember to reset the `ProcessorNode` to the previous node once `context.forward` has completed.
    
    ```
           final KTable<String, String> one = builder.table(Serdes.String(), Serdes.String(), tableOne, tableOne);
            final KTable<Long, String> two = builder.table(Serdes.Long(), Serdes.String(), tableTwo, tableTwo);
            final KTable<String, Long> reduce = two.groupBy(new KeyValueMapper<Long, String, KeyValue<String, Long>>() {
                Override
                public KeyValue<String, Long> apply(final Long key, final String value) {
                    return new KeyValue<>(value, key);
                }
            }, Serdes.String(), Serdes.Long())
                    .reduce(new Reducer<Long>() {..}, new Reducer<Long>() {..}, "reducer-store");
    
        one.leftJoin(reduce, new ValueJoiner<String, Long, String>() {..})
            .mapValues(new ValueMapper<String, String>() {..});
    
    ```
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Eno Thereska, Guozhang Wang
    
    Closes #2051 from dguy/kafka-4311
    dguy committed with guozhangwang Nov 9, 2016
  6. MINOR: fix typos and incorrect docs

    Author: Xavier Léauté <xavier@confluent.io>
    
    Reviewers: Matthias J. Sax, Guozhang Wang
    
    Closes #2112 from xvrl/minor-doc-fixes
    xvrl committed with guozhangwang Nov 9, 2016
  7. KAFKA-4360:Controller may deadLock when autoLeaderRebalance encounter…

    … zk expired
    
    Author: tuyang <tuyang@meituan.com>
    Author: xiguantiaozhan <kafkausr@126.com>
    
    Reviewers: Ismael Juma, Jiangjie Qin, Guozhang Wang
    
    Closes #2094 from xiguantiaozhan/rebalance_deadlock
    Tu Yang committed with guozhangwang Nov 9, 2016
Commits on Nov 8, 2016
  1. MINOR: some trace logging for streams debugging

    Author: Ubuntu <norwood@confluent.io>
    
    Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #1882 from norwood/streams-logging
    norwood committed with ewencp Nov 8, 2016
  2. MINOR: remove commented out code and System.out.println

    Remove commented out code and System.out.println from KTableKTableJoinIntegrationTest
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Matthias J. Sax, Eno Thereska, Guozhang Wang
    
    Closes #2092 from dguy/cleanup-comments
    dguy committed with guozhangwang Nov 8, 2016
  3. KAFKA-4284: Make Partitioner a Closeable and close it when closing th…

    …e producer
    
    [KAFKA-4284](https://issues.apache.org/jira/browse/KAFKA-4284)
    
    Even though Partitioner has a close method it is not closed when the producer is closed. Serializers, interceptors and metrics are all closed, so partitioners should be closed to.
    
    To be able to use the same mechanism to close the partitioner as the serializers, etc. I had to make the `Partitioner` interface extend `Closeable`. Since this doesn't change the interface that feels ok and should be backwards compatible.
    
    Looking at [KAFKA-2091](https://issues.apache.org/jira/browse/KAFKA-2091) (d6c45c7) that introduced the `Partitioner` interface it looks like the intention was that the producer should close the partitioner.
    
    This contribution is my original work and I license the work to the project under the project's open source license.
    
    Author: Theo <theo@iconara.net>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #2000 from iconara/kafka-4284
    iconara committed with ewencp Nov 8, 2016
Commits on Nov 7, 2016
  1. MINOR: Fix regex on connector path param in ConnectorsResource

    Author: Ewen Cheslack-Postava <me@ewencp.org>
    
    Reviewers: Shikhar Bhushan <shikhar@schmizz.net>, Konstantine Karantasis <konstantine@confluent.io>, Jason Gustafson <jason@confluent.io>
    
    Closes #2108 from ewencp/fix-rest-api-regex
    ewencp committed with hachikuji Nov 7, 2016
Commits on Nov 4, 2016
  1. MINOR: Fix re-raise of python error in system tests

    Author: Ewen Cheslack-Postava <me@ewencp.org>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #2099 from ewencp/fix-python-reraise
    ewencp committed Nov 4, 2016
  2. MINOR: missing fullstop in doc for `max.partition.fetch.bytes`

    Author: Shikhar Bhushan <shikhar@schmizz.net>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #1969 from shikhar/patch-2
    shikhar committed with ewencp Nov 4, 2016
  3. KAFKA-4372: Kafka Connect REST API does not handle DELETE of connecto…

    …r with slashes in their names
    
    Kafka Connect REST API does not handle in many places connectors with slashes in their names because it expects PathParams, this PR intends to :
    * Reject as bad requests API calls trying to create connectors with slashes in their names
    * Add support for connector with slashes in their names in the DELETE part of the API to allow users to cleanup their connectors without dropping everything.
    
    This PR adds as well the Unit Test needed for the creation part and was tested manually for the DELETE part.
    
    Author: Olivier Girardot <o.girardot@lateral-thoughts.com>
    
    Reviewers: Shikhar Bhushan <shikhar@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #2096 from ogirardot/fix/connectors-with-slashes-cannot-be-deleted
    ogirardot committed with ewencp Nov 4, 2016
  4. MINOR: Add description of how consumer wakeup acts if no threads are …

    …awakened
    
    I think the Javadoc should describe what happens if wakeup is called and no other thread is currently blocking. This may be important in some cases, e.g. trying to shut down a poll thread, followed by manually committing offsets.
    
    Author: Stig Rohde Døssing <sdo@it-minds.dk>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #2093 from srdo/minor-expand-wakeup-javadoc
    Stig Rohde Døssing committed with hachikuji Nov 4, 2016
  5. KAFKA-4024; Override client metadata backoff on topic changes and avo…

    …id unnecessary connections
    
    Fixes a bug that inappropriately applies backoff as interval between metadata updates even though the current one is outdated.
    
    Author: Yuto Kawamura <kawamuray.dadada@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
    
    Closes #1707 from kawamuray/KAFKA-4024-metadata-backoff
    kawamuray committed with hachikuji Nov 4, 2016
Commits on Nov 3, 2016
  1. MINOR: Remove stray `%s` in `TopicMetadata.toString`

    This bug was recently introduced via 2959bc2.
    
    Author: Himani Arora <1himani.arora@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #2091 from himani1/minor_fix
    himani1 committed with ijuma Nov 3, 2016
Commits on Nov 2, 2016
  1. MINOR: Fix NPE when Connect offset contains non-primitive type

    When storing a non-primitive type in a Connect offset, the following NullPointerException will occur:
    
    ```
    07:18:23.702 [pool-3-thread-1] ERROR o.a.k.c.storage.OffsetStorageWriter - CRITICAL: Failed to serialize offset data, making it impossible to commit offsets under namespace tenant-db-bootstrap-source. This likely won't recover unless the unserializable partition or offset information is overwritten.
    07:18:23.702 [pool-3-thread-1] ERROR o.a.k.c.storage.OffsetStorageWriter - Cause of serialization failure:
    java.lang.NullPointerException: null
    	at org.apache.kafka.connect.storage.OffsetUtils.validateFormat(OffsetUtils.java:51)
    	at org.apache.kafka.connect.storage.OffsetStorageWriter.doFlush(OffsetStorageWriter.java:143)
    	at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:319)
    ... snip ...
    ```
    
    The attached patch fixes the specific case where OffsetUtils.validateFormat is attempting to provide a useful error message, but fails to because the schemaType method could return null.
    
    This contribution is my original work and I license the work to the project under the project's open source license.
    
    Author: Mathieu Fenniak <mathieu.fenniak@replicon.com>
    
    Reviewers: Gwen Shapira
    
    Closes #2087 from mfenniak/fix-npr-with-clearer-error-message
    mfenniak committed with gwenshap Nov 2, 2016
  2. MINOR: Replaced unnecessary isDefined and get on option values with fold

    Author: himani1 <1himani.arora@gmail.com>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #2050 from himani1/refactored_code
    himani1 committed with hachikuji Nov 2, 2016
  3. MINOR: Fix documentation of compaction

    Removed a duplicate line and also cleaned up some of the language around compaction guarantees.
    
    Author: Apurva Mehta <apurva.1618@gmail.com>
    
    Reviewers: Gwen Shapira
    
    Closes #2089 from apurvam/fix-documentation-of-compaction and squashes the following commits:
    
    03c5bdd [Apurva Mehta] Fix line length to be consistent with the rest of the file
    0af1a86 [Apurva Mehta] MINOR: fix duplicate line in docs for compaction.
    apurvam committed with gwenshap Nov 2, 2016
Commits on Nov 1, 2016
  1. KAFKA-4357; Fix consumer group describe output when there is no activ…

    …e member (old consumer)
    
    Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
    
    Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Jason Gustafson <jason@confluent.io>
    
    Closes #2075 from vahidhashemian/KAFKA-4357
    vahidhashemian committed with hachikuji Nov 1, 2016
  2. KAFKA-4352: instable ResetTool integration test

     - increased timeout to stabilize test
    
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Eno Thereska, Guozhang Wang
    
    Closes #2082 from mjsax/kafka-4352-hotfix
    mjsax committed with guozhangwang Nov 1, 2016
  3. KAFKA-4361: Streams does not respect user configs for "default" params

    Enable user provided consumer and producer configs to override the streams default configs.
    
    Author: Damian Guy <damian.guy@gmail.com>
    
    Reviewers: Eno Thereska, Matthias J. Sax, Guozhang Wang
    
    Closes #2084 from dguy/kafka-4361
    dguy committed with guozhangwang Nov 1, 2016
  4. MINOR: Fix issue in `AsyncProducerTest` where it expects the `port` c…

    …onfig to be set
    
    This test fails locally when I run it, but somehow Jenkins builds are passed. Not clear how.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #2073 from ijuma/async-producer-test-port-config
    ijuma committed with hachikuji Nov 1, 2016
Commits on Oct 30, 2016
  1. HOTFIX: improve error message on invalid input record timestamp

    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Guozhang Wang, Ismael Juma, Michael G. Noll, Eno Thereska
    
    Closes #2076 from mjsax/hotfixTSExtractor
    mjsax committed with guozhangwang Oct 30, 2016
  2. KAFKA-4302: Simplify KTableSource

    KTableSource is always materialized since IQ:
      - removed flag KTableSource#materialized
      - removed MaterializedKTableSourceProcessor
    
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Eno Thereska, Guozhang Wang
    
    Closes #2065 from mjsax/kafka-4302-simplify-ktablesource
    mjsax committed with guozhangwang Oct 30, 2016
  3. KAFKA-3559: Recycle old tasks when possible

    Author: Eno Thereska <eno.thereska@gmail.com>
    
    Reviewers: Damian Guy, Guozhang Wang
    
    Closes #2032 from enothereska/KAFKA-3559-onPartitionAssigned
    enothereska committed with guozhangwang Oct 30, 2016