Permalink
Commits on Nov 7, 2017
  1. MINOR: Eliminate unnecessary Topic(And)Partition allocations in Contr…

    ijuma committed Nov 7, 2017
    …oller
    
    - Eliminated all the unnecessary allocations of `TopicPartition` and
    `TopicAndPartition` in the Controller. We now use the former
    in the Controller (bringing it inline with the rest of the non legacy
    code).
    - Fixed missed `Listener` -> `Handler` renames for companion
    objects.
    - More String.format -> String interpolation conversions (the former
    is roughly 5 times more expensive).
    - Some other minor clean-ups.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Onur Karaman <okaraman@linkedin.com>, Viktor Somogyi <viktorsomogyi@gmail.com>
    
    Closes #4152 from ijuma/controller-topic-partition-and-other-clean-ups
  2. KAFKA-5894; add the notion of max inflight requests to async ZooKeepe…

    onurkaraman committed Nov 7, 2017
    …rClient
    
    ZooKeeperClient is a zookeeper client that encourages pipelined requests to zookeeper. We want to add the notion of max inflight requests to the client for several reasons:
    1. to bound memory overhead associated with async requests on the client.
    2. to not overwhelm the zookeeper ensemble with a burst of requests.
    
    Author: Onur Karaman <okaraman@linkedin.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>, Ted Yu <yuzhihong@gmail.com>, Jun Rao <junrao@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
    
    Closes #3860 from onurkaraman/KAFKA-5894
Commits on Nov 6, 2017
  1. MINOR: PartitionReassignmentHandler should only generate event when z…

    ijuma committed Nov 6, 2017
    …node is created
    
    We only need to generate the event when the znode is created or deleted.
    In the former case, we start the reassignment while in the latter we
    re-register the watcher (necessary for the Controller to detect future
    reassignments).
    
    During Controller failover, we restart the reassignment without generating
    an event so it's not affected by this change.
    
    Also use the Controller cache (`ControllerContext.partitionsBeingReassigned`)
    in `removePartitionFromReassignedPartitions` instead of reloading the
    data from ZooKeeper.
    
    Overall, we would previously load the reassignment data from ZooKeeper twice
    per completed partition whereas now as don't do it at all. As an example,
    say there were 30k partitions being reassigned, these changes save the
    allocation of 900 million `TopicAndPartition` and `Seq[Int]` (replicas)
    instances (could easily amount to 20-40 GB depending on the topic name
    length). This matters most in cases where the partitions being reassigned
    don't have much data allowing the reassignment to complete reasonably
    fast for many of the partitions.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Jun Rao <junrao@gmail.com>, Onur Karaman <okaraman@linkedin.com>
    
    Closes #4143 from ijuma/partition-reassignment-ignore-handle-deletion-and-data-change
  2. KAFKA-6120: RecordCollector should not retry sending

    mjsax committed with guozhangwang Nov 6, 2017
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4148 from mjsax/kafka-6120-recordCollector
  3. KAFKA-6115: TaskManager should be type aware

    mjsax committed with guozhangwang Nov 6, 2017
     - remove type specific methods from Task interface
     - add generics to preserve task type
     - add sub classes for different task types
    
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4129 from mjsax/kafka-6115-taskManager-should-be-type-aware
  4. KAFKA-6172; Cache lastEntry in TimeIndex to avoid unnecessary disk ac…

    lindong28 committed with becketqin Nov 6, 2017
    …cess
    
    Author: Dong Lin <lindong28@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
    
    Closes #4177 from lindong28/KAFKA-6172
  5. KAFKA-6156; Metric tag values with colons must be sanitized

    huxihx committed with rajinisivaram Nov 6, 2017
    Windows directory paths often contain colons which are not allowed in
    yammer metrics. Metric tag values with special characters must be
    quoted.
    
    Author: huxihx <huxi_2b@hotmail.com>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Closes #4173 from huxihx/KAFKA-6156
  6. KAFKA-6157; Fix repeated words words in JavaDoc and comments.

    efeg committed with becketqin Nov 6, 2017
    Author: Adem Efe Gencer <agencer@linkedin.com>
    
    Reviewers: Jiangjie Qin <becket.qin@gmail.com>
    
    Closes #4170 from efeg/bug/typoFix
Commits on Nov 3, 2017
  1. MINOR: Update docs for new version

    guozhangwang committed Nov 3, 2017
    1. Update the Streams hello world examples with the new API.
    2. Update the version references in various places.
    3. Update version templates to 1.1.x.
    
    Author: Guozhang Wang <wangguoz@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Derrick Or <derrickor@gmail.com>
    
    Closes #4169 from guozhangwang/KMINOR-streams-docs
  2. KAFKA-6060; Add workload generation capabilities to Trogdor

    cmccabe committed with rajinisivaram Nov 3, 2017
    Previously, Trogdor only handled "Faults."  Now, Trogdor can handle
    "Tasks" which may be either faults, or workloads to execute in the
    background.
    
    The Agent and Coordinator have been refactored from a
    mutexes-and-condition-variables paradigm into a message passing
    paradigm.  No locks are necessary, because only one thread can access
    the task state or worker state.  This makes them a lot easier to reason
    about.
    
    The MockTime class can now handle mocking deferred message passing
    (adding a message to an ExecutorService with a delay).  I added a
    MockTimeTest.
    
    MiniTrogdorCluster now starts up Agent and Coordinator classes in
    paralle in order to minimize junit test time.
    
    RPC messages now inherit from a common Message.java class.  This class
    handles implementing serialization, equals, hashCode, etc.
    
    Remove FaultSet, since it is no longer necessary.
    
    Previously, if CoordinatorClient or AgentClient hit a networking
    problem, they would throw an exception.  They now retry several times
    before giving up.  Additionally, the REST RPCs to the Coordinator and
    Agent have been changed to be idempotent.  If a response is lost, and
    the request is resent, no harm will be done.
    
    Author: Colin P. Mccabe <cmccabe@confluent.io>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
    
    Closes #4073 from cmccabe/KAFKA-6060
  3. MINOR: update producer client request timeout in system test

    bbejeck committed with guozhangwang Nov 3, 2017
    Author: Bill Bejeck <bill@confluent.io>
    
    Reviewers: Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4168 from bbejeck/MINOR_update_streams_produer_timeout_in_system_test
  4. MINOR: Change version format in release notes python code

    guozhangwang committed Nov 3, 2017
    ijuma ewencp
    
    Author: Guozhang Wang <wangguoz@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #4171 from guozhangwang/KMinor-update-releasepy
Commits on Nov 2, 2017
  1. MINOR: Remove clients/out directory

    ijuma committed with rajinisivaram Nov 2, 2017
    It was committed inadvertently.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Closes #4172 from ijuma/remove-out-folder
Commits on Nov 1, 2017
  1. KAFKA-6072; User ZookeeperClient in GroupCoordinator and TransactionC…

    omkreddy committed with junrao Nov 1, 2017
    …oordinator
    
    Author: Manikumar Reddy <manikumar.reddy@gmail.com>
    
    Reviewers: Ted Yu <yuzhihong@gmail.com>, Jun Rao <junrao@gmail.com>
    
    Closes #4126 from omkreddy/KAFKA-6072-ZK-IN-GRoupCoordinator
Commits on Oct 31, 2017
  1. MINOR: Adding Trivago logo to Streams landing page

    manjuapu committed with guozhangwang Oct 31, 2017
    guozhangwang Please review
    
    Author: Manjula K <manjula@kafka-summit.org>
    
    Reviewers: Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4164 from manjuapu/ny-trivago-logos
  2. MINOR: Tighten up locking when aborting expired transactions

    apurvam committed with hachikuji Oct 31, 2017
    This is a followup to #4137
    
    Author: Apurva Mehta <apurva@confluent.io>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
    
    Closes #4146 from apurvam/MINOR-followups-to-bump-epoch-on-expire-patch
  3. MINOR: Fix inconsistency in StopReplica/LeaderAndIsr error counts

    hachikuji committed Oct 31, 2017
    Author: Jason Gustafson <jason@confluent.io>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #4147 from hachikuji/fix-error-inconsistencies
Commits on Oct 30, 2017
  1. MINOR: Code cleanup and JavaDoc improvements for clients and Streams

    mjsax committed with guozhangwang Oct 30, 2017
    Author: Matthias J. Sax <matthias@confluent.io>
    
    Reviewers: Bill Bejeck <bill@confluent.io>, Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4128 from mjsax/minor-cleanup
    
    minor fix
  2. KAFKA-5212; Consumer ListOffsets request can starve group heartbeats

    Richard Yu committed with hachikuji Oct 30, 2017
    Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #4110 from ConcurrencyPractitioner/trunk
  3. KAFKA-2903; FileRecords.read doesn't handle size > sizeInBytes when s…

    ijuma committed with hachikuji Oct 30, 2017
    …tart is not zero
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #4158 from ijuma/kafka-2903-file-records-read-slice-size-greater
  4. KAFKA-6130; Ensure VerifiableConsumer halts when --max-messages is re…

    tombentley committed with hachikuji Oct 30, 2017
    …ached
    
    Author: Tom Bentley <tbentley@redhat.com>
    
    Reviewers: Jason Gustafson <jason@confluent.io>
    
    Closes #4157 from tombentley/KAFKA-6130-verifiable-consumer-max-messages
  5. KAFKA-6073; Use ZookeeperClient in KafkaApis

    mimaison committed with junrao Oct 30, 2017
    I kept zkUtils for the call to AdminUtils.createTopic(). AdminUtils can be done in another PR.
    
    Is there a reason why we use TopicAndPartition instead of TopicPartition in KafkaControllerZkUtils ?
    
    Author: Mickael Maison <mickael.maison@gmail.com>
    
    Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
    
    Closes #4111 from mimaison/KAFKA-6073
Commits on Oct 27, 2017
  1. MINOR: Remove TLS renegotiation code

    ijuma committed Oct 27, 2017
    This has been disabled since the start and since
    it's removed in TLS 1.3, there are no plans to
    ever support it.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Closes #4034 from ijuma/remove-tls-renegotiation-support
  2. MINOR: Ensure that the producer in testAlterReplicaLogDirs is always …

    ijuma committed Oct 27, 2017
    …closed
    
    Failure to close the producer could cause a transient failure, more details
    below.
    
    The request timeout was only 2 seconds, exceptions thrown were not
    propagated and the producer would not be closed. If the exception
    was thrown during `send`, we did not increment `numMessages`
    allowing the test to pass.
    
    I have increased the timeout to 10 seconds and made sure that
    exceptions are propagated.
    
    Example of the error:
    
    ```text
    kafka.api.SaslSslAdminClientIntegrationTest > classMethod STARTED
    
    kafka.api.SaslSslAdminClientIntegrationTest > classMethod FAILED
        java.lang.AssertionError: Found unexpected threads, allThreads=Set(metrics-meter-tick-thread-2, Signal Dispatcher, main, Reference Handler, scala-execution-context-global-164, kafka-producer-network-thread | producer-1, scala-execution-context-global-166, Test worker, scala-execution-context-global-1249, /0:0:0:0:0:0:0:1:58910 to /0:0:0:0:0:0:0:1:43025 workers Thread 2, Finalizer, /0:0:0:0:0:0:0:1:58910 to /0:0:0:0:0:0:0:1:43025 workers Thread 3, scala-execution-context-global-163, metrics-meter-tick-thread-1)
    ```
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Closes #4144 from ijuma/ensure-producer-is-closed-test-alter-replica-log-dirs
  3. MINOR: Mention "per listener" security overrides in listener.security…

    omkreddy committed with ijuma Oct 27, 2017
    ….protocol.map config doc
    
    Author: Manikumar Reddy <manikumar.reddy@gmail.com>
    
    Reviewers: Tom Bentley <tbentley@redhat.com>, Ismael Juma <ismael@juma.me.uk>
    
    Closes #3951 from omkreddy/KIP-103-DOCS
  4. MINOR: Update docs wrt topic deletion being enabled by default

    omkreddy committed with ijuma Oct 27, 2017
    Author: Manikumar Reddy <manikumar.reddy@gmail.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #3835 from omkreddy/update-delete-topic-doc
  5. MINOR: Document "high watermark" magic value for delete records request

    ppatierno committed with ijuma Oct 27, 2017
    Author: Paolo Patierno <ppatierno@live.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #4119 from ppatierno/minor-delrecords-prot
  6. MINOR: Fix indentation in KafkaApis.handleOffsetFetchRequest

    vahidhashemian committed with ijuma Oct 27, 2017
    Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
    
    Reviewers: Ismael Juma <ismael@juma.me.uk>
    
    Closes #4139 from vahidhashemian/minor/indentation_fix_1710
  7. KAFKA-6119: Bump epoch when expiring transactions in the TransactionC…

    apurvam committed with guozhangwang Oct 27, 2017
    …oordinator
    
    A description of the problem is in the JIRA. I have added an integration test which reproduces the original scenario, and also added unit test cases.
    
    Author: Apurva Mehta <apurva@confluent.io>
    
    Reviewers: Jason Gustafson <jason@confluent.io>, Ted Yu <yuzhihong@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4137 from apurvam/KAFKA-6119-bump-epoch-when-expiring-transactions
  8. KAFKA-6131; Use atomic putIfAbsent to create txn marker queues

    rajinisivaram committed with ijuma Oct 27, 2017
    Author: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
    
    Closes #4140 from rajinisivaram/KAFKA-6131-txn-concurrentmap
Commits on Oct 26, 2017
  1. MINOR: Improve a Windows quickstart instruction

    vahidhashemian committed with guozhangwang Oct 26, 2017
    The output of `wmic` can be very long and could truncate the search keywords in the existing command. If those keywords are truncated no process is returned in the output. An update is suggested to the command by which the query is performed inside the `wmic` command itself instead of using pipes and `find`.
    
    Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
    
    Reviewers: Guozhang Wang <wangguoz@gmail.com>
    
    Closes #4083 from vahidhashemian/minor/improve_quickstart_for_windows_wmic
  2. KAFKA-6100: Down-grade RocksDB to 5.7.3

    guozhangwang committed Oct 26, 2017
    Author: Guozhang Wang <wangguoz@gmail.com>
    
    Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
    
    Closes #4136 from guozhangwang/K6100-rocksdb-580-regression
  3. MINOR: Add missing semicolon to example jaas configuration

    omkreddy committed with ijuma Oct 26, 2017
    Author: Manikumar Reddy <manikumar.reddy@gmail.com>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
    
    Closes #4101 from omkreddy/SCRAM-DOCS
  4. MINOR: Revert EmbeddedZooKeeper rename

    ijuma committed Oct 26, 2017
    Even though this class is internal, it's widely
    used by other projects and it's better to avoid
    breaking them until we have a publicly supported
    test library.
    
    Author: Ismael Juma <ismael@juma.me.uk>
    
    Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
    
    Closes #4138 from ijuma/revert-embedded-zookeeper-rename