Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) #6363

Merged
merged 66 commits into from
May 17, 2019
Merged

KAFKA-5505: Incremental cooperative rebalancing in Connect (KIP-415) #6363

merged 66 commits into from
May 17, 2019

Commits on May 16, 2019

  1. KIP-415: First iteration on IncrementalCooperativeConnectProtocol

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    421f827 View commit details
    Browse the repository at this point in the history

  2. KIP-415: Add Javadoc for V0 and V1 of the connect protocol

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d44c85c View commit details
    Browse the repository at this point in the history

  3. KIP-415: Add mockito as test dependency in connect:runtime

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    a9d9fb6 View commit details
    Browse the repository at this point in the history

  4. KIP-415: Add a sketch of unit tests for Connect protocol compatibilit… 

    …y (one fails)

KIP-415: Add connect protocol compatibility config and enum and extend metadata (one test fails)

KIP-415: Make WorkerCoordinator compatible with the incremental protocol (no revocations yet)

KIP-415: Separate protocol version and add assigments in metadata

KIP-415: Use extended inner classes and move revocation after join request
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    9c1a33b View commit details
    Browse the repository at this point in the history

  5. KIP-415: Include a necessary running(current) assignment in the herde… 

    …r and its LogContext
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    4f2e92b View commit details
    Browse the repository at this point in the history

  6. KIP-415: Integration tests for rebalancing

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    67cede9 View commit details
    Browse the repository at this point in the history

  7. KIP-415: Connect protocols to use Collection instead of List and empt… 

    …y list instead of null
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    31d275c View commit details
    Browse the repository at this point in the history

  8. KIP-415: Add test for task load balancing

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    4fa27ab View commit details
    Browse the repository at this point in the history

  9. KIP-415: Fix stop-the-world assignment for new tasks without new work… 

    …ers.

* Also add a separate connect scheduler (assignor) class.

KIP-415: Fix assignment balancing and add revocation logic

KIP-415: Base assignors to a common ConnectAssignor interface
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d7baa0f View commit details
    Browse the repository at this point in the history

  10. KIP-415: Introduce scheduled delay in IncrementalCooperativeConnectPr… 

    …otocol
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    86135a8 View commit details
    Browse the repository at this point in the history

  11. KIP-415: Rename rebalance integration test.

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    23d384a View commit details
    Browse the repository at this point in the history

  12. KIP-415: Rename modes to EAGER, COMPATIBLE and COOPERATIVE 

    KIP-415: Factor out ConnectAssignment and set current protocol during runtime from join response

KIP-415: draft delay based assignment

KIP-415: Fix connector deletes
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    b712dfa View commit details
    Browse the repository at this point in the history

  13. KIP-415: Handle lost assignments based on delayed rebalancing

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    40c9575 View commit details
    Browse the repository at this point in the history

  14. KIP-415: Use scheduled delay for rebalacing in every member

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    c287215 View commit details
    Browse the repository at this point in the history

  15. KIP-415: Adapt tests to changes in Metadata constructor

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    7046b8e View commit details
    Browse the repository at this point in the history

  16. KIP-415: Adjust code to changes in WorkerCoordinator and JoinGroupRes… 

    …ponse
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    095ae09 View commit details
    Browse the repository at this point in the history

  17. KIP-415: Fix unit tests after updates in JoinGroupRequest and JoinGro… 

    …upResponse
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    f1f59ef View commit details
    Browse the repository at this point in the history

  18. KIP-415: Add javadoc and checks for non nullable fields

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    2da8091 View commit details
    Browse the repository at this point in the history

  19. KIP-415: Replace forEachOrdered with forEach where applicable

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    0829a9c View commit details
    Browse the repository at this point in the history

  20. KIP-415: Remove unnecessary check for null in assignmentDifference

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    55c76ca View commit details
    Browse the repository at this point in the history

  21. KIP-415: Small fixes based on reviews

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    31088ed View commit details
    Browse the repository at this point in the history

  22. KIP-415: Add more guards against NPEs

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    51f9f66 View commit details
    Browse the repository at this point in the history

  23. KIP-415: Replace lambdas with method references

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    25002a0 View commit details
    Browse the repository at this point in the history

  24. KIP-415: Adapt to final version of KAFKA-8013

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    db52cbd View commit details
    Browse the repository at this point in the history

  25. KIP-415: Use Time class in RebalanceListener

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    1fc2745 View commit details
    Browse the repository at this point in the history

  26. KIP-415: Keep only two modes for connect protocol and make compatible… 

    … mode the default
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    bfd43df View commit details
    Browse the repository at this point in the history

  27. KIP-415: Document reference equality for ConnectAssignment

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    022f0d6 View commit details
    Browse the repository at this point in the history

  28. KIP-415: Remove unused method and rename test in IncrementalCooperati… 

    …veAssignor
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    7613331 View commit details
    Browse the repository at this point in the history

  29. KIP-415: Add a validator that accepts lambdas for inline validators

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    71c204a View commit details
    Browse the repository at this point in the history

  30. KIP-415: Add a validator for connect.protocol

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d29d48e View commit details
    Browse the repository at this point in the history

  31. KIP-415: Add note regarding thread safety of connect assignors in imp… 

    …lementions
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d50c38e View commit details
    Browse the repository at this point in the history

  32. KIP-415: Simplify test case

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    4e249b4 View commit details
    Browse the repository at this point in the history

  33. KIP-415: Simple clean up in connect protocols

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    eee31c9 View commit details
    Browse the repository at this point in the history

  34. KIP-415: Simplify enum mapping in ConnectProtocolCompatibility

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    479a7b1 View commit details
    Browse the repository at this point in the history

  35. KIP-415: Use Time class for delays as well

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    b42e1d7 View commit details
    Browse the repository at this point in the history

  36. KIP-415: Remove unused method clientId

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    0ad5243 View commit details
    Browse the repository at this point in the history

  37. KIP-415: Improve javadoc of WorkerRebalanceListener

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    8eed0eb View commit details
    Browse the repository at this point in the history

  38. KIP-415: Cleanup test code

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    0a6839b View commit details
    Browse the repository at this point in the history

  39. KIP-415: Add more javadoc in IncrementalCooperativeAssignor

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    f48da26 View commit details
    Browse the repository at this point in the history

  40. KIP-415: Add tests for the new assignor

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    0b9de70 View commit details
    Browse the repository at this point in the history

  41. KIP-415: Do not perform revocations in consecutive rebalances

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    4b590c9 View commit details
    Browse the repository at this point in the history

  42. KIP-415: Adjust new tests to changes upstream

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    90a4f86 View commit details
    Browse the repository at this point in the history

  43. KIP-415: Fix delay logic in new assignor

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    78d552d View commit details
    Browse the repository at this point in the history

  44. KIP-415: Reapply revocations if task assignment fails

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    00ef17b View commit details
    Browse the repository at this point in the history

  45. KIP-415: Fix checkstyle and reduce method length in IncrementalCooper… 

    …ativeAssignor
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    ddfb1f1 View commit details
    Browse the repository at this point in the history

  46. KIP-415: Depend on protocol returned by join responses

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    291d9dc View commit details
    Browse the repository at this point in the history

  47. KIP-415: Adjust code to changes introduced by KAFKA-8332

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d6ff6b7 View commit details
    Browse the repository at this point in the history

  48. KIP-415: Skip consecutive revocations correctly

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    8d05ff7 View commit details
    Browse the repository at this point in the history

  49. KIP-415: Fix requesting a rebalance when schedule rebalance timeout i… 

    …s reached
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    e6c2585 View commit details
    Browse the repository at this point in the history

  50. KIP-415: Refactor WorkerCoordinatorIncrementalTest

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    8bb2dfa View commit details
    Browse the repository at this point in the history

  51. KIP-415: Upgrade comment in protocol assignments to javadoc

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    d8234a9 View commit details
    Browse the repository at this point in the history

  52. KIP-415: Add validation for SCHEDULED_REBALANCE_MAX_DELAY_MS_CONFIG

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    8502032 View commit details
    Browse the repository at this point in the history

  53. KIP-415: More unit tests for the worker coordinator

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    8669030 View commit details
    Browse the repository at this point in the history

  54. KIP-415: Add herder unit tests for incremental cooperative rebalancing

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    44f2f59 View commit details
    Browse the repository at this point in the history

  55. KIP-415: Use a builder for WorkerLoad and ConnectorsAndTasks classes

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    c5d267d View commit details
    Browse the repository at this point in the history

  56. KIP-415: Remove warning that is a unreachable

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    a6e307e View commit details
    Browse the repository at this point in the history

  57. KIP-415: Improve and rearrange log messages in WorkerCoordinator#onJo… 

    …inComplete
    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    3def33c View commit details
    Browse the repository at this point in the history

  58. KIP-415: Fix handling of lost assignments and add specific unit tests

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    22a5f47 View commit details
    Browse the repository at this point in the history

  59. KIP-415: Factor out metadataRequest to a method in Connect protocols

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    fe66c21 View commit details
    Browse the repository at this point in the history

  60. KIP-415: Refactor ExtendedWorkerState as a separate public class

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    9545219 View commit details
    Browse the repository at this point in the history

  61. KIP-415: Rename ConnectAssignment to ExtendedAssignment

    @kkonstantine
    kkonstantine committed May 16, 2019
    Configuration menu
    Copy the full SHA
    e2de086 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2019

  1. KIP-415: Remove embed method from WorkerLoad which was missing in pre… 

    …vious commit
    @kkonstantine
    kkonstantine committed May 17, 2019
    Configuration menu
    Copy the full SHA
    90ca7d0 View commit details
    Browse the repository at this point in the history

  2. KIP-415: Improve logging and javadocs in IncrementalCooperativeAssignor

    @kkonstantine
    kkonstantine committed May 17, 2019
    Configuration menu
    Copy the full SHA
    b1c5424 View commit details
    Browse the repository at this point in the history

  3. KIP-415: Clarify javadocs

    @kkonstantine
    kkonstantine committed May 17, 2019
    Configuration menu
    Copy the full SHA
    cffcbad View commit details
    Browse the repository at this point in the history

  4. KIP-415: Rename enum method to fromProtocol to make meaning more clear

    @kkonstantine
    kkonstantine committed May 17, 2019
    Configuration menu
    Copy the full SHA
    59785bb View commit details
    Browse the repository at this point in the history

  5. KIP-415: Small fixes in connect integration tests to avoid flakiness

    @kkonstantine
    kkonstantine committed May 17, 2019
    Configuration menu
    Copy the full SHA
    eb1853d View commit details
    Browse the repository at this point in the history