Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka Index Task that supports Incremental handoffs #4815

Merged
merged 14 commits into from
Nov 17, 2017
Merged

Kafka Index Task that supports Incremental handoffs #4815

merged 14 commits into from
Nov 17, 2017

Commits on Nov 16, 2017

  1. Kafka Index Task that supports Incremental handoffs 

    - Incrementally handoff segments when they hit maxRowsPerSegment limit
- Decouple segment partitioning from Kafka partitioning, all records from consumed partitions go to a single druid segment
- Support for restoring task on middle manager restarts by check pointing end offsets for segments
    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    d350830 View commit details
    Browse the repository at this point in the history

  2. take care of review comments

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    9ecd5d1 View commit details
    Browse the repository at this point in the history

  3. make getCurrentOffsets call async, keep track of publishing sequence,… 

    … review comments
    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    1597a6a View commit details
    Browse the repository at this point in the history

  4. fix setEndoffset duplicate request handling, formatting

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    04bc451 View commit details
    Browse the repository at this point in the history

  5. fix unit test

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    2320517 View commit details
    Browse the repository at this point in the history

  6. backward compatibility

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    53ca5dc View commit details
    Browse the repository at this point in the history

  7. make AppenderatorDriverMetadata backwards compatible

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    06c86ab View commit details
    Browse the repository at this point in the history

  8. add unit test

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    4d9194e View commit details
    Browse the repository at this point in the history

  9. fix deadlock between persist and push executors in AppenderatorImpl

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    c3b68cf View commit details
    Browse the repository at this point in the history

  10. fix formatting

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    8295673 View commit details
    Browse the repository at this point in the history

  11. use persist dir instead of work dir

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    80cc3bb View commit details
    Browse the repository at this point in the history

  12. review comments

    @pjain1
    pjain1 committed Nov 16, 2017
    Configuration menu
    Copy the full SHA
    32da00b View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2017

  1. fix deadlock

    @pjain1
    pjain1 committed Nov 17, 2017
    Configuration menu
    Copy the full SHA
    6a161ef View commit details
    Browse the repository at this point in the history

  2. actually fix deadlock

    @pjain1
    pjain1 committed Nov 17, 2017
    Configuration menu
    Copy the full SHA
    4335dc6 View commit details
    Browse the repository at this point in the history