New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAFKA-9472: Remove deleted Connect tasks from status store #8118
Conversation
@ncliang @gharris1727, would either you mind taking a look when you have a chance? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very concise change. Just one question about the integration test, otherwise LGTM.
...runtime/src/test/java/org/apache/kafka/connect/integration/ConnectWorkerIntegrationTest.java
Show resolved
Hide resolved
Thanks for the review, @ncliang! @rhauch @kkonstantine would you mind taking a look when you have a chance? |
ok to test |
retest this please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Thanks @C0urante.
One comment on the javadoc, aiming to keep it similar to the other javadocs.
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/TaskStatus.java
Outdated
Show resolved
Hide resolved
ok to test |
retest this please |
jdk11 succeeded, and previously jdk8 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Although the statuses for tasks are removed from the status store when their connector is deleted, their statuses are not removed when only the task is deleted, which happens in the case that the number of tasks for a connector is reduced. This commit adds logic for deleting the statuses for those tasks from the status store whenever a rebalance has completed and the leader of a distributed cluster has detected that there are recently-deleted tasks. Standalone is also updated to accomplish this. Unit tests for the `DistributedHerder` and `StandaloneHerder` classes are updated and an integration test has been added. Reviewers: Nigel Liang <nigel@nigelliang.com>, Konstantine Karantasis <konstantine@confluent.io>
Although the statuses for tasks are removed from the status store when their connector is deleted, their statuses are not removed when only the task is deleted, which happens in the case that the number of tasks for a connector is reduced. This commit adds logic for deleting the statuses for those tasks from the status store whenever a rebalance has completed and the leader of a distributed cluster has detected that there are recently-deleted tasks. Standalone is also updated to accomplish this. Unit tests for the `DistributedHerder` and `StandaloneHerder` classes are updated and an integration test has been added. Reviewers: Nigel Liang <nigel@nigelliang.com>, Konstantine Karantasis <konstantine@confluent.io>
Although the statuses for tasks are removed from the status store when their connector is deleted, their statuses are not removed when only the task is deleted, which happens in the case that the number of tasks for a connector is reduced. This commit adds logic for deleting the statuses for those tasks from the status store whenever a rebalance has completed and the leader of a distributed cluster has detected that there are recently-deleted tasks. Standalone is also updated to accomplish this. Unit tests for the `DistributedHerder` and `StandaloneHerder` classes are updated and an integration test has been added. Reviewers: Nigel Liang <nigel@nigelliang.com>, Konstantine Karantasis <konstantine@confluent.io>
Jira
Although the statuses for tasks are removed from the status store when their connector is deleted, their statuses are not removed when only the task is deleted, which happens in the case that the number of tasks for a connector is reduced.
These changes add logic for deleting the statuses for those tasks from the status store whenever a rebalance has completed and the leader of a distributed cluster has detected that there are recently-deleted tasks. Standalone is also updated to accomplish this.
Unit tests for the
DistributedHerder
andStandaloneHerder
classes are updated, and an integration test has been added.