KAFKA-10199: Remove lost tasks in state updater with new remove #15870
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucasbru
reviewed
May 6, 2024
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java
Show resolved
Hide resolved
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java
Show resolved
Hide resolved
| } catch (final ExecutionException executionException) { | ||
| log.warn("An exception happened when removing task {} from the state updater. The exception will be handled later: ", | ||
| taskId, executionException); | ||
| } catch (final InterruptedException ignored) { } |
There was a problem hiding this comment.
Can we just ignore this? I see other classes in the package rethrowing it
There was a problem hiding this comment.
We are quite inconsistent on how we treat InterruptedException. In some places we ignore them because they should not happen and in others we treat them as fatal and throw an IllegalStateException because they should not happen [1].
[1]
There was a problem hiding this comment.
I am actually in favor of treating them as fatal and throw an IllegalStateException to make it more explicit that interruption of a stream thread should not happen. WDYT?
| @@ -1421,15 +1422,20 @@ public void shouldRemoveAllActiveTasksFromStateUpdaterOnPartitionLost() { | |||
| .withInputPartitions(taskId02Partitions).build(); | |||
| final TasksRegistry tasks = mock(TasksRegistry.class); | |||
| final TaskManager taskManager = setupForRevocationAndLost(mkSet(task1, task2, task3), tasks); | |||
| final CompletableFuture<StateUpdater.RemovedTaskResult> future1 = new CompletableFuture<>(); | |||
There was a problem hiding this comment.
Do we need a test that covers the part where we get an exception?
lucasbru
approved these changes
May 7, 2024
gongxuanzhang
pushed a commit
to gongxuanzhang/kafka
that referenced
this pull request
Jun 12, 2024
…he#15870) Uses the new remove operation of the state updater that returns a future to remove lost tasks from the state udpater. Reviewer: Lucas Brutschy <lbrutschy@confluent.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses the new remove operation of the state updater that returns a future to remove lost tasks from the state udpater.
Committer Checklist (excluded from commit message)