KAFKA-20721: Removed timeout on task removal from state updater. by Nikita-Shupletsov · Pull Request #23093 · apache/kafka

Nikita-Shupletsov · 2026-08-06T00:42:02Z

Added logic to catch and propagate to the stream thread fatal state update errors.
Improved the error handling logic in the state updater thread. Now when the thread is failing it marks all tasks as failed.
Added logic to fail task removal futures if the thread is dead.

Reviewers: Matthias J. Sax matthias@confluent.io

Added logic to catch and propagate to the stream thread fatal state update errors. Improved the error handling logic in the state updater thread. Now when the thread is failing it marks all tasks as failed. Added logic to fail task removal futures if the thread is dead.

mjsax

Nice fix! Overall this make sense to me.

Few more Claude comments:

future.get() blocks the StreamThread inside the rebalance callback with zero log output — the old code at least printed "The state updater wasn't able to remove task X in time." Suggest looping future.get(1, MINUTES) with a WARN per iteration and retrying forever: keeps the fix, restores observability. Same applies to shutdownStateUpdater(), where KafkaStreams.close(Duration) will now return false and leak the StreamThread instead of the previously-bounded 5-min-per-future.

Integration test coverage regression. StateUpdaterFailureIntegrationTest was added by c48c50d (KAFKA-19831) specifically for "potential failures in Task#maybeCheckpoint". The PR deletes the flush() override that injected a ProcessorStateException during maybeCheckpointTasks and replaces it with a throw from onRestoreEnd, i.e. inside restoreTasks. Different handler — maybeCheckpointTasks catches StreamsException per task and continues; restoreTasks goes through handleStreamsException. The flush case should stay as a third Arguments rather than being replaced.

StateUpdater.shutdown() semantics changed for restored tasks. failRemainingTasks() moves everything in restoredActiveTasks into the failed queue, and shouldShutdownStateUpdater now asserts that. Those tasks restored successfully; being "failed" means closeTaskDirty, discarding the checkpoint. It doesn't bite production today because shutdownStateUpdater() drains them via removeRestoredTask() before calling shutdown() — but it makes shutdown() unsafe to call on its own, which the unit test now enshrines as expected behavior.

mjsax · 2026-08-06T05:18:51Z

-                clearInputQueue();
-                clearUpdatingAndPausedTasks();
+                failRemainingTasks();
+                failPendingActions();


We are changing the order here -- is this an intended change?

yes, it's in case we are waiting for removal, for example. once we fail the task, the thread is unblocked. so we want to do that after failing everything else and marking the state updater as stopped

Might be worth a comment that the order is load bearing

mjsax · 2026-08-06T05:31:13Z

        if (stateUpdater.hasExceptionsAndFailedTasks()) {
            handleExceptionsFromStateUpdater();
        }
+        maybeThrowFatalExceptionFromStateUpdater();


What is the reason we insert this check exactly here, but not at the very beginning or very end?

If the state-updater is already dead, why would we want to still add tasks?

it goes after handleExceptionsFromStateUpdater, because it calls drainExceptionsAndFailedTasks and processes them(adds them to the failed tasks). so we first let that logic do its thing, then we check if we need to fail the thread

But what if state-updater is already dead before-hand? Also, why not check after the removal step completed too?

Claude also has concerns (did not verify the details):

The current code assumes handleExceptionsFromStateUpdater() returns. It can't when it has anything to report: maybeThrowTaskExceptions throws on all three branches (lastFatal, lastTaskMigrated, aggregated TaskCorrupted). And handleFatalThrowable stamps the fatal exception onto every owned task while also setting fatalException. So in the scenario the check exists for, line 872 is unreachable; it only fires when the updater died owning zero tasks.

The concrete damage shows up when addTasksToStateUpdater() — which still runs first, per your follow-up question — has pending tasks in the same iteration. Those get failAction'd with the generic StreamsException("The state updater is not running.") and land after the real cause in the drain order. maybeThrowTaskExceptions keeps the last plain StreamsException as lastFatal, so the StreamThread dies with "The state updater is not running." and the actual cause survives only in the state updater's own ERROR log. That directly defeats the comment Nikita wrote three lines above fatalException.set(exception): "publish the exception before reporting the tasks as failed, so that the stream thread never sees a task that failed because this thread died without also seeing why this thread died."

Moving the check to the top of checkStateUpdater, ahead of addTasksToStateUpdater(), fixes both that and your "why would we still add tasks to a dead updater" question in one move.

mjsax · 2026-08-06T05:35:33Z

            } catch (final StreamsException streamsException) {
                handleStreamsExceptionWithTask(streamsException, taskId);
                future.completeExceptionally(streamsException);
-            } catch (final RuntimeException runtimeException) {


Why do we keep StreamsException as-is but only remove RuntimeException?

because we don't stop the thread on StreamsException, but we do on RuntimeException. so in order to unify the shutdown and cleanup logic, I remove that catch block. so now the exception will be caught in the run method. it allows us to have the cleanup logic in one place instead of two

because we don't stop the thread on StreamsException

We don't? This sounds off? A StreamsException is supposed to be treated as fatal... I find this a little bit confusing. Maybe I am missing something.

when we receive a StreamsException we shutdown the stream thread as well. it's just that a StreamsException is an expected exception, so we deal with it gracefully: we put the task in the list of failed tasks and then the TaskManager will deal with them in maybeThrowTaskExceptions. so we expect that something will throw a StreamsException and we have a graceful handling of it. We don't expect other kinds of exceptions/errors, so we treat them as a stop the world sort of event

mjsax · 2026-08-06T05:39:37Z

+            log.error("An unexpected error occurred within the state updater thread: {}", String.valueOf(throwable));
+            final RuntimeException exception = throwable instanceof RuntimeException
+                ? (RuntimeException) throwable
+                : new StreamsException("The state updater thread failed with a fatal error.", throwable);


Not sure why we need this ? Both a general RuntimeException, StreamsException, and Throwable are all fatal. So why these different cases?

ExceptionAndTask works only with RuntimeException. so we need to wrap the Throwable into one in order to be able to pass it there(to addToExceptionsAndFailedTasksThenClearUpdatingAndPausedTasks).
I decided to keep fatalException the same for the sake of consistency, but it can be left as a throwable

Could it make sense to change ExceptionAndTask?

I wouldn't. mostly because we rely on everything to be an uchecked(runtime) exception. Throwable is checked. so we would need to change every place that deals with that exception downstream to be able to throw the Throwable. which would basically mean a similar check and a similar wrapping.

Nikita-Shupletsov · 2026-08-06T17:48:26Z

future.get() blocks the StreamThread inside the rebalance callback with zero log output — the old code at least printed "The state updater wasn't able to remove task X in time." Suggest looping future.get(1, MINUTES) with a WARN per iteration and retrying forever: keeps the fix, restores observability. Same applies to shutdownStateUpdater(), where KafkaStreams.close(Duration) will now return false and leak the StreamThread instead of the previously-bounded 5-min-
per-future.

sounds good. will do

Integration test coverage regression. StateUpdaterFailureIntegrationTest was added by c48c50d (KAFKA-19831) specifically for "potential failures in Task#maybeCheckpoint". The PR deletes the flush() override that injected a ProcessorStateException during maybeCheckpointTasks and replaces it with a throw from onRestoreEnd, i.e. inside restoreTasks. Different handler — maybeCheckpointTasks catches StreamsException per task and continues; restoreTasks goes through handleStreamsException. The flush case should stay as a third Arguments rather than being replaced.

flush is no longer called because of the changes from https://issues.apache.org/jira/browse/KAFKA-19712. the original test kept passing because the failure was not induced anymore.
That's why I rewrote the test a bit, so we test the old case again faithfully.

StateUpdater.shutdown() semantics changed for restored tasks. failRemainingTasks() moves everything in restoredActiveTasks into the failed queue, and shouldShutdownStateUpdater now asserts that. Those tasks restored successfully; being "failed" means closeTaskDirty, discarding the checkpoint. It doesn't bite production today because shutdownStateUpdater() drains them via removeRestoredTask() before calling shutdown() — but it makes shutdown() unsafe to call on its own, which the unit test now enshrines as expected behavior.

so yes. it's another edge case my claude found: right now we don't and can't reuse a state updater. it lives as long as the thread it belongs to lives. but there was a unit test that was testing that we can shutdown and restart the same state updater which never happens in the real life. I decided to tightened it a bit to adhere to how we actually use it, to make it less errorprone. If we ever decide to reuse state updater or change the relationship between stream threads and state updater, there will at least be exceptions showing what we need to change instead of being in this gray zone.

Nikita-Shupletsov · 2026-08-06T18:31:50Z

Thank you @mjsax for reviewing the PR!

I answered and addressed yuor comments. please let me know if they make sense.

mjsax · 2026-08-07T16:24:32Z

+                    log.warn("Waiting for the removal of task {} from the state updater for {} minute(s).", taskId, minutesWaited);
+                }
+            }
            if (removedTaskResult == null) {


Nit (also from Claude): we can move this check inside the loop, including return removedTaskResult; plus the declaration of the variable, and can use final StateUpdater.RemovedTaskResult removedTaskResult = future.get(REMOVAL_LOG_INTERVAL_MINUTES, TimeUnit.MINUTES);.

Might make the code a little cleaner?

mjsax · 2026-08-07T16:25:57Z

+                    break;
+                } catch (final java.util.concurrent.TimeoutException retryTimeout) {
+                    minutesWaited += REMOVAL_LOG_INTERVAL_MINUTES;
+                    log.warn("Waiting for the removal of task {} from the state updater for {} minute(s).", taskId, minutesWaited);


Should we additionally keep the original hint about The state updater thread may be dead ?

I don't think so. as we are changing the approach here: if the thread dies, every task needs to be failed unconditionally. previously we thought about it as the thread shouldn't just die, so when it happened, we wanted to flag it.
Now it just means the task is taking longer(as it was reported in the ticket)

mjsax

Made another pass.

github-actions Bot added triage PRs from the community streams labels Aug 6, 2026

mjsax reviewed Aug 6, 2026

View reviewed changes

mjsax added ci-approved and removed triage PRs from the community labels Aug 6, 2026

Addressed review comments.

6183a2f

mjsax reviewed Aug 7, 2026

View reviewed changes

Nikita-Shupletsov added 2 commits August 7, 2026 11:29

Addressed comments.

9252195

Merge remote-tracking branch 'origin/trunk' into KAFKA-20721

27a0ada

Uh oh!

Conversation

Nikita-Shupletsov commented Aug 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Nikita-Shupletsov commented Aug 6, 2026

Uh oh!

Nikita-Shupletsov commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Nikita-Shupletsov commented Aug 6, 2026 •

edited by github-actions Bot

Loading