[Bug]: Kafka rolling reconciliation can fail and loop during Strimzi-driven rolling updates instead of waiting for ISR recovery #12650
Replies: 1 comment 9 replies
|
To be honest, I'm not sure what is supposed to be a bug here. The oeprator does not wait forever for the broker to get in-sync. It will fial the reconciliation after osme time. But the next reconciliation continues where it left of by waiting. That is by design. This is importnt feature because it allows th eoperator to step back and check if there isn't some issue with some dependent resource etc. So this is fully intentional and having the operator wait forever would serve no purpose. Also, I want to point out that the init-container has only a very ismple task of getting the node information. If that takes minutes, there is some major issue in your Kubernetes cluster. That said, the code is ready for it and it should not matter how long the init phase takes. The behaviour sounds correct. |
Uh oh!
There was an error while loading. Please reload this page.
Bug Description
Summary
Strimzi 0.40.0 appears to fail a valid broker rolling update flow when a restart causes temporary ISR shrinkage, even though the cluster is healthy before the operation and would likely become safe to continue if the operator waited longer.
This was first observed during migration from a Strimzi-managed client CA to an externally managed client CA, but it does not appear to be specific to client-CA migration. It seems applicable to any Strimzi-managed broker rolling update.
In this environment, each broker pod has an init phase before Kafka starts, and that init phase alone can take about 5 minutes. After that, the broker still needs additional time to rejoin Kafka, catch up replication, and re-enter ISR. On a large and active cluster, that end-to-end recovery time can exceed the operator’s current retry window.
Observed behavior
The cluster starts healthy (no URPs, all brokers healthy). Strimzi begins a rolling update, restarts one broker, and then evaluates the next broker while the previous one is still catching up and has not fully re-entered ISR. At that point, Strimzi decides the next restart is unsafe because partitions would become under-replicated.
Instead of continuing to wait for ISR recovery, the operator retries for a fixed budget (10 attempts / ~127750 ms), fails reconciliation, and then later starts a new reconciliation that re-enters the same restart/fail/requeue pattern for the same Kafka cluster in the same namespace.
This effectively turns a valid rolling update into a restart loop when broker recovery time is longer than the operator’s wait budget.
Why this seems like a bug
The ISR safety check itself makes sense. The issue is that a temporary and expected recovery state during rolling maintenance is treated as a reconciliation failure instead of a wait condition.
In practice, the operator:
Scope and impact
This is likely broader than client-CA migration. The same pattern should be possible in any Strimzi-driven broker rolling update, including certificate/CA changes, Kafka configuration changes requiring broker restarts, image/version changes, or other Strimzi-managed broker rolls.
This makes rolling updates unreliable on large, active clusters, especially when broker pods have significant init/startup delay and ISR recovery takes longer than the operator is willing to wait.
Request
Could you please confirm whether this is considered a bug?
If yes, would you consider one or more of the following improvements?
Steps to reproduce
Steps to reproduce
Deploy a Strimzi 0.40.0 Kafka cluster with multiple brokers, for example 9 brokers.
Ensure broker restarts are slow enough that a restarted broker does not become safe for the next rolling step within the operator’s retry budget:
Create or use topics with:
Keep the cluster under normal active load so that, after one broker restart, the restarted broker does not immediately re-enter ISR for all partitions it hosts.
Trigger any Strimzi-managed rolling update that requires sequential broker restarts.
Allow Strimzi to restart the first broker.
Wait until the restarted broker pod becomes Kubernetes-Ready, but before it has fully caught up and re-entered ISR for the partitions it hosts.
Observe Strimzi evaluating the next broker for restart and reporting messages such as:
will be under-replicated ... if broker X is restartedPod ... cannot be updated right nowObserve KafkaRoller retrying only for a bounded retry window and then failing with messages such as:
giving up after 10 attemptsTotal delay between attempts 127750msObserve the reconciliation fail, update the Kafka resource status, and then a new reconciliation start again for the same namespace / same cluster.
Observe the same pattern repeat:
Why this reproduces the issue
The key condition is that the total time for a restarted broker to become safe for the next rolling step is longer than the KafkaRoller retry budget:
If that combined time exceeds the fixed retry window, the reconciliation fails even though the cluster may become safe again if the operator simply waits longer.
Expected behavior
Expected behavior
During a Strimzi-driven rolling operation, if the next broker is temporarily unsafe to restart because ISR has not yet recovered after the previous broker restart, the operator should continue waiting until the cluster becomes safe to proceed, or at least allow this waiting behavior to be configured.
In particular, the operator should not assume that broker recovery is bounded only by container restart time. In real environments, recovery can include pod init time, broker startup time, partition catch-up time, and ISR re-entry time.
It should not repeatedly fail the reconciliation and restart the process for the same cluster when the blocking condition is temporary and caused by the operator-driven rolling restart itself.
Strimzi version
0.40.0
Kubernetes version
v1.31.1
Installation method
helm chart
Infrastructure
No response
Configuration files and logs
Environment
0..8appear in the logs)Notes on redaction
I redacted:
<redacted-namespace><redacted-cluster>The same placeholder is used throughout so it is clear that the repeated restarts / failures are happening against the same namespace and same Kafka cluster.
Logs (redacted)
Log set - affected namespace only (2026-03-19)
Additional context
Notes
All reactions