[KRaft] Stale ControllerRegistrationRecord for removed controller ID 8 blocks metadata.version 3.9-IV0 and is invisible to Strimzi status #13117
Unanswered
erikmartos18
asked this question in
Q&A
Replies: 2 comments 2 replies
|
Kafka does not currently support unregistration of controller nodes. So until that is fixed, your only option is to add the nodes with given ID 🤷. This has been discussed multiple times in the past, so please look for the previous discussions. If there is something not answered there, please provide exact questions. |
2 replies
|
This issue related to missing controllers unregistration should not happen anymore when using Apache Kafka 4.4.0 in the upcoming new Strimzi release. It was fixed via #13296, supporting the new unregister controller API within Kafka. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
We are requesting an official Strimzi position and a supported in-place recovery procedure for a production Kafka cluster blocked by stale
ControllerRegistrationRecordentries.The cluster is healthy and
Readyon Kafka 3.9.0, butmetadata.versionmust remain pinned at3.8-IV0because the KRaft metadata image still contains a registration for a removed controller ID8that advertisesmetadata.versionsupport only up to version 20.The current controller quorum consists exclusively of IDs 5, 6 and 7. Controller ID 8 has no Pod, no PVC, no
KafkaNodePoolassignment, noStrimziPodSetentry, and no membership inCurrentVoters. It is entirely absent fromKafka.status.registeredNodeIds. Nevertheless, Kafka evaluates its persisted registration during feature compatibility validation and deterministically rejects the upgrade.This is not a transient reconciliation warning — it is persistent KRaft metadata, present in every metadata snapshot.
We acknowledge that the historical replacement of a controller-role
KafkaNodePoolin a static quorum is outside the operations supported by Strimzi. However, the operator accepted and reconciled that desired state without detecting that stale controller registrations would remain and later block the Kafka metadata lifecycle.The issue therefore exposes three gaps:
Environment
kafkakafka-clusterclusterIdgxSj-ATITr6ZVsclWccOLgstrimzi-cluster-operator-v0.45.0)status.operatorLastSuccessfulVersionmetadata.versionmetadata.versionkafkaMetadataStatekraft.version(finalized)Ready=True(+Warning/KafkaMetadataVersion)Cluster-level annotations:
strimzi.io/kraft: enabled,strimzi.io/node-pools: enabled.No
strimzi.io/next-node-idsorstrimzi.io/remove-node-idsannotations are currently present on either pool.Current Strimzi state
KafkaNodePoolassignments:status.nodeIdsbroker-kafka-cluster[broker][0, 1, 2]controller-kafka-cluster-dc1[controller][5, 6, 7]Kafkastatus:Controller ID 8 does not appear in Strimzi status, in any
KafkaNodePool, in anyStrimziPodSet, in any Pod, in any PVC, or in any ConfigMap.We independently verified that no Kubernetes-side trace of IDs 3, 4, 8 or 11 remains for this cluster:
broker-...-0/1/2andcontroller-...-dc1-5/6/7data-...-broker-...-0/1/2anddata-...-controller-...-dc1-5/6/7Released/Terminatingvolumes bound to this namespace (reclaim policyDelete)StrimziPodSetspec.pods:[0,1,2]and[5,6,7]The only place where IDs 3, 4, 8 and 11 still exist is the KRaft metadata log.
Current KRaft quorum
Note the
directoryId: nulland all-zeroDirectoryIdon the voters, consistent with a static quorum (kraft.version=0).Controller ID 8 is not a current voter and is not required to maintain the active quorum.
Current finalized features
All live nodes support up to 3.9-IV0.
kraft.versionremains finalized at 0 (static quorum).Deterministic reproduction
oc exec -n kafka -c kafka kafka-cluster-broker-kafka-cluster-0 -- \ bin/kafka-features.sh --bootstrap-server localhost:9092 \ upgrade --metadata 3.9-IV0 --dry-runResult:
This is fully reproducible and proves that controller ID 8 is still part of Kafka's effective controller registration state and is the direct blocker for
metadata.version3.9-IV0.KRaft metadata evidence
Decoded from the latest metadata snapshot on the leader controller
(
/var/lib/kafka/data/kafka-log5/__cluster_metadata-0/00000000000072362029-0000001044.checkpoint,reproduced identically in the preceding snapshot
...072354831-0000001044.checkpoint):(These are snapshot offsets; the snapshot materializes current state, so the consecutive offsets and identical
CreateTimedo not indicate simultaneous registration.)The full record for the blocking controller ID 8:
{"type":"REGISTER_CONTROLLER_RECORD","version":0,"data":{ "controllerId":8, "incarnationId":"U9rVAIPzTESlIhFE73_vew", "zkMigrationReady":false, "endPoints":[{"name":"CONTROLPLANE-9090","host":"0.0.0.0","port":9090,"securityProtocol":1}], "features":[{"name":"metadata.version","minSupportedVersion":1,"maxSupportedVersion":20}] }}Compare with a healthy, live controller (ID 5):
{"type":"REGISTER_CONTROLLER_RECORD","version":0,"data":{ "controllerId":5, "incarnationId":"tA8W_BwcQCSEiunGRAOI4A", "zkMigrationReady":false, "endPoints":[{"name":"CONTROLPLANE-9090","host":"kafka-cluster-controller-kafka-cluster-dc1-5.kafka-cluster-kafka-brokers.kafka.svc","port":9090,"securityProtocol":1}], "features":[{"name":"kraft.version","minSupportedVersion":0,"maxSupportedVersion":1}, {"name":"metadata.version","minSupportedVersion":1,"maxSupportedVersion":21}] }}Two anomalies stand out in the ID 8 record and may be the key to understanding how this state was reached:
0.0.0.0instead of the pod FQDN that every other controller advertises.kraft.versionfeature at all, whereas every other registration advertiseskraft.version0–1. Its serializedvalueSizeis 80 bytes versus ~170 for the others.This strongly suggests controller ID 8 registered from a node that had not completed initialization (or was running an older image) and never re-registered afterwards, leaving a permanently truncated feature set.
Endpoint hostnames also confirm the provenance of the stale entries — IDs 3, 4 and 11 advertise
kafka-cluster-controller-kafka-cluster-{3,4,11}..., i.e. the previous controller pool naming (controller-kafka-cluster), while the live controllers use the current pool naming (controller-kafka-cluster-dc1). All of these belong to this cluster (clusterId gxSj-ATITr6ZVsclWccOLg), not to any other cluster.Historical context
The reported historical operation replaced the previous controller-role
KafkaNodePool(controller-kafka-cluster, IDs 3, 4, 8, 11) with a new controller-roleKafkaNodePool(controller-kafka-cluster-dc1, IDs 5, 6, 7), created on 2026-08-16.Controller ID 8 appears to have registered while in an incomplete state and was removed before ever registering with the full Kafka 3.9.0 feature range, so its last persisted
ControllerRegistrationRecordstill advertisesmetadata.versionsupport only through version 20.We acknowledge that adding, removing, scaling or replacing controllers in a static KRaft quorum is documented as unsupported in Strimzi 0.45.0. We are not asking Strimzi to classify that historical operation as supported. We are asking for an officially supported recovery path now that the operator has already reconciled such a state, and for guardrails preventing other clusters from reaching this condition.
Observed behavior
Readyand fully operational; the data plane is healthy.CurrentVoters.metadata.version3.9-IV0 fails exclusively because ID 8 advertises support only through version 20.kafka-cluster.sh unregisterapplies to broker registrations, not controller registrations.kafka-metadata-quorum.sh remove-controllerdoes not clean up this staleControllerRegistrationRecordin an existing static Kafka 3.9 quorum.kraft.versionfinalized at 0).UnregisterControllerRecordrequires a newerMetadataVersion— and this stale registration is precisely what prevents reaching it.Expected behavior
At least one of the following should exist:
CurrentVoters.metadata.version3.8-IV0.KafkaNodePoolmutations before Kubernetes resources and KRaft metadata diverge.KafkaNodePoolassignments.Production impact
The data plane is operational, but the cluster lifecycle is blocked:
metadata.versioncannot be finalized at 3.9-IV0 and must stay pinned at 3.8-IV0;Warning/KafkaMetadataVersioncondition;Kafkaresource isReady;Refreshing only ID 8 to version 21 would unblock the immediate 3.9-IV0 transition, but would not clean the metadata image — the registrations for IDs 3, 4 and 11 would remain and could block later
MetadataVersions. We therefore need both an immediate recovery answer and a permanent cleanup strategy.Actions already analyzed and ruled out
The following are not valid cleanup procedures in this environment:
status.registeredNodeIdskafka-cluster.sh unregisterfor controller ID 8kafka-metadata-quorum.sh remove-controllerkafka-features.sh upgradewithout--dry-run(fails identically)Specific questions for Strimzi maintainers
ControllerRegistrationRecordfrom a Kafka 3.9.0 cluster running Strimzi 0.45.0 withkraft.version=0?KafkaNodePooltransition and rolling sequence does Strimzi require to prevent voter-set divergence or quorum loss?metadata.version3.9-IV0, how should the registrations for IDs 3, 4, 8 and 11 be permanently removed?0.0.0.0, missingkraft.versionfeature) a known failure mode of a controller that registers before completing initialization? Could Strimzi prevent a partially-initialized controller from persisting a registration?KafkaNodePoolcontroller IDs,Kafka.status.registeredNodeIds,CurrentVoters, and the effective controller registrations in the KRaft metadata image?KafkaNodePoolreplacements that are unsupported with a static quorum, instead of reconciling them?MetadataVersionupgrade?MetadataVersionthatUnregisterControllerRecordrequires?Requested outcome
We request one of the following:
KafkaNodePoolmanifests, controller rollout order, voter-set expectations at each stage, quorum health gates, rollback procedure, and final cleanup of all stale registrations.A procedure that merely refreshes ID 8 is not sufficient on its own unless it also explains how the remaining stale registrations will be handled for subsequent metadata upgrades.
Related upstream work
Final assessment
This is a confirmed, reproducible, persistent KRaft metadata condition — not a cosmetic Strimzi warning. The active quorum is healthy, but controller ID 8 remains registered in the effective KRaft metadata image with
maxSupportedVersion=20and deterministically blocksmetadata.version3.9-IV0.The cluster cannot use the KIP-1312 unregister mechanism, because that mechanism requires a
MetadataVersionthe cluster cannot reach while the stale registration remains. This creates a recovery deadlock for already-affected clusters.Until Strimzi provides or approves an in-place recovery path, the only safe operational position is to keep
metadata.versionpinned at 3.8-IV0 and avoid any unsupported change to the controller voter set. We need an authoritative maintainer answer before performing any controller recreation,KafkaNodePooltransition or metadata manipulation in production.All reactions