Stale KRaft ControllerRegistration IDs block metadata upgrade to 3.9-IV0 from 3.7.IV4 after controller NodePool ID migration

[ChetanKoneru]

We have a Strimzi KRaft cluster on AKS that is healthy and running Kafka 3.9.2, but it cannot finalize the metadata upgrade from 3.7-IV4 to 3.9-IV0 after upgrading from Strimzi 0.42.0 to 0.45.2, Kafka from 3.7.1 to 3.9.2.

The controller rejects the metadata feature update because historical controller registrations remain in the metadata log:

InvalidUpdateVersionException:
Invalid update version 21 for feature metadata.version.
Controller 5 only supports versions 1-19

The active cluster is healthy and reports =Ready=True= , but its metadata version remains 3.7-IV4.

Environment

Strimzi: 0.45.2
Operator image: strimzi/operator:0.45.2
Kafka: 3.9.2
Platform: AKS / Kubernetes
Kafka metadata state: KRaft
Current metadata version: 3.7-IV4
Requested metadata version: 3.9-IV0

Current active nodes:

Brokers:     0, 1, 2
Controllers: 90, 91, 92
status.registeredNodeIds: [0,1,2,90,91,92]

Historical NodePool ID allocation

Initially, the controller NodePool did not have reserved node-ID ranges. During the ZooKeeper-to-KRaft migration, Strimzi assigned controllers IDs 3, 4, and 5, while brokers used 0, 1, and 2.

We later introduced explicit allocation ranges:

# Kafka broker NodePool
metadata:
  annotations:
    strimzi.io/next-node-ids: "[0-49]"

# Kafka controller NodePool
metadata:
  annotations:
    strimzi.io/next-node-ids: "[90-99]"

The active controller NodePool now uses IDs 90, 91, and 92. The original controllers 3, 4, and 5 are no longer Pods or quorum voters, but their registrations remain in the KRaft metadata.

Relevant historical registrations observed in controller logs:

ControllerRegistration(id=3, supportedFeatures={metadata.version: 1-19})
ControllerRegistration(id=4, supportedFeatures={metadata.version: 1-19})
ControllerRegistration(id=5, supportedFeatures={metadata.version: 1-19})

The active controllers and brokers re-registered with support for metadata feature levels 1-21.

Reproduction / observed result

With:

spec:
  kafka:
    version: 3.9.2
    metadataVersion: "3.9-IV0"

Strimzi attempts the metadata feature update, but Kafka rejects it:

Failed to update metadata version to 3.9-IV0
InvalidUpdateVersionException:
Invalid update version 21 for feature metadata.version.
Controller 5 only supports versions 1-19

Current safe mitigation

We changed only the desired metadata version back to the finalized version:

spec:
  kafka:
    version: 3.9.2
    metadataVersion: "3.7-IV4"

This retains Kafka 3.9.2, avoids repeated failed update attempts, and the cluster is Ready=True.

The remaining status warning is expected:

Metadata version is older than the Kafka version used by the cluster,
which suggests that an upgrade is incomplete.

Due diligence performed

• Confirmed the active KRaft voters are only 90,91,92; IDs 3,4,5 are not current controller Pods or voters.

• Confirmed active nodes support metadata feature level 21; the historical controller registrations cap support at 19.

• Reviewed Strimzi discussion Unable to upgrade to 3.9.0: Could not upgrade metadata.version to 21 #11147, which identifies the static-controller-quorum limitation for this class of issue.

• Reviewed KAFKA-17094 / KIP-1073. Kafka 4.0 adds visibility of fenced brokers, but does not remove stale controller registrations.

• Reviewed Kafka PR #22191, which proposes the missing controller-unregistration capability: a new Admin#unregisterController API, kafka-cluster.sh unregister-controller, and an UnregisterControllerRecord that removes a controller from the metadata image.

  However, the PR is not currently an in-place repair path for this cluster. It introduces the unregistration record behind new metadata version IBP_4_4_IV1. Our cluster is finalized at 3.7-IV4 and cannot even advance to 3.9-IV0, because the stale controller registrations 3, 4, and 5 advertise maximum metadata.version feature level 19.

  Therefore, upgrading Kafka binaries to a future release containing this PR would not automatically remove the stale registrations. The cluster would still be at metadata 3.7-IV4, and the new unregister-controller operation would be unavailable because the cluster has not finalized the metadata capability required to write its new record. Advancing metadata to that capability remains blocked by the same stale controller registrations—creating a circular dependency.

  We need a supported repair procedure that can remove obsolete controller registrations while the cluster remains at its existing finalized metadata version, or another supported migration path that breaks this dependency.

Request

Is there a supported Strimzi procedure for removing/pruning obsolete ControllerRegistration entries from a migrated static KRaft quorum when the old controller IDs are no longer voters or Pods?

Specifically, we need a safe way to remove controller registrations 3, 4, and 5, allowing the cluster to advance from metadata 3.7-IV4 to 3.9-IV0 , without recreating the cluster or manually modifying __cluster_metadata .

[ChetanKoneru]

Also commented on the PR here
apache/kafka#22191 (comment)

[scholzj]

Thsi has been discussed multiple times in the past. The only workaround is as far as I know to add the nodes again, upgrade the metadata version and then remove them. I do not think Kafka offers any other way.

[ChetanKoneru]

We have this now,

Kafka controller NodePool

metadata:
annotations:
strimzi.io/next-node-ids: "[90-99]"

Kafka broker NodePool

metadata:
annotations:
strimzi.io/next-node-ids: "[0-49]"

to re-add nodes, above needs to be modified and reverted. Will there be any unintended consequences?

[scholzj]

Well, depends on how you define unintended consequences. The node IDs 3, 4, and 5 do not seem to be used. So you can add them as controllers and remove them again. Of course the whole things sucks. And it will involve hacky rolling updates and so on. :-/

[ChetanKoneru]

If merged, will this PR help remediate this use case?
apache/kafka#22191 (comment)

Based on latest release document https://kafka.apache.org/43/getting-started/upgrade/, latest version of Kafka still supports metadata versions > 3.3.X.

I will try the work around you suggested, but If I am unsuccessful and since Kafka support's older metadata versions, I might just stay with metadataVersion: "3.7-IV4".

Thank you for your support @scholzj

[scholzj]

I do not know the PR. If it allows unregistration of controller nodes as its name suggests, it should. But I did not studied the PR, the JIRA, not any related KIPs. So they may be some gaps on the Kafka side etc. Also, please keep in mind that it is not only about merging -> you need a Kafka release, then some integration of it into Strimzi, etc.

[ChetanKoneru]

For Future travelers

Recovering a KRaft metadata-version upgrade blocked by stale controller IDs

Scope and safety boundary

This is a controlled recovery runbook :

Commands are hard-coded with below example values, please change accordingly -

• Example Context: bld-aks-cluster-eastus2 [Change accordingly]
• Example Namespace: bld-kafka [Change accordingly]
• Example Kafka resource: strimzi [Change accordingly]
• Kafka / Strimzi: Kafka 3.9.2, Strimzi 0.45.2
• Current metadata version: 3.7-IV4
• Target metadata version: 3.9-IV0

The historical controller IDs 3, 4, and 5, which were removed and moved to 90,91,92 , remain registered in KRaft metadata and advertise maximum feature level 19. The active controllers 90, 91, and 92 support level 21, but Kafka rejects the metadata upgrade while the old registrations remain.

The Strimzi workaround is to re-add IDs 3, 4, and 5, advance the metadata version after they re-register with the current Kafka version, then remove those temporary controllers again. Controller-pool scaling is a material KRaft quorum change. Run each phase only in an approved maintenance window, and do not proceed past a failed validation gate.

Desired steady state

# broker KafkaNodePool
metadata:
  annotations:
    strimzi.io/next-node-ids: "[0-49]"

# controller KafkaNodePool
metadata:
  annotations:
    strimzi.io/next-node-ids: "[90-99]"
spec:
  replicas: 3

# Kafka
spec:
  kafka:
    version: 3.9.2
    metadataVersion: "3.9-IV0"

Phase 0: preflight

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafka strimzi

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafkanodepool broker controller \
  -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas,ROLES:.spec.roles,NEXT:.metadata.annotations.strimzi\\.io/next-node-ids,REMOVE:.metadata.annotations.strimzi\\.io/remove-node-ids

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafka strimzi \
  -o jsonpath='metadata={.status.kafkaMetadataVersion}{"\\n"}registered={.status.registeredNodeIds}{"\\n"}conditions={.status.conditions}{"\\n"}'

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get pods -l strimzi.io/cluster=strimzi

⚠ Before Phase 1, the cluster must be Ready=True, metadata must be 3.7-IV4, the controller pool must have three replicas (90, 91, 92), and there must be capacity for three additional controller Pods.⚠

Keep GitOps aligned with every intermediate annotation and replica value. Otherwise the next reconciliation can revert an in-progress stage.

Phase 1: reserve IDs and add controllers 3, 4, and 5

# Reserve 3–5 for the temporary controllers. Do not leave them available
# to a broker scale-up during this procedure.
kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool broker \
  strimzi.io/next-node-ids='[0-2,6-49]' \
  --overwrite

# Select 3, 4, then 5 for the scale-up while retaining 90–99 as the
# normal future controller allocation range.
kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool controller \
  strimzi.io/next-node-ids='[3-5,90-99]' \
  --overwrite

# Add exactly three temporary controllers. Expected IDs: 3, 4, and 5.
kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  patch kafkanodepool controller --type=merge \
  -p '{"spec":{"replicas":6}}'

Wait until all three temporary Pods are Ready:

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  wait --for=condition=Ready pod/strimzi-controller-3 \
  pod/strimzi-controller-4 \
  pod/strimzi-controller-5 \
  --timeout=20m

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafka strimzi \
  -o jsonpath='metadata={.status.kafkaMetadataVersion}{"\\n"}registered={.status.registeredNodeIds}{"\\n"}conditions={.status.conditions}{"\\n"}'

Stop if controllers 3, 4, and 5 are not all Ready and registered, or if the existing quorum loses readiness.

Phase 2: finalize the metadata version

Update the GitOps Kafka manifest to:

spec:
  kafka:
    version: 3.9.2
    metadataVersion: "3.9-IV0"

If an imperative update is approved for the maintenance window, apply the same
change directly to the Kafka resource:

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  patch kafka strimzi --type=merge \
  -p '{"spec":{"kafka":{"metadataVersion":"3.9-IV0"}}}'

Ensure the GitOps manifest has the same value before its next sync. After the
change is applied, verify progress:

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafka strimzi --watch

Continue only after the resource reports:

status.kafkaMetadataVersion = 3.9-IV0
Ready = True

Confirm that no new feature-level rejection occurs:

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  logs strimzi-controller-90 -c kafka --since=15m \
  | grep -F 'Invalid update version' || true

Phase 3: remove temporary controllers and restore original values

# Choose only the temporary controller IDs for removal.
kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool controller \
  strimzi.io/remove-node-ids='[3-5]' \
  --overwrite

# Restore the original three-controller pool.
kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  patch kafkanodepool controller --type=merge \
  -p '{"spec":{"replicas":3}}'

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  wait --for=delete pod/strimzi-controller-3 \
  pod/strimzi-controller-4 \
  pod/strimzi-controller-5 \
  --timeout=20m

Only after IDs 3, 4, and 5 are gone, restore the original allocation policy:

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool controller \
  strimzi.io/next-node-ids='[90-99]' \
  --overwrite

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool controller \
  strimzi.io/remove-node-ids-

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  annotate kafkanodepool broker \
  strimzi.io/next-node-ids='[0-49]' \
  --overwrite

Final validation

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafka strimzi \
  -o jsonpath='metadata={.status.kafkaMetadataVersion}{"\\n"}registered={.status.registeredNodeIds}{"\\n"}conditions={.status.conditions}{"\\n"}'

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get kafkanodepool broker controller \
  -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas,NEXT:.metadata.annotations.strimzi\\.io/next-node-ids,REMOVE:.metadata.annotations.strimzi\\.io/remove-node-ids

kubectl --context bld-aks-cluster-eastus2 -n bld-kafka \
  get pods -l strimzi.io/cluster=strimzi

Expected final state:

metadata=3.9-IV0
Ready=True
controllers=90,91,92
broker next-node-ids=[0-49]
controller next-node-ids=[90-99]
controller remove-node-ids is absent