[Bug]: Full Kafka upgrade (controllers + brokers) gets stuck with permanent UnforceableProblem in KRaft mode (Kafka 4.1.1 → 4.2.0)

[SSgarima]

Bug Description

🐞 Full Kafka upgrade (controllers + brokers) gets stuck with permanent UnforceableProblem in KRaft mode (Kafka 4.1.1 → 4.2.0)
Environment

Strimzi Operator: 1.0.0
Kafka: upgrading from 4.1.1 to 4.2.0
Upgrade type: Full cluster upgrade (controllers and brokers)
Mode: KRaft (no ZooKeeper)
Controllers: 3 (dedicated controllers)
Brokers: multiple (data plane not yet reached)
Kubernetes: EKS (AWS), CoreDNS
TLS: enabled for controller and inter-broker communication

Description
During a full Kafka upgrade (controllers and brokers) in KRaft mode, the upgrade consistently becomes stuck during the controller rolling phase, before broker rolling begins.
After upgrading one controller, the Strimzi operator repeatedly fails reconciliation and never progresses. The upgrade cannot continue without manual intervention.

Observed Logs
KafkaRoller$UnforceableProblem: Pod eba1-nweba1-controller-2 cannot be updated right now
ForceableProblem: Pod eba1-nweba1-controller-0 is the active controller and there are other pods to verify first
Could not verify pod ... giving up after 10 attempts

The operator retries indefinitely with increasing backoff and does not advance to the next controller or to broker rolling.

Upgrade State When Stuck

One controller upgraded to 4.2.0
Two controllers still on 4.1.1
Active controller cannot be restarted
One follower controller is permanently marked as unforceable
Broker rolling never starts
metadata.version remains at 4.1-IV0

Steps to reproduce

upgrade strimzi kafka 4.1.1 to 4.2.0

Expected behavior

All controllers shou;d upgrade and restart one by one

Strimzi version

4.1.1

Kubernetes version

1.35.2

Installation method

helm chart

Infrastructure

No response

Configuration files and logs

Additional context

No response

[scholzj]

Please share:

• The full configuration (the custom resources)
• The full logs from the operator and the Kafka nodes

Without that, it might not be possible to say what the issue is.

[ChetanKoneru]

Check your cruise control UI.
Kafka roller is not able to roll your broker 2. Check if there are any under replicated partitions or Sole ISR's in broker 2.
You will need to manage those topics before proceeding. Once you manage those topics you can trigger a manual-rolling-update or reconcile depending on your state.

Also was the cluster in ready state before you triggered the upgrade.
I usually also check kafkarebalance proposals before the upgrade, but you will not have any proposals mid upgrade now.