KAFKA-13407: Always start controller when broker wins election #11476
Conversation
|
Is this also an issue on trunk? If so let's fix trunk first then we can backport it to older branches. |
|
Hi @mimaison ! As for doing the fix in trunk, we attempted to reproduce it using the ducktape on docker Kafka system tests any suggestions are welcome. |
edoardocomar
force-pushed
the
KAFKA-13407
branch
from
96e6680
to
3fb0870
Compare
Add a call to `onControllerFailover` into code path where `elect` is called, and the broker discovers it has already been elected. We found that by restarting the ZK leader we could occasionally trigger this code path, and prior to this change it would not start a controller failover. This left our Kafka cluster in a state where the `/controller` znode existed, and named the broker that had "won" the controller election, but in terms of runtime state: all the brokers had resigned from being the controller. Without a running controller, restarting brokers would typically cause partitions to become under-replicated as the restarted brokers never received the UpdateMetadata or LeaderAndISR requests required to correctly lead / follow any of their replicas. Also add some info level logging and more descriptive log messages for the log lines that were helpful in tracking the controller failover. Co-authored-by: Tina Selenge <tina.selenge@gmail.com> Co-authored-by: Adrian Preston <prestona@uk.ibm.com> Co-authored-by: Edoardo Comar <ecomar@euk.ibm.com.com>
edoardocomar
force-pushed
the
KAFKA-13407
branch
from
3fb0870
to
3d7406d
Compare
|
I was seeing this issue in 3.0 in our on-prem 3 nodes K8s cluster Thank You!! |
|
@edoardocomar : Thanks for the PR. As I mentioned in the jira, I believe this issue is fixed by https://issues.apache.org/jira/browse/KAFKA-13461 |
|
This has been fixed in #11563, closing |
Add a call to
onControllerFailoverinto code path whereelectiscalled, and the broker discovers it has already been elected. We
found that by restarting the ZK leader we could occasionally trigger
this code path, and prior to this change it would not start a
controller failover. This left our Kafka cluster in a state where the
/controllerznode existed, and named the broker that had "won" thecontroller election, but in terms of runtime state: all the brokers
had resigned from being the controller. Without a running controller,
restarting brokers would typically cause partitions to become
under-replicated as the restarted brokers never received the
UpdateMetadata or LeaderAndISR requests required to correctly lead /
follow any of their replicas.
Also add some info level logging and more descriptive log messages for
the log lines that were helpful in tracking the controller failover.
proposed fix for https://issues.apache.org/jira/browse/KAFKA-13407
Co-authored-by: Tina Selenge tina.selenge@gmail.com
Co-authored-by: Adrian Preston prestona@uk.ibm.com
Co-authored-by: Edoardo Comar ecomar@euk.ibm.com.com
Committer Checklist (excluded from commit message)