Configuration in Zookeeper (/config/brokers/{id}) isn't refreshed after changes made to KafkaCluster.spec.listenersConfig.externalListeners

[play-io]

Hello.

I have recently had problems with a cluster - it stopped working. I guess it was caused by invalid configuration.
With fixing configuration I also changed KafkaCluster.spec.listenersConfig.externalListeners from
listenersConfig:
externalListeners:
- ...
containerPort: 9094
To
listenersConfig:
externalListeners:
- ...
containerPort: 9095
And number of brokers to 2.

After the change my cluster was in Running state, but I could not access it:

• envoy had correct (updated) configuration in the configmap
• brokers exposed correct (updated) container port
• BUT zookeeper. get /config/brokers/0 had advertised.listeners still pointing to port 9094
• BUT zookeeper. get /config/brokers/1 did not exist

Did it happen to you? Do you know maybe how to force refresh? In my case I replaced /config/brokers/0 with empty value to make the cluster operable.

[panyuenlau]

@play-io Can you provide detailed steps for reproducing this issue?

Based on the given information, I can only tell that the new broker never gets registered to ZooKeeper, but can't tell what causes this without seeing the detailed steps and some logs

[bartam1]

Dear @play-io !
Are you sure about you wanted to change the containerPort for the external listener?
Kafka listens on the specified containerPort in the pod but it is not the same as the exposed port which is available outside of the Kubernetes cluster through envoy. You have to configure the externalStartingPort to modify the advertised.listener port number. Currently the exposed port will be the externalStartingPort + brokerID.
More details:

koperator/config/samples/banzaicloud_v1beta1_kafkacluster.yaml

Line 279 in 08b2a9f

externalStartingPort: 19090

and here: https://smm-docs.eticloud.io/sdm/koperator/external-listener/

[play-io]

Hi @panyuenlau

but can't tell what causes this without seeing the detailed steps and some logs

Makes sense. Steps to reproduce:

1. Put incorrect (config options start with -) configuraiton into brokers template config
   brokerConfigGroups:
   default:
   config: |-
   • "auto.create.topics.enable=false"
   • "cruise.control.metrics.topic.auto.create=true"
   • "cruise.control.metrics.topic.num.partitions=1"
   • "cruise.control.metrics.topic.replication.factor=1"
2. Assign the template to all brokers
   brokers:

• id: 0
  brokerConfigGroup: "default"
  ...

3. Apply the new config to a cluster
4. If after the new config is applied and brokers accessible then make a change to containerPort in externalListeners configuration.
   Given you have
   externalStartingPort: 19090
   containerPort: 9095
   Then change to
   externalStartingPort: 19090
   containerPort: 9096

Observations:

• Incorrect configuration somehow hasn't broken cluster from the beginning
• Just after making changes in externalListener ports.

Troubleshooting:
After the change my cluster was in Running state, but I could not access it:

envoy had correct (updated) configuration in the configmap
brokers exposed correct (updated) container port
BUT zookeeper. get /config/brokers/0 had advertised.listeners still pointing to port 9094
BUT zookeeper. get /config/brokers/1 did not exist

[play-io]

H @bartam1

Are you sure about you wanted to change the containerPort for the external listener?

Yes an I'm aware about its sense. Thanks for your comment