Downtime due to pod readinessProbe when rolling update Kafka pod

[manhnt98]

Bug Description

Context:

• I'm running Strimzi on AWS EKS.
• I'm using the rack awareness feature.
• I'm upgrading EKS version, so I need to move broker pods to new EKS nodes by rolling each pod at a time.

When I do rolling update for Kafka broker pod, one by one, by deleting each pod at a time. The broker container will run and register itself to Zookeeper before pod readinessProbe complete !
As I'm using rack awareness and the broker (I'm rolling the client's prefered broker) is registered to Zookeeper (registration done in less than 4s), but because pod readinessProbe is not completed yet (default 15s initialDelaySeconds), the client will try to change the connection back to the prefered broker and fail because the pod is not ready yet.

I assume that if the issue could be solved by having option to remove the readinessProbe.

Steps to reproduce

1. Configure rack awareness on brokers and clients
2. Continously consume from client
3. Rolling update (delete) the prefered broker
4. Watch the client logs, there should be timeout when broker pod is in [ 0/1 Running ] state

Expected behavior

No timeout connection when client change connection back to its prefered broker

Strimzi version

0.32.0

Kubernetes version

1.27

Installation method

Helm

Infrastructure

Amazon EKS

Configuration files and logs

No response

Additional context

No response

[scholzj]

You should probably start by sharing some logs and configurations and explaining why do you think this is an issue (or what exactly is the issue). Because I'm afraid it is not really clear.

[manhnt98]

To summary, I think there is issue that kafka broker process started before readinessProbe is completed.
For example:
If I delete benchmark-kafka-0, the pod will be recreated.

benchmark-kafka-0                            0/1     Running    0          5s
benchmark-kafka-1                            1/1     Running    0          29h
benchmark-kafka-2                            1/1     Running    0          6d5h

From the benchmark-kafka-0 pod's logs, in my understanding, Kafka broker registers to Zookeeper and starts running in about 4 seconds. This means the broker is advertised and clients can connect to it.

2024-04-18 15:04:56,457 INFO Starting KafkaAgent with brokerReadyFile=/var/opt/kafka/kafka-ready and sessionConnectedFile=/var/opt/kafka/zk-connected (io.strimzi.kafka.agent.KafkaAgent) [main]
...
2024-04-18 15:05:00,119 INFO Registered broker 0 at path /brokers/ids/0 with addresses: CONTROLPLANE-9090://benchmark-kafka-0.benchmark-kafka-brokers.benchmark.svc:9090,REPLICATION-9091://benchmark-kafka-0.benchmark-kafka-brokers.benchmark.svc:9091,INTERNAL-9092://<IP>:30001,EXTERNAL-9095://<domain>:30101, <...> (broker epoch): <...> (kafka.zk.KafkaZkClient) [main]
2024-04-18 15:05:00,177 INFO [ControllerEventThread controllerId=0] Starting (kafka.controller.ControllerEventManager$ControllerEventThread) [controller-event-thread]
...
2024-04-18 15:05:00,558 INFO [KafkaServer id=0] started (kafka.server.KafkaServer) [main]

But the readinessProbe is not completed yet (initialDelaySeconds is 15s), the pod status is still in (0/1).

benchmark-kafka-0                            0/1     Running    0          10s

If there are any client try to connect to this broker (because the broker is advertised), the connection will timeout since kube-proxy is not allow routing to it yet.

Please see if my understanding is missing and there are any configurations to solve this. Thank you.

[manhnt98]

Hi, is there any updates on this ? Just wanting to check in

[scholzj]

I do not think we have any plans to remove the readiness probe. As I said, you can finetune its parameters if you think your cluster is faster.

[manhnt98]

Tuning readinessProbe to its minimum with initialdelayseconds=0 and periodSeconds=1 still have a chance of 1s downtime.
I'm trying to bring my cluster to zero downtime, so could I request an option to disable readinessProbe ? If couldn't, are there reasons why is this readiness checks needed when Zookeeper could check for broker healthiness through heartbeat already ?

[scholzj]

No, you cannot, because the Kafka broker is not ready within 0 seconds. It is not ready even within 1 second. So, you do not have any downtime because of that.

[manhnt98]

Let me make it more clear. Let's say Kafka broker takes 4.5s to start. initialDelaySeconds=0 means k8s start healthcheck immediately after pod is scheduled and retry every 1 second (periodSeconds=1) until the pod passes. This means there will be 5 retry times that takes 5 seconds from initital check, 4 times is not enough because Kafka takes 4.5s to start, right ? So there is still 0.5s from when Kafka start til k8s done healthchecking, that 0.5s (could be more up to 0.9s) is the downtime I'm mentioning