Strimzi
Replies: 3 comments 18 replies
-
|
TBH, I don't really follow what are you doing and mainly why. Maybe you should describe it step by step including the YAMLs and the corresponding logs. But still ... why would you change the scheduling every time some chart version changes? The chart is not part of the Kafka deployment. Neither Strimzi nor Kafka care about the Helm chart and its version. So locking it into the deployment makes no sense to me. With things like Kafka, you want stability. That means not rolling them because some completely unrelevant changes. The log you shared from the operator suggests that the operator tries to roll the ZooKeeper pods (that is expected as you changed the labels and the topology spread). So what you need to do is to look why the ZooKeeper pod is not getting ready. Does it start? Is it scheduled? Or what exactly is causing it to not be ready. That is not clear from what you provided and clearly it is what blocks the operator.
I don't understand what this mean. What is
That is hard to comment on without seeing the actual YAMLs and the logs. Two more things I noticed:
|
Beta Was this translation helpful? Give feedback.
-
You can edit the Cluster Operator Config map and change the log level to DEBUG there. If the log does not cover reproducing the issue, it is not clear if it is needed. But it would definitely not hurt I guess. I think it is important to capture the rolling / deletion of the pod and it not being recreated. |
Beta Was this translation helpful? Give feedback.
-
|
Another user reported a similar problem on Slack. In his case, it seemed that the Pod informer died and that is why the operator didn't restart the pod. Not really clear what the cause was - it could be for example fabric8io/kubernetes-client#4781 but not sure TBH. |
Beta Was this translation helpful? Give feedback.
-
|
It seems that 0.35.0 does not fix the issue for us. We've updated yesterday on one of our dev environments. Today I manually killed one of our three kafka pods (kafka-1), but it was not recreated. Some log highlights (logs containing 'kafka-1') can be found below. The logs are combined from our 3 operator instances. If that's a problem then I can provide the individual pod logs as well. I've sent the complete log file through Slack since it is too big for github. |
Beta Was this translation helpful? Give feedback.
-
|
The operator logs: |
Beta Was this translation helpful? Give feedback.
-
|
Logs from an operator that was recently (within 2 hours) restarted. A pod was deleted and was recreated at around 8:45. This was on another environment, also running on 0.35 and AKS 1.26.3. |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj we are actually seeing this behavior significantly more since we stopped adding the |
Beta Was this translation helpful? Give feedback.
-
|
Maybe you can share your environment at #8528 -> maybe that way we figure out some common ground between the environments having this issue and find out what is causing it. You can possibly also enable the debug logging for the |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj how do we enable debug logging for io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener? |
Beta Was this translation helpful? Give feedback.
-
|
It seems to be already enabled in the logs attached above. |
Beta Was this translation helpful? Give feedback.
-
|
Issue #8528 with the 0.35.1 update has fixed the issue. Thanks for the help @scholzj |
Beta Was this translation helpful? Give feedback.
-
We've configured topologyspreadconstraints in an effort to keep our setup balanced/spread out over multiple nodes. Part of the constraint is a label that contains a version of our deployment which increments on each update.
What we expect to happen
Zookeeper, kafka and cruisecontrol should get the new version label and be scheduled across different nodes.
What happens
Zookeeper goes offline (the pods are killed), and no new pods are recreated. The strimzi operator keeps trying to reconcile the pods, but gives a timeout. This is only resolved by manually restarting the operator deployment so that the operator gets a new 'strimzi revision' which triggers a successful reconciliation round.
The zookeeper strimzipodset has a status that says 1 pod is healthy even when no pods are present.
Configuration
We wrap external charts in a chart of our own. We also cache all images in a registry of our own.
Our operator setup is pretty much the default setup:
Our zookeeper setup:
The chartVersion can change from e.g. '1.7.0-master-44644' to '1.7.0-master-44700'.
Logs
Operator:
Question
Is this expected behavior? What can we do to prevent the faulty state from happening on each update?
This happens with both 1 and 3 replicas of zookeeper. When using 3 replicas only 2 pods are killed and the other is stuck trying to find the missing instances.
Zookeeper logs:
Beta Was this translation helpful? Give feedback.
All reactions