Liveness and Readiness probe failed / failing.

[zawistowskipiotr]

After migration to KRaft mode:

  kafkaMetadataState: KRaft
  kafkaMetadataVersion: 3.7-IV4
  kafkaNodePools:
    - name: controller
    - name: kafka
  kafkaVersion: 3.7.0

strimzi-kafka-operator-0.40.0

I experience failing probes (liveness and readiness) on both controller and broker nodes.
Can't find anything helpful in logs, very few WARNs usually just after start of the node, then any logs even when status is reported as unhealthy due to failing probes.
I've tried extending timeout, which is currently at:

    livenessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 120
    readinessProbe:
      initialDelaySeconds: 60
      timeoutSeconds: 120

Cluster is running on 3 controller nodes and 5 broker nodes:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: kafka-cluster-001
spec:
  replicas: 5
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 3
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: azuredisk-kafka

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: kafka-cluster-001
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 3
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
        class: azuredisk-kafka

CPU & memory usage:

[scholzj]

I think you should start by configuring some reasonable resource and memory requests for all the nodes -> that way we will know for sure what amount of resources they have and if that can be an issue or not. I do not know what you use your Kafka cluster for, but for development and testing locally, I normally give it something like:

  resources:
    requests:
      memory: 2Gi
      cpu: 500m
    limits:
      memory: 2Gi
      cpu: "1"

which might be a good start.

[zawistowskipiotr]

Sure, that was done - forgot to mention in the post, as you can see at the screenshot I've provided from Grafana - pods are running under limit.

    resources:
      requests:
        memory: 5Gi
        cpu: "1"
      limits:
        memory: 8Gi
        cpu: "2"

[zawistowskipiotr]

Updated to strimzi-kafka-operator-0.41.0 - same issue.

[scholzj]

I guess you would need to provide some kind of reproducer as you seem to be the only having this kind of issue.

[zawistowskipiotr]

Problem disappeared and I didn't manage to reproduce it on different clusters.