[Bug]: strimzi-cluster-operator keeps restarting with issue: WARN BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for

[vietnguyenmipro]

Bug Description

strimzi-cluster-operator keeps restarting with issue: WARN BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for xxxx ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked

Steps to reproduce

1. Install strimzi-kafka-operator as dependency in a application helm chart:

dependencies:
  - name: strimzi-kafka-operator
    version: "0.40.0"
    repository: "oci://quay.io/strimzi-helm"

3. In helm templates folders: create new cluster with this definition:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: myapp
spec:
  kafka:
    version: 3.7.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.7"
    storage:
      type: persistent-claim
      class: {{ .Values.kafka.cluster.kafka.storage.class }}
      size: {{ .Values.kafka.cluster.kafka.storage.size }}
      deleteClaim: true
  zookeeper:
    livenessProbe:
      initialDelaySeconds: 120
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 120
      timeoutSeconds: 5
    replicas: {{ .Values.kafka.cluster.zookeeper.replicas }}
    storage:
      type: persistent-claim
      class: {{ .Values.kafka.cluster.zookeeper.storage.class }}
      size: {{ .Values.kafka.cluster.zookeeper.storage.size }}
      deleteClaim: true
  entityOperator:
    template:
      topicOperatorContainer:
        env:
        - name: STRIMZI_USE_ZOOKEEPER_TOPIC_STORE
          value: "true"
    topicOperator: {}
    userOperator: {}

4. Install the app with its dependency: helm install
5. Strimzi-cluster-operator keeps retartings with log attached (I was trying my best to format it but it was so ugly), please find the attachment.

2024-04-29` 19:19:36 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 128580 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked
        at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]

2024-04-29 19:19:37 ERROR AbstractOperator:284 - Reconciliation #1(watch) Kafka(forme/myapp): createOrUpdate failed io.fabric8.kubernetes.client.KubernetesClientException: Operation: [patch]  for kind: [Deployment]  with name: [myapp-entity-operator]  in namespace: [forme]  failed.

`

Expected behavior

The strimzi-cluster-operator works without above issue.

Strimzi version

0.40.0

Kubernetes version

1,29

Installation method

helm

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Please make sure to:

• Properly format all code, YAMLs or logs to make them readable
• Make sure you share the full log of the Cluster Operator from the beginning

[vietnguyenmipro]

Oh, i just formatted it right before I got your reply, then, my formatting gone :(. I will do it again now. Thank you for letting me know.

[vietnguyenmipro]

Tried to find solutions in all discussions. I tried to delete CRD, delete the whole namespace, delete Clusterrole and ClusterRoleBinding (grep strimzi, kafka), increase resource (CPU and memory). No thing helps :(.

Any suggestions are more than helpful... Could you please give an advice @scholzj thank you so much in advance.

[scholzj]

I cannot give you any advice because you:

• Did not share any proper log from the Cluster Operator
• Your custom resource contains some templating -> you should do kubectl get kafka ... -o yaml and share what you get that way, not share some templaces

[scholzj]

The log does not seem to really correspond to the custom resource. The custom resource suggests the cluster deployment got stuck because of talking with your Kubernetes API sever:

  - lastTransitionTime: "2024-04-29T20:37:00.397227213Z"
    message: 'Operation: [patch]  for kind: [Deployment]  with name: [tms-entity-operator]  in
      namespace: [forme]  failed.'
    reason: KubernetesClientException
    status: "True"
    type: NotReady

But the operator log seems to just show that the operator has some blocked threads. That can happen for different reasons. It can be for example insufficient resources such as CPU (did you configure resource request for the Cluster Operator?). I guess it could also be that it is stuck waiting for some response from your Kubernetes API server. But it does not seem to have any failed Kubernetes requests, just some taking a very long time.

[scholzj]

Various Admission controllers also tend to cause weird errors like this. So that and the logs of your Kubernetes API server might be something you can look into as well.

[vietnguyenmipro]

Thank you so much. I figured out that jaeger is trying to call webhook from the logs of Kube API server. So, after fixing jaeger, the issue gone away. :)

[scholzj]

Great, glad it helped.