[Question] Exporter keep restarting because loss connection with broker

[sinbadonline]

When I try to publish metric onto Prometheus but with no luck. Check exporter found that it is keep restarting. Exporter's stdout are complaining that it lost connection with all brokers as below. Can anybody help me to provide some clue for resolving this ?

====exporter's stdout=======

[sarama] 2020/09/10 07:35:49 client/metadata fetching metadata for all topics from broker my-cluster-kafka-bootstrap:9091
[sarama] 2020/09/10 07:35:49 Connected to broker at my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #1)
[sarama] 2020/09/10 07:35:51 Closed connection to broker my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:51 Closed connection to broker my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:51 client/metadata fetching metadata for [__consumer_offsets] from broker my-cluster-kafka-bootstrap:9091
[sarama] 2020/09/10 07:35:51 Closed connection to broker my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:52 Connected to broker at my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #2)
[sarama] 2020/09/10 07:35:53 Connected to broker at my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #0)
[sarama] 2020/09/10 07:35:53 Closed connection to broker my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:53 Closed connection to broker my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:53 Connected to broker at my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #1)
[sarama] 2020/09/10 07:35:54 Closed connection to broker my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:54 client/metadata fetching metadata for all topics from broker my-cluster-kafka-bootstrap:9091
[sarama] 2020/09/10 07:35:54 Connected to broker at my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #2)
[sarama] 2020/09/10 07:35:55 Connected to broker at my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #0)
[sarama] 2020/09/10 07:35:56 Connected to broker at my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #1)
[sarama] 2020/09/10 07:35:58 Closed connection to broker my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:58 Closed connection to broker my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:58 Closed connection to broker my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:35:59 client/metadata fetching metadata for all topics from broker my-cluster-kafka-bootstrap:9091
[sarama] 2020/09/10 07:35:59 Connected to broker at my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #0)
[sarama] 2020/09/10 07:36:00 Connected to broker at my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #2)
[sarama] 2020/09/10 07:36:00 Connected to broker at my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #1)
[sarama] 2020/09/10 07:36:02 Closed connection to broker my-cluster-kafka-2.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:36:02 Closed connection to broker my-cluster-kafka-1.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:36:03 Closed connection to broker my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091
[sarama] 2020/09/10 07:36:03 client/metadata fetching metadata for [__consumer_offsets] from broker my-cluster-kafka-bootstrap:9091
[sarama] 2020/09/10 07:36:03 Connected to broker at my-cluster-kafka-0.my-cluster-kafka-brokers.kafka.svc:9091 (registered as #0)

====My Kafka/Exporter setting as below

> apiVersion: v1
> items:
> - apiVersion: kafka.strimzi.io/v1beta1
>   kind: Kafka
>   metadata:
>     creationTimestamp: "2020-09-08T03:18:12Z"
>     generation: 7
>     name: my-cluster
>     namespace: kafka
>     resourceVersion: "15809922"
>     selfLink: /apis/kafka.strimzi.io/v1beta1/namespaces/kafka/kafkas/my-cluster
>     uid: 7efd0f82-41a1-4049-9b95-0b59e7e4c119
>   spec:
>     entityOperator:
>       topicOperator: {}
>       userOperator: {}
>     jmxTrans:
>       kafkaQueries: []
>       outputDefinitions: []
>     kafka:
>       authorization:
>         type: simple
>       config:
>         log.message.format.version: "2.5"
>         offsets.topic.replication.factor: 3
>         transaction.state.log.min.isr: 2
>         transaction.state.log.replication.factor: 3
>       jmxOptions: {}
>       listeners:
>         plain: {}
>         tls: {}
>       replicas: 3
>       storage:
>         type: ephemeral
>       version: 2.5.0
>     kafkaExporter:
>       enableSaramaLogging: true
>       groupRegex: .*
>       topicRegex: .*
>     zookeeper:
>       replicas: 3
>       storage:
>         type: ephemeral

Strimzi version: 0.18.0
Installation method: Operator
Kubernetes cluster: Kubernetes 1.16
Infrastructure: Bare metal

[kanak-kabara]

+1

I am seeing the same issue with a similar configuration:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
      delete.topic.enable: true
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    jvmOptions:
      javaSystemProperties:
        - name: zookeeper.ssl.hostnameVerification
          value: "false"
        - name: zookeeper.ssl.quorum.hostnameVerification
          value: "false"
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    logging: debug
    enableSaramaLogging: true

[scholzj]

Is this during rolling update of the Kafka cluster or something like that? Or while the cluster is stable and running all the time?

[kanak-kabara]

@scholzj When the cluster is stable and running all the time for me.

[sinbadonline]

@scholzj When the cluster is stable and running all the time for me too. Sending and receive msg is OK too.

[scholzj]

Yeah, I think the Exporter (and mainly the Sarama library it uses) have some issues in handling the restarts. There are some bug issues for it already. We will need to look into it. I think #2480 is the bug for tracking it.

[sinbadonline]

Probably because too limited CPU resource causing Kafka pod working un-normally.

[aholbreich]

Any changes? See same issue