Seeing errors in kafka-exporter

[saloni-2404]

Hey Jakub,
Observed that one of the kafka and zookeeper pods got restarted, following which the kafka-exporter logs showed errors such as:
time="2021-03-11T06:39:08Z" level=error msg="Cannot get offset of group ultra-annotations-consumer-group: kafka: broker not connected" source="kafka_exporter.go:384"
time="2021-03-11T08:47:28Z" level=error msg="Cannot get consumer group: kafka: broker not connected" source="kafka_exporter.go:361"
time="2021-03-11T11:26:28Z" level=error msg="Cannot get offset of group ultra-annotations-consumer-group: kafka: broker not connected" source="kafka_exporter.go:384"
time="2021-03-11T11:26:28Z" level=error msg="Cannot get offset of group ultra-annotations-consumer-group: kafka: broker not connected" source="kafka_exporter.go:384"
time="2021-03-11T17:48:58Z" level=error msg="Cannot get describe groups: kafka: broker not connected" source="kafka_exporter.go:373"
time="2021-03-11T17:48:58Z" level=error msg="Cannot get describe groups: kafka: broker not connected" source="kafka_exporter.go:373"
time="2021-03-11T17:49:58Z" level=error msg="Cannot get offset of group ultra-annotations-consumer-group: kafka: broker not connected" source="kafka_exporter.go:384"
time="2021-03-11T18:09:58Z" level=error msg="Cannot get describe groups: kafka: broker not connected" source="kafka_exporter.go:373"

---

Attaching more logs.

Operator pod kept showing these logs repeatedly:
2021-03-11 18:24:00 WARN AbstractOperator:106 - Kafka resource kafka-cluster in namespace sedstarhseblocverify-gr-ultra: Contains object at path spec.zookeeper with an unknown property: jmxOptions
2021-03-11 18:24:00 INFO AbstractOperator:122 - Reconciliation #57326(timer) Kafka(sedstarhseblocverify-gr-ultra/kafka-cluster): Kafka kafka-cluster should be created or updated
2021-03-11T18:24:00.926+0000: 6856562.985: [GC (Allocation Failure) [PSYoungGen: 22188K->416K(21504K)] 814393K->792869K(828416K), 0.0186608 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2021-03-11T18:24:02.123+0000: 6856564.182: [GC (Allocation Failure) [PSYoungGen: 21408K->688K(21504K)] 813861K->793472K(828416K), 0.0126113 secs] [Times: user=0.03 sys=0.01, real=0.01 secs]
2021-03-11 18:24:02 INFO AbstractOperator:262 - Reconciliation #57326(timer) Kafka(sedstarhseblocverify-gr-ultra/kafka-cluster): reconciled
2021-03-11 18:26:00 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace sedstarhseblocverify-gr-ultra...

[saloni-2404]

kafka-2-logs.txt
zk-0-logs.txt
zk-1-logs.txt
zk-2-logs.txt
kafka-0-logs.txt
kafka-1-logs.txt

[scholzj]

I'm afraid I do not know anything more than the error suggests. This IIRC sometimes happens for example during rolling updates etc. As for the operator log, it says what it says:

2021-03-11 18:24:00 WARN AbstractOperator:106 - Kafka resource kafka-cluster in namespace sedstarhseblocverify-gr-ultra: Contains object at path spec.zookeeper with an unknown property: jmxOptions

JMX is not supported on Zookeeper. But I do not think this is in anyway related.

[saloni-2404]

I meant that, the errors/warning we are seeing in kafka-exporter(keeps restarting) logs, is similar to these bugs that have been reported previously:
#2480 #2842
#3146
#3624

[saloni-2404]

@scholzj
Hey Jakub,
So i was looking for a way to exclude certain topic, say "__consumer_offset"
i found this link- danielqsj/kafka_exporter#103

would this be available for kafka 17?
I'm looking to include these in the yaml, was putting under spec.kafka.config (should i mention elsewhere?)

topic.exclude: "__consumer_offset"
max.poll.interval.ms: 300000

[scholzj]

Sorry, not sure what Kafka 17 is. But the issue you linked to is opened. so I do not think it is addressed anywhere.

PS: @alesj is working on improving the Kafka Exporter. So if you want, you can discuss your problems with him and help him test what he got.

[saloni-2404]

Sure, thanks, Jakub. By kafka 17 i meant version 0.17.0 of kafka-operator.
@alesj

1. Is there any way that I could exclude a particular topic, say "__consumer_offset"
2. If i wish to increase the poll interval (the intention was to have more gap in between rather than the default of some seconds), is this the correct way to do that-
   spec.kafka.config:
   max.poll.interval.ms: 300000

Thank you for your response :)

[alesj]

1. topic.filter is the regexp parameter which is used to match topics
2. afais, the only thing you can set is: "refresh.metadata", "Metadata refresh interval").Default("1m").StringVar(&opts.metadataRefreshInterval) -- that's on the exporter side ... dunno how this fits/works with Prometheus

My branch is here:

• https://github.com/alesj/kafka_exporter/tree/fork1