kafka consumer metrics missing in micrometer after SpringBoot upgrade

[dlipofsky]

I no longer have kafka consumer metrics in micrometer or prometheus after my SpringBoot upgrade.

It sounds similar to spring-attic/spring-cloud-stream-binder-kafka#914
but I tried what was mentioned in that ticket and did not have any luck solving my issue.

Here are my changes:

• Upgraded SpringBoot from 2.1.18 to 2.5.7.
• Upgraded Spring from 5.1.19 to 5.3.13.
• Upgraded Kafka from 2.8.0 to 2.8.1.
• Upgraded Micrometer from 1.6.9 to 1.7.6.
• Upgraded Springfox Swagger from 2.9.2 to 3.0.0.
• Added org.springframework.kafka:spring-kafka:2.7.9
• Added org.jolokia:jolokia-core:jar:1.6.2
• Removed spring-cloud since we're not loading config from a server
• Added spring.jmx.enabled:true to my application.yml

[wilkinsona]

We're not aware of any problems with Kafka metrics in Spring Boot 2.5.7. If you would like us to spend some time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

[dlipofsky]

I have created a very simple example. I can confirm this works

• SpringBoot 2.2.13.RELEASE
• kafka-clients 2.8.1
• micrometer 1.7.6
• spring.jmx.enabled=true
• spring-kafka and jolokia-core are not needed, it works without them. I tried adding them in a few combinations below because of hints on the other ticket that they may help, but they didn't.

But I tried all of these combinations and they do not work

• SpringBoot 2.3.0.RELEASE, micrometer 1.7.6, no jolokia, no spring-kafka
• SpringBoot 2.3.12.RELEASE, micrometer 1.7.6, no jolokia, no spring-kafka
• SpringBoot 2.4.13, micrometer 1.7.6, no jolokia, no spring-kafka
• SpringBoot 2.5.7, micrometer 1.7.6, no jolokia, no spring-kafka
• SpringBoot 2.5.7, micrometer 1.7.6, jolokia-core 1.6.2, spring-kafka 2.7.9
• SpringBoot 2.6.1, micrometer 1.8.1 (does not work with micrometer 1.7.6), no jolokia, no spring-kafka
• SpringBoot 2.6.1, micrometer 1.8.1, jolokia-core 1.7.1, spring-kafka 2.8.0

I tested all these by running the producer and confirming in the consumer logs that it had consumed messages (because it may be necessary to actually consume to create these metrics), then running

• curl http://localhost:8080/manage/prometheus
• curl http://localhost:8080/manage/metrics | jq

My sample code:

• example-consumer-2.5.7.zip
• example-consumer-2.2.13.zip
• example-producer.zip

[dlipofsky]

To clarify, I am manually creating my consumers and streams (not using spring-kafka)
and I can get metrics back by manually calling
new KafkaStreamsMetrics(stream).bindTo(registry) or
new KafkaClientMetrics(consumer).bindTo(registry)
but it would still be nice to have this happen automatically like it did in SpringBoot 2.2.x.
I've got a lot of apps to updates, and modifying each one at a time is a lot of work, and is how mistakes are made.

This is a regression, because it worked this way in SpringBoot 2.2.x but not in 2.3.x or beyond.

[ivan-zaitsev]

Since Spring Boot 2.3 Kafka metrics names have been changed.

kafka_consumer_records_consumed_total_records_total -> kafka_consumer_fetch_manager_records_consumed_total
kafka_consumer_records_lag_records -> kafka_consumer_fetch_manager_records_lag
kafka_consumer_records_lag_max_records -> kafka_consumer_fetch_manager_records_lag_max
kafka_consumer_fetch_latency_max_seconds -> kafka_consumer_fetch_manager_fetch_latency_max
kafka_consumer_bytes_consumed_total_bytes_total -> kafka_consumer_fetch_manager_bytes_consumed_total
kafka_consumer_records_per_request_avg_records -> kafka_consumer_fetch_manager_records_per_request_avg
kafka_consumer_heartbeat_rate_heartbeats -> kafka_consumer_coordinator_heartbeat_rate

If you have custom consumer factories, add customizers.orderedStream().forEach(customizer -> customizer.customize(consumerFactory));

@Bean
public ConsumerFactory<Integer, String> customConsumerFactory(ObjectProvider<DefaultKafkaConsumerFactoryCustomizer> customizers) {
    DefaultKafkaConsumerFactory<Integer, String> consumerFactory = new DefaultKafkaConsumerFactory<>(customConsumerConfigs());
    customizers.orderedStream().forEach(customizer -> customizer.customize(consumerFactory));
    return consumerFactory;
}

[dlipofsky]

@ivan909020 Those factories are spring-kafka and I'm not using spring-kafka, just new KafkaConsumer(...) or new KafkaStreams(...).

I noticed the name change, which is surprising. Why did the names change with SpringBoot? I would think it would be based on the version of kafka itself or the version of micrometer, but it's not. It's frustrating since I have a bunch of established dashboards and alerts in datadog.

[ivan-zaitsev]

@dlipofsky
Ok, then you can create KafkaConsumer through Spring - DefaultKafkaConsumerFactory.

@Autowired
private final PrometheusMeterRegistry meterRegistry;


DefaultKafkaConsumerFactory<String, String> consumerFactory = new DefaultKafkaConsumerFactory<>(props);
consumerFactory.addListener(new MicrometerConsumerListener<>(meterRegistry));

Consumer<String, String> consumer = consumerFactory.createConsumer();

[dlipofsky]

@ivan909020 , the point is I am trying to not rewrite all my apps to use spring-kafka, I would like to just be able to upgrade spring-boot (for security reasons) but I have this regression causing me pain. Also I have one app where the KafkaConsumer is created by a 3rd party SDK and therefore out of my control.

[ivan-zaitsev]

@dlipofsky
Ok, then you can create KafkaClientMetrics and bind to PrometheusMeterRegistry.

@Autowired
private final PrometheusMeterRegistry meterRegistry;


Consumer<String, String> consumer = new KafkaConsumer<>(props);

KafkaClientMetrics metrics = new KafkaClientMetrics(consumer);
metrics.bindTo(meterRegistry);

[dlipofsky]

Yes, that is what I am doing (see my comment above from yesterday) but it still involves changing all my apps. At this point I've done it, so that ship has sailed for me, but this is still a regression and I'd like to see if it can be addressed, since the old way was better. Maybe it can help others.

[ivan-zaitsev]

Yes, I see. I'm sorry I didn't notice. I suppose this is because prior to Spring Boot 2.3 Kafka metrics worked through JMX which implementation inside of KafkaConsumer and for this you did not need to configure anything, but since 2.3 it works through KafkaClientMetrics, and for this you need to configure it manually.

[snicoll]

@dlipofsky this change is covered in the release notes for 2.3 that's out of OSS support by then. Not relying on JMX is important as it was disabled by default in the previous version and that led to issue there as well.

Like you said the ship has sailed here but I can imagine that it would be fairly easy for you to retrofit that behavior in a custom arrangement of yours.