Non-monotonic kafka metric reported as counter

[jack-berg]

Describe the bug
KafkaMetrics converts the metric kafka.consumer.connection.count to a counter despite the number of connections not being monotonically increasing.

This is causing issues in in the opentelemetry micrometer shim, which relies counters being monotonically increasing for a proper translation to the opentelemetry data model.

Environment

• Micrometer version: latest
• Micrometer registry: opentelemetry micrometer shim
• OS: any
• Java version: any

To Reproduce
Run the KafkaClientMetricsIntegrationTest and observe that the printed meters should kafka.consumer.connection.count is a counter where it should probably be a gauge.

Expected behavior
kafka.consumer.connection.count and other non-monotonically increasing kafka metrics should probably be gauges instead of counters.

Originally discussed in CNCF slack here.

[shakuzen]

Thank you for reporting the issue.
It looks like it would be correct for us to remove the OR conditional checking for names ending in count when deciding whether to make a counter or gauge since all the names ending in count appear to be an active count (gauge).

micrometer/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka/KafkaMetrics.java

Line 252 in b617100

if (meterName.endsWith("total") || meterName.endsWith("count")) {

[jack-berg]

It looks like it would be correct for us to remove the OR conditional checking for names ending in count when deciding whether to make a counter or gauge

That does appear to be the case. I implemented something similar in opentelemetry recently and inspected the type of the KafkaMetric's org.apache.kafka.common.metrics.Measurable, and defined a mapping from measurable implementation to corresponding instrument type. This seemed safer to me than relying on naming naming conventions.

[shakuzen]

That does sound like a safer way to do it. Thank you for sharing.