Select individual metrics

[NeQuissimus]

I am looking to use this to build a Kubernetes HPA based on consumer lag.

In order for this to be much easier, it would be nice if I could select a single specific metric.

If something like http://localhost:8000/kafka_consumergroup_group_lag_seconds?cluster_name=dev&topic=high-volume-topic&group=consumer-group returned just those metrics I am querying for, I could easily grab the max value and use that to have Kubernetes decide whether I need more pods. Similarly, I could spin down pods if that number is low.

[seglo]

Thanks for your interest @NeQuissimus. That's the exact sort of use case I was hoping to enable!

Kafka Lag Exporter was meant to work within the Prometheus ecosystem. When its endpoint is scraped by a Prometheus server you can then query for the exact metrics you need with PromQL (prometheus query language). This project exposes its endpoint with the Prometheus simple Java HTTP client library, which provides a simplistic way of publishing the metrics endpoint and AFAICT doesn't offer any sort of filtering criteria.

I understand that you may not be interested in setting up a full Prometheus stack so I'm open to discussing other solutions (kafka-lag-exporter as a library? a separate queryable API?), but if you're still in the prototype phase of your project the easiest path forward would be to run Prometheus server, configure it to scrape or discover Kafka Lag Exporter's endpoint, and query that.

[NeQuissimus]

I am planning to have my metrics delivered to DataDog, I am currently not really looking to change that.
Just trying to figure out the simplest possible solution to get the HPA experiment going :)

I found something like this may work :D

curl -s http://localhost:8000/ | grep 'kafka_consumergroup_group_lag\{' | grep 'group="mygroup"' | awk '{print $2}' | grep '^[0-9]*.[0-9]*$' | sort -n | tail -1

Maybe I'll wrap this in something, we'll see :)

This is not super high on my list right now but I will definitely come back to it.
Funny thing is I actually had an (internal) JIRA ticket to build something like Kafka Lag Exporter and now you've already done a good chunk of the work for me, so thanks :D

[seglo]

Got it. That should do the trick for a quick experiment! I don't have much experience with datadog, but it looks like the agent supports prometheus endpoints too, so that's another option.

Funny thing is I actually had an (internal) JIRA ticket to build something like Kafka Lag Exporter and now you've already done a good chunk of the work for me, so thanks :D

You're welcome! It's still early days so if you have anything you'd like to see in the backlog LMK. :)

[seglo]

@NeQuissimus I was poking around the Prometheus HTTP simple client and I noticed it does have a filtering mechanism (though I couldn't find any GitHub issue or documentation around it). You can filter by some arbitrary metric name with the query param name[]=metric_name.

Ex)

http://localhost:8080/?name[]=kafka_consumergroup_group_max_lag

EDIT: The parsing code for reference: https://github.com/prometheus/client_java/blob/4e0e7527b048f1ffd0382dcb74c0b9dab23b4d9f/simpleclient_httpserver/src/main/java/io/prometheus/client/exporter/HTTPServer.java#L101

[NeQuissimus]

That is both weird and awesome :-) good find!

[seglo]

Documented in #29