Support for Prometheus Exemplars

[jonatan-ivanov]

It seems more and more people are interested in using Exemplars with Micrometer and Prometheus (I'm not sure if other backends support it), see:

• How to add timestamp into metric data? #2627
• SO: How to get spring boot cloud/actuator to support prometheus Exemplars?

Exemplars are metadata attached to metrics (see: Prometheus Exemplars), e.g.: traceId so that you can get an example traceId for your metrics.

This issue is for starting a discussion about this feature and checking the popularity of this possible feature.

Prometheus has Exemplar support for Counter and Histogram, so the subtasks for this issue are:

• Exemplars support for Prometheus Counter #2811
• Exemplars support for Prometheus Histogram #2812

[kalali]

We are looking to have Exemplars enabled in our environments. For us monitoring backend is Prometheus and the services are mainly developed using spring boot (taking advantage of spring cloud sleuth and actuator).

[Matthevv-k4y]

I asked that question on SO; I was thinking there might be a way to use some customisation/configuration class to use for this, but it looks like it is more involved than what I thought. Thank you for filing the ticket.

[jonatan-ivanov]

@Matthevv-k4y I haven't looked into this but you might have a chance to implement this in a simple way. Take a look at DefaultExemplarSampler, as you can see, it gets the data from a SpanContextSupplier that you should be able to implement easily if you use Sleuth or Brave: just inject a Tracer to your implementation and call the getters.
I'm not sure though that through Micrometer Prometheus will use the sampler.

[HaloFour]

I have a similar need for something like Prometheus exemplars.

We're using a custom backend which we've wrapped in Micrometer, as well as Prometheus. We would like to associate several key/value pairs with metrics as they are recorded but that aren't treated as tags, such as account IDs. We want to capture these key/value pairs for metrics recorded by Spring as well, such as http.server.requests and http.client.requests where we're already using Spring components to contribute additional tags for each metric.

[roadSurfer]

Thanks @jonatan-ivanov, being able to jump from a metric of interest to a trace would be a huge boon. I only have passing familiarity with Sleuth (and, indeed, Micrometer), but I will at least take a look and see how much work is involved.

[jjneely]

Sleuth 3.1.0 was announced not long ago with Prometheus Exemplar support, provided you are using the Prometheus client_java library. So it looks like support from Micrometer for Exemplars is just around the corner.

I am mostly interested in working with OpenTelemetry here, and the Prometheus client_java library supports adding OpenTelemetry exemplars (span_id and trace_id) with no code changes. You load up the Java Agent and it just works. Why doesn't this auto-instrumentation feature also work with Micrometer?

[jonatan-ivanov]

Hi @jjneely,

Yepp, it is around the corner, we are planning to add it in the next minor release of Micrometer (1.9.0).

Why don't we have it right now?
The exemplar support is quite new in the field: as far as I know (this might have changed) Prometheus is the only backend that supports exemplars and this feature is experimental. OTel Metrics is experimental too so as far as I know, there is no production-ready exemplar support exists as of today (I might be mistaken, please correct me if I'm wrong).

Our plan is to add Exemplar support to Micrometer in 1.9; as you pointed out, Sleuth already has it so after the feature is added to Micrometer the plan is to offer out-of-the-box Exemplar support with Micrometer and Spring Cloud Sleuth (with Prometheus as the backend).

[fscellos]

Hello. I push a PR (#3019) that try to answer support for counter and histogram.
I also integrate code from your PR about counter but i took account of client sampling too (ie. not sample exemplars that won't be recorded; ie TraceFlags set to "00")