Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Prometheus Exemplars #2672

Closed
2 tasks done
jonatan-ivanov opened this issue Jun 29, 2021 · 9 comments · Fixed by #3091
Closed
2 tasks done

Support for Prometheus Exemplars #2672

jonatan-ivanov opened this issue Jun 29, 2021 · 9 comments · Fixed by #3091
Labels
registry: prometheus A Prometheus Registry related issue

Comments

@jonatan-ivanov
Copy link
Member

jonatan-ivanov commented Jun 29, 2021

It seems more and more people are interested in using Exemplars with Micrometer and Prometheus (I'm not sure if other backends support it), see:

Exemplars are metadata attached to metrics (see: Prometheus Exemplars), e.g.: traceId so that you can get an example traceId for your metrics.

This issue is for starting a discussion about this feature and checking the popularity of this possible feature.

Prometheus has Exemplar support for Counter and Histogram, so the subtasks for this issue are:

@jonatan-ivanov jonatan-ivanov added waiting for team An issue we need members of the team to review registry: prometheus A Prometheus Registry related issue module: micrometer-core An issue that is related to our core module labels Jun 29, 2021
@kalali
Copy link

kalali commented Jul 2, 2021

We are looking to have Exemplars enabled in our environments. For us monitoring backend is Prometheus and the services are mainly developed using spring boot (taking advantage of spring cloud sleuth and actuator).

@Matthevv-k4y
Copy link

I asked that question on SO; I was thinking there might be a way to use some customisation/configuration class to use for this, but it looks like it is more involved than what I thought. Thank you for filing the ticket.

@jonatan-ivanov
Copy link
Member Author

jonatan-ivanov commented Jul 5, 2021

@Matthevv-k4y I haven't looked into this but you might have a chance to implement this in a simple way. Take a look at DefaultExemplarSampler, as you can see, it gets the data from a SpanContextSupplier that you should be able to implement easily if you use Sleuth or Brave: just inject a Tracer to your implementation and call the getters.
I'm not sure though that through Micrometer Prometheus will use the sampler.

@HaloFour
Copy link

HaloFour commented Jul 6, 2021

I have a similar need for something like Prometheus exemplars.

We're using a custom backend which we've wrapped in Micrometer, as well as Prometheus. We would like to associate several key/value pairs with metrics as they are recorded but that aren't treated as tags, such as account IDs. We want to capture these key/value pairs for metrics recorded by Spring as well, such as http.server.requests and http.client.requests where we're already using Spring components to contribute additional tags for each metric.

@roadSurfer
Copy link

Thanks @jonatan-ivanov, being able to jump from a metric of interest to a trace would be a huge boon. I only have passing familiarity with Sleuth (and, indeed, Micrometer), but I will at least take a look and see how much work is involved.

@jjneely
Copy link

jjneely commented Jan 10, 2022

Sleuth 3.1.0 was announced not long ago with Prometheus Exemplar support, provided you are using the Prometheus client_java library. So it looks like support from Micrometer for Exemplars is just around the corner.

I am mostly interested in working with OpenTelemetry here, and the Prometheus client_java library supports adding OpenTelemetry exemplars (span_id and trace_id) with no code changes. You load up the Java Agent and it just works. Why doesn't this auto-instrumentation feature also work with Micrometer?

@jonatan-ivanov
Copy link
Member Author

jonatan-ivanov commented Jan 11, 2022

Hi @jjneely,

Yepp, it is around the corner, we are planning to add it in the next minor release of Micrometer (1.9.0).

Why don't we have it right now?
The exemplar support is quite new in the field: as far as I know (this might have changed) Prometheus is the only backend that supports exemplars and this feature is experimental. OTel Metrics is experimental too so as far as I know, there is no production-ready exemplar support exists as of today (I might be mistaken, please correct me if I'm wrong).

Our plan is to add Exemplar support to Micrometer in 1.9; as you pointed out, Sleuth already has it so after the feature is added to Micrometer the plan is to offer out-of-the-box Exemplar support with Micrometer and Spring Cloud Sleuth (with Prometheus as the backend).

@fscellos
Copy link

Hello. I push a PR (#3019) that try to answer support for counter and histogram.
I also integrate code from your PR about counter but i took account of client sampling too (ie. not sample exemplars that won't be recorded; ie TraceFlags set to "00")

@jonatan-ivanov jonatan-ivanov changed the title Support for Exemplars Support for Prometheus Exemplars Mar 22, 2022
@jonatan-ivanov jonatan-ivanov removed this from the 1.9 backlog milestone Mar 22, 2022
@jonatan-ivanov jonatan-ivanov linked a pull request Mar 22, 2022 that will close this issue
@shakuzen shakuzen removed waiting for team An issue we need members of the team to review module: micrometer-core An issue that is related to our core module labels Mar 25, 2022
@shakuzen
Copy link
Member

shakuzen commented Mar 25, 2022

Closing this tracking issue now that the two sub-tasks have been completed.
Exemplars support in the PrometheusMeterRegistry is now available in snapshots for 1.9 (1.9.0-SNAPSHOT) and should be part of the upcoming 1.9.0-RC1 pre-release. We would appreciate any feedback from folks that are able to try this feature in snapshots.
There is a good explanation of how to use this feature in this description. We'll follow-up with adding documentation to the official docs. Let us know if there are any questions or feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
registry: prometheus A Prometheus Registry related issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants