-
Notifications
You must be signed in to change notification settings - Fork 961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Includes exemplars for count and sum aggregate metrics for histograms #3460
Includes exemplars for count and sum aggregate metrics for histograms #3460
Conversation
|
There is a discussion about this on Micrometer Slack, let me copy here some interesting bits, in separate comments. This is missing right now because OpenMetrics does not support Exemplars for Summary, I’m not sure why (counter and sum). And it seems Histogram only supports it on the buckets. Other than this, I’m with you, I’m also missing this, e.g.: what if I have a
as it is defined in the specs. |
I’m not sure that this implementation will work, it seems it will not give you the latest exemplar but it will give you the exemplar to the highest bucket.
do this:
After you fix the buckets, I think the test will be still broken because the counter will get the exemplar to the first recording because it is the highest. This behavior can result in the counter exemplar being initialized and never updated again. Maybe it’s just me but I think the behavior matters (assuming that OpenMetrics will support exemplars there) because of two reasons:
|
@HaloFour asked questions to get feedback on the exemplars support of the OpenMetrics specs on CNCF slack: https://cloud-native.slack.com/archives/C01NP3BV26R/p1665087946929039 |
Sounds good, we can revisit once we get clarity about the OpenMetrics spec. If we would require a "better" approach to determine which observation to use would that require waiting on a solution from the Prometheus client? |
I think this depends on what feedback we will get back and how we want to implement it. I think it is likely that we don't need to wait for the Prometheus Client but let's see. |
For the sake of documentation/being transparent: it seems Prometheus will enable exemplars for all time series: https://groups.google.com/g/prometheus-developers/c/zgu5hwV_2oo/m/5VfUiOfmAgAJ |
@jonatan-ivanov do you know if there is a GitHub issue open tracking the work on the Prometheus server side to accept exemplars on all time series? |
Superseded by #3996 |
Currently exemplars are not included in the counter-like metrics recorded with histograms (e.g.
foo_seconds_count
andfoo_seconds_sum
). This change grabs the last non-null exemplar from any of the histogram buckets and applies it to those sampled values.The lack of exemplars in these cases was surprising to our team as we do use the
http_server_requests_seconds_count
metrics to aggregate all Spring requests regardless of their duration and drive alerting from those metrics, we feel there would be value in following exemplars to example traced requests at that granularity.As for which metrics include exemplars or which exemplar to use, that is up for debate. I am treating
_seconds_count
and_seconds_sum
as they feel like part of a counter, which does support exemplars in Prometheus, whereas_seconds_max
is a gauge. I also am grabbing the last non-null exemplar which may favor the larger buckets but I wanted it to be predictable as well as not requiring any additional iterations of the exemplars array.