Where are the Exemplars coming from?

[fstab]

Sorry if this is the wrong place to ask a beginners question, but I'm trying to figure out where Exemplars are coming from.

I ran the query as described in the Readme:

histogram_quantile(.99, sum(rate(tns_request_duration_seconds_bucket{}[1m])) by (le))

It works. I see exemplars:

So I wanted to have a look at the raw metrics:

 kubectl port-forward db-5dff567d6f-897vf 8081:80 --namespace tns

However, on http://localhost:8081/metrics I see just the metrics, but no exemplars:

tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.005"} 17293
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.01"} 17296
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.025"} 17309
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.05"} 17340
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.1"} 17353
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.25"} 17358
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.5"} 17358
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="1"} 17358
...

So I'm wondering: Where are the exemplars coming from?

[ajayanaj]

I'm also curios to know this, just started play around this but couldn't find where this is adding.

[csmarchbanks]

Hello, in order to expose exemplars on a target you have to negotiate to use OpenMetrics exposition format, which a browser will not do. If you run:

curl -H 'Accept: application/openmetrics-text' http://localhost:8001/metrics

you will see the exemplars in your output. E.g.

tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.005"} 875 # {traceID="22a4246b4fa9d054"} 0.001348869 1.6233427882468455e+09
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.01"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.025"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.05"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.1"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.25"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="0.5"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="1.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="2.5"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="5.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="10.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="25.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="50.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="100.0"} 875
tns_request_duration_seconds_bucket{method="GET",route="root",status_code="200",ws="false",le="+Inf"} 875
tns_request_duration_seconds_sum{method="GET",route="root",status_code="200",ws="false"} 0.9905661310000008
tns_request_duration_seconds_count{method="GET",route="root",status_code="200",ws="false"} 875

[boniek83]

How to add them in the code though? Is there any guide available anywhere?

[VineethReddy02]

You can instrument OpenMetrics exposition by enabling it in the handler level

func main(){
  http.Handle("/metrics", promhttp.HandleFor(
      promhttp.HandlerOpts{
         EnableOpenMetrics: true,
      },
  ))
}

For more details on Exemplars here is write-up that explains inner working of metrics with traces.

[pkrishnanreflexion]

I am not able to see the exemplars for the above query. They are not visible.