Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prometheus exemplar example #96

Merged
merged 1 commit into from
Feb 14, 2023

Conversation

dashpole
Copy link
Contributor

@dashpole dashpole commented Feb 6, 2023

Add an example of using the prometheus exporter to produce prometheus exemplars.

This PR does the following:

  • Add tracerprovider initialization with a logging exporter
  • Add trace instrumentation, which creates a span each loop
  • Use a counter and histogram metric, instead of a gauge for the example. Instrumentation is taken from the OTLP example.

If I curl the endpoint using OpenMetrics, I can see exemplars

curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:19090/metrics

# TYPE super_timer histogram
# HELP super_timer
super_timer_count{otel_scope_name="io.opentelemetry.example.prometheus"} 219.0 1676324922.849
super_timer_sum{otel_scope_name="io.opentelemetry.example.prometheus"} 219088.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="0.0"} 0.0 1676324922.849 # {span_id="b68bdb5d2557fe43",trace_id="72f259bbe3a8e9c7bc2885d5ad97e206"} 0.0 1676324922.020
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="5.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="10.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="25.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="50.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="75.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="100.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="250.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="500.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="750.0"} 0.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="1000.0"} 157.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="2500.0"} 219.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="5000.0"} 219.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="7500.0"} 219.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="10000.0"} 219.0 1676324922.849
super_timer_bucket{otel_scope_name="io.opentelemetry.example.prometheus",le="+Inf"} 219.0 1676324922.849
# TYPE example_counter counter
# HELP example_counter
example_counter_total{otel_scope_name="io.opentelemetry.example.prometheus"} 220.0 1676324922.849

The associated trace for the exemplar:

INFO: 'exampleSpan' : 72f259bbe3a8e9c7bc2885d5ad97e206 b68bdb5d2557fe43 INTERNAL [tracer: io.opentelemetry.example.prometheus:] AttributesMap{data={good=true, exampleNumber=218}, capacity=128, totalAddedValues=2}

cc @jsuereth

@dashpole
Copy link
Contributor Author

I fixed the example by moving the histogram.record call inside the try block. (thanks @jack-berg !).

Now, the only problem i'm facing is that the value of the exemplar seems to be recorded incorrectly. All iterations are about 1000 milliseconds long, which can be seen in the histogram. All observations fall into the le="1000.0" or le="2500.0" buckets. But as you can see, the exemplar appears to have a value of 0.0, and is placed on the le="0.0" bucket.

@dashpole
Copy link
Contributor Author

Using the logging exporter, I can confirm that it isn't an issue with the prometheus exporter:

INFO: metric: ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={service.name="PrometheusExporterExample", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.22.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.example.prometheus, version=null, schemaUrl=null, attributes={}}, name=super.timer, description=, unit=ms, type=HISTOGRAM, data=ImmutableHistogramData{aggregationTemporality=CUMULATIVE, points=[ImmutableHistogramPointData{getStartEpochNanos=1676326373284000000, getEpochNanos=1676326433302000000, getAttributes={}, getSum=59053.0, getCount=59, hasMin=true, getMin=1000.0, hasMax=true, getMax=1026.0, getBoundaries=[0.0, 5.0, 10.0, 25.0, 50.0, 75.0, 100.0, 250.0, 500.0, 750.0, 1000.0, 2500.0, 5000.0, 7500.0, 10000.0], getCounts=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 31, 28, 0, 0, 0, 0], getExemplars=[ImmutableDoubleExemplarData{filteredAttributes={}, epochNanos=1676326429485000000, spanContext=ImmutableSpanContext{traceId=ef29eebb7cde692aa527d9c114b384af, spanId=e0a4334cc4242518, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, value=0.0}, ImmutableDoubleExemplarData{filteredAttributes={}, epochNanos=1676326432488000000, spanContext=ImmutableSpanContext{traceId=9c16ce50f3788156dc0fb8424810b7bb, spanId=94e329af9852cc7c, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, value=0.0}]}]}}

The values of both are 0.0, even though only higher buckets have any observations.

@dashpole
Copy link
Contributor Author

Even though exemplar recording isn't currently working, I think this example is now correct, and should work after that is fixed. I consider this ready for review.

@dashpole
Copy link
Contributor Author

(sorry, tests are passing now)

@trask
Copy link
Member

trask commented Feb 14, 2023

thx!

@dashpole dashpole deleted the prom_tracer_example branch February 15, 2023 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants