Add scope info and version to Prometheus exporter #5086

robertcoltheart · 2023-11-27T10:53:04Z

Changes

Add instrument scope info and version to Prometheus exporter as per https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#instrumentation-scope-1

Scope info is automatically added to the Prometheus exporters.

Before this change:

# TYPE MyFruitCounter_total counter
MyFruitCounter_total{color="red",name="apple"} 66 1702578315791
MyFruitCounter_total{color="yellow",name="lemon"} 77 1702578315791
MyFruitCounter_total{color="green",name="apple"} 22 1702578315791

# EOF

After (with no Accept header):

# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="MyCompany.MyProduct.MyLibrary"} 1
# TYPE MyFruitCounter_total counter
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="red",name="apple"} 102 1702577575197
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="yellow",name="lemon"} 119 1702577575197
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="green",name="apple"} 34 1702577575197
# EOF

After (when passing Accept: application/openmetrics-text:

# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="MyCompany.MyProduct.MyLibrary"} 1
# TYPE MyFruitCounter_total counter
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="red",name="apple"} 4692 1702577811.302
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="yellow",name="lemon"} 5474 1702577811.302
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="green",name="apple"} 1564 1702577811.302
# EOF

Merge requirement checklist

CONTRIBUTING guidelines followed (nullable enabled, static analysis, etc.)
Unit tests added/updated
Appropriate CHANGELOG.md files updated for non-trivial changes
Changes in public API reviewed (if applicable)

linux-foundation-easycla · 2023-11-27T10:53:08Z

The committers listed above are authorized under a signed CLA.

✅ login: robertcoltheart / name: Robert Coltheart (665537c, 676b700, 0203e64, 44eaa4d, f8ee51b, 395a7fc, 4ba86a8, ab9292c, 6192827, 74014ac, 9526647, da28e9d, 937b7da, 42da2d2, 5b5f0df, 4e5e577, b5c80db, f0b263d, 6753ad9, d452ab2, 8727a79, 63f0b36, a21c6ce, 033a903, b33199a, 89f735d, 50a9bde, 0e1be38, aa16875, fca0b18, 22bfa08, 026cfeb, a7cc5fd)
✅ login: cijothomas / name: Cijo Thomas (3c134cd)
✅ login: utpilla / name: Utkarsh Umesan Pillai (2260d8b, 03d3953)

utpilla · 2023-11-29T00:05:25Z

@robertcoltheart Thanks for sending out this PR! Could you fix the build errors?

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs

robertcoltheart · 2023-11-29T09:01:25Z

Fixed build errors, I think, can you re-run?

codecov · 2023-11-29T16:17:27Z

Codecov Report

Merging #5086 (a7cc5fd) into main (c0969d0) will increase coverage by 0.18%.
Report is 2 commits behind head on main.
The diff coverage is 88.09%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5086      +/-   ##
==========================================
+ Coverage   83.26%   83.45%   +0.18%     
==========================================
  Files         297      297              
  Lines       12366    12386      +20     
==========================================
+ Hits        10297    10337      +40     
+ Misses       2069     2049      -20

Flag	Coverage Δ
unittests	`83.45% <88.09%> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...s.HttpListener/Internal/PrometheusSerializerExt.cs	`100.00% <100.00%> (ø)`
...heus.HttpListener/Internal/PrometheusSerializer.cs	`84.86% <96.29%> (+3.26%)`	⬆️
...tpListener/Internal/PrometheusCollectionManager.cs	`77.58% <63.63%> (-1.47%)`	⬇️

... and 6 files with indirect coverage changes

utpilla · 2023-11-29T19:36:40Z

@robertcoltheart Could you run this example with a PrometheusExporter and share the scraped output? We would like to get an idea of how the serialized data looks like with these changes.

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs

robertcoltheart · 2023-11-29T21:04:19Z

@robertcoltheart Could you run this example with a PrometheusExporter and share the scraped output? We would like to get an idea of how the serialized data looks like with these changes.

Updated in the PR description from the example ASPNET app. The output is shortened a bit for brevity but shows histograms and counters with and without meter versions, and their respective otel_scope_info.

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

cijothomas · 2023-11-30T01:57:06Z

@robertcoltheart Could you run this example with a PrometheusExporter and share the scraped output? We would like to get an idea of how the serialized data looks like with these changes.

Updated in the PR description from the example ASPNET app. The output is shortened a bit for brevity but shows histograms and counters with and without meter versions, and their respective otel_scope_info.

can you use this example? https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/getting-started-prometheus-grafana/Program.cs and share the output with and without the feature?
I am not very sure where is meter.name,version supposed to show up.. Is that needed as part of every line? Or just once under "otel_scope_info"? Couple of examples would help with understanding the expected behavior. Thank you!

robertcoltheart · 2023-11-30T12:55:21Z

Here is the before and after using the sample above:

Before:

# TYPE MyFruitCounter_total counter
MyFruitCounter_total{color="red",name="apple"} 66 1701348839180
MyFruitCounter_total{color="yellow",name="lemon"} 77 1701348839180
MyFruitCounter_total{color="green",name="apple"} 22 1701348839180
# EOF

After:

# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="MyCompany.MyProduct.MyLibrary"} 1
# TYPE MyFruitCounter_total counter
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="red",name="apple"} 486 1701348794.500
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="yellow",name="lemon"} 567 1701348794.500
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="green",name="apple"} 162 1701348794.500
# EOF

robertcoltheart · 2023-11-30T13:03:05Z

I took a lot of the changes in this PR from the Java version of OTEL. An example output of the Java metrics using the above format is below:

# TYPE target info
# HELP target Target metadata
target_info{service_name="PromExample",service_namespace="io.my.namespace",service_version="1.2.3",telemetry_sdk_language="java",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.26.0"} 1
# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="io.example"} 1
# TYPE example_counter counter
# HELP example_counter 
example_counter_total{otel_scope_name="io.example"} 27.0 1701349347.895
# TYPE super_timer histogram
# HELP super_timer 
super_timer_count{otel_scope_name="io.example"} 27.0 1701349347.895
super_timer_sum{otel_scope_name="io.example"} 27293.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="0.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="5.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="10.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="25.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="50.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="75.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="100.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="250.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="500.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="750.0"} 0.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="1000.0"} 1.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="2500.0"} 27.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="5000.0"} 27.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="7500.0"} 27.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="10000.0"} 27.0 1701349347.895
super_timer_bucket{otel_scope_name="io.example",le="+Inf"} 27.0 1701349347.895
# EOF

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusExporter.cs

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializerExt.cs

src/OpenTelemetry.Exporter.Prometheus.AspNetCore/.publicApi/PublicAPI.Unshipped.txt

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md

cijothomas · 2023-12-13T19:31:44Z

Here is the before and after using the sample above:

Before:

# TYPE MyFruitCounter_total counter
MyFruitCounter_total{color="red",name="apple"} 66 1701348839180
MyFruitCounter_total{color="yellow",name="lemon"} 77 1701348839180
MyFruitCounter_total{color="green",name="apple"} 22 1701348839180
# EOF

After:

# TYPE otel_scope_info info
# HELP otel_scope_info Scope metadata
otel_scope_info{otel_scope_name="MyCompany.MyProduct.MyLibrary"} 1
# TYPE MyFruitCounter_total counter
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="red",name="apple"} 486 1701348794.500
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="yellow",name="lemon"} 567 1701348794.500
MyFruitCounter_total{otel_scope_name="MyCompany.MyProduct.MyLibrary",otel_scope_version="1.0",color="green",name="apple"} 162 1701348794.500
# EOF

Could you make this example in the PR description to keep it easy to follow?

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

utpilla · 2023-12-15T01:33:23Z

src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Unreleased

 * Export OpenMetrics format from Prometheus exporters ([#5107](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5107))
+* For requests with OpenMetrics format, scope info is automatically added ([#5086](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5086))


@robertcoltheart Scope info is added regardless of whether OpenMetrics is added, right?

otel_scope_name and otel_scope_version tags are added regardless of what is requested, yes. This mirrors the behavior of the Java OTEL library. The otel_scope_info is only added when OpenMetrics is requested, otherwise Prometheus doesn't know how to handle info types and will fault.

The otel_scope_info is only added when OpenMetrics is requested

The example that you provided in the PR description has otel_scope_info added for both the cases though.

Damn, I don't know how I missed that. Sorry about that, let me raise a PR for that right now.

No worries!

Add scope info and version to Prometheus exporter

665537c

robertcoltheart added 3 commits November 28, 2023 08:36

Add tests and unshipped api

676b700

Fix tests

0203e64

Add to changelog

44eaa4d

robertcoltheart marked this pull request as ready for review November 27, 2023 22:16

robertcoltheart requested a review from a team as a code owner November 27, 2023 22:16

cijothomas mentioned this pull request Nov 28, 2023

Add support for instrumentation scope attributes (Meter tags) #5089

Merged

4 tasks

cijothomas reviewed Nov 29, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs Outdated Show resolved Hide resolved

robertcoltheart and others added 2 commits November 29, 2023 19:55

Fix build and reuse scopes

f8ee51b

Merge branch 'main' into feature/prometheus-scope-info

395a7fc

robertcoltheart and others added 2 commits November 30, 2023 07:49

Fix build, add extra newline for scope_info

4ba86a8

Merge branch 'main' into feature/prometheus-scope-info

ab9292c

robertcoltheart commented Nov 29, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs Outdated Show resolved Hide resolved

robertcoltheart added 2 commits November 30, 2023 08:13

Fix tests

6192827

Merge branch 'feature/prometheus-scope-info' of https://github.com/ro…

74014ac

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

robertcoltheart and others added 2 commits November 30, 2023 23:47

Working with prometheus now

9526647

Merge branch 'main' into feature/prometheus-scope-info

da28e9d

robertcoltheart and others added 2 commits December 1, 2023 08:50

Modify output depending on accept headers

937b7da

Merge branch 'main' into feature/prometheus-scope-info

42da2d2

robertcoltheart commented Nov 30, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusExporter.cs Outdated Show resolved Hide resolved

Add tests

5b5f0df

robertcoltheart added 5 commits December 7, 2023 20:09

Tidy

f0b263d

Fixed changelog

6753ad9

Add tests and ability to disable scope info

d452ab2

Fix build

8727a79

Spelling

63f0b36

robertcoltheart marked this pull request as ready for review December 7, 2023 12:07

cijothomas and others added 2 commits December 7, 2023 10:14

Merge branch 'main' into feature/prometheus-scope-info

3c134cd

Merge branch 'main' into feature/prometheus-scope-info

a21c6ce

reyang reviewed Dec 9, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializerExt.cs Outdated Show resolved Hide resolved

reyang reviewed Dec 9, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.AspNetCore/.publicApi/PublicAPI.Unshipped.txt Outdated Show resolved Hide resolved

robertcoltheart added 5 commits December 10, 2023 19:19

Scope info is not optional

033a903

Merge branch 'feature/prometheus-scope-info' of https://github.com/ro…

b33199a

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

Tidy code

89f735d

Fix tests

50a9bde

Test naming

0e1be38

cijothomas reviewed Dec 13, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.AspNetCore/CHANGELOG.md Outdated Show resolved Hide resolved

robertcoltheart and others added 2 commits December 15, 2023 05:05

Updated changelog

aa16875

Merge branch 'main' into feature/prometheus-scope-info

fca0b18

reyang approved these changes Dec 14, 2023

View reviewed changes

robertcoltheart and others added 3 commits December 15, 2023 05:28

Fix build

22bfa08

Merge branch 'main' into feature/prometheus-scope-info

2260d8b

Merge branch 'main' into feature/prometheus-scope-info

03d3953

utpilla reviewed Dec 14, 2023

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs Outdated Show resolved Hide resolved

robertcoltheart added 2 commits December 15, 2023 11:14

Combine scope writing with hashset

026cfeb

Merge branch 'feature/prometheus-scope-info' of https://github.com/ro…

a7cc5fd

…bertcoltheart/opentelemetry-dotnet into feature/prometheus-scope-info

utpilla approved these changes Dec 15, 2023

View reviewed changes

utpilla merged commit 3c986e8 into open-telemetry:main Dec 15, 2023
78 checks passed

robertcoltheart deleted the feature/prometheus-scope-info branch December 15, 2023 00:58

utpilla reviewed Dec 15, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scope info and version to Prometheus exporter #5086

Add scope info and version to Prometheus exporter #5086

robertcoltheart commented Nov 27, 2023 •

edited

linux-foundation-easycla bot commented Nov 27, 2023 •

edited

utpilla commented Nov 29, 2023

robertcoltheart commented Nov 29, 2023

codecov bot commented Nov 29, 2023 •

edited

utpilla commented Nov 29, 2023

robertcoltheart commented Nov 29, 2023

cijothomas commented Nov 30, 2023

robertcoltheart commented Nov 30, 2023

robertcoltheart commented Nov 30, 2023

cijothomas commented Dec 13, 2023

utpilla Dec 15, 2023

robertcoltheart Dec 15, 2023

utpilla Dec 15, 2023

robertcoltheart Dec 15, 2023

utpilla Dec 15, 2023

robertcoltheart Dec 15, 2023

Add scope info and version to Prometheus exporter #5086

Add scope info and version to Prometheus exporter #5086

Conversation

robertcoltheart commented Nov 27, 2023 • edited

Changes

Merge requirement checklist

linux-foundation-easycla bot commented Nov 27, 2023 • edited

utpilla commented Nov 29, 2023

robertcoltheart commented Nov 29, 2023

codecov bot commented Nov 29, 2023 • edited

Codecov Report

utpilla commented Nov 29, 2023

robertcoltheart commented Nov 29, 2023

cijothomas commented Nov 30, 2023

robertcoltheart commented Nov 30, 2023

robertcoltheart commented Nov 30, 2023

cijothomas commented Dec 13, 2023

utpilla Dec 15, 2023

Choose a reason for hiding this comment

robertcoltheart Dec 15, 2023

Choose a reason for hiding this comment

utpilla Dec 15, 2023

Choose a reason for hiding this comment

robertcoltheart Dec 15, 2023

Choose a reason for hiding this comment

utpilla Dec 15, 2023

Choose a reason for hiding this comment

robertcoltheart Dec 15, 2023

Choose a reason for hiding this comment

robertcoltheart commented Nov 27, 2023 •

edited

linux-foundation-easycla bot commented Nov 27, 2023 •

edited

codecov bot commented Nov 29, 2023 •

edited