Introduce shared OtelMetricRegistry to track metrics by instrumentation scope#10854
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits intomasterfrom Mar 17, 2026
Merged
Conversation
PerfectSlayer
approved these changes
Mar 16, 2026
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
Looking good but I will let the SDK Cap have a look!
I left one quick question for my understand, thanks!
| } | ||
|
|
||
| public void recordLong(long value, Attributes attributes) { | ||
| public void recordLong(long value, Object attributes) { |
Contributor
There was a problem hiding this comment.
❔ question: I guess the type was erased to not have OTel attributes into agent bootstrap.
How is that working with the CARDINALITY_OVERFLOW constant then?
Contributor
Author
There was a problem hiding this comment.
Correct:
To achieve a single metric registry we cannot use the specific Attributes class in the generic type of the map or certain method argument types. This is because the Attributes type for drop-in will be different to the Attributes type provided on the application class-path. We use Object since there is no other common type.
At collection time we'll use the appropriate function to map either Attributes type to the right OTLP content.
CARDINALITY_OVERFLOW is a shared internal constant, so both drop-in and the instrumentation will end up using the same key for that case.
using OTel attributes from both bootstrap and application classpaths
…on scope. This will allow metrics produced by drop-in support and the OTel API to coexist.
…the bootstrap classpath
…extended mid-build (Needed for the extra muzzleBootstrap entry in opentelemetry-1.47 to take effect in CI)
mcculls
force-pushed
the
mcculls/otel-metric-registry
branch
from
4dc8767 to
c7bbdcc
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1055321
Total [baseline] (8.842 s) : 0, 8842487
Agent [candidate] (1.069 s) : 0, 1068539
Total [candidate] (8.845 s) : 0, 8845046
section iast
Agent [baseline] (1.243 s) : 0, 1242633
Total [baseline] (9.57 s) : 0, 9569705
Agent [candidate] (1.229 s) : 0, 1228752
Total [candidate] (9.558 s) : 0, 9558436
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.204 ms) : 0, 1204
BytebuddyAgent [baseline] (626.635 ms) : 0, 626635
BytebuddyAgent [candidate] (633.922 ms) : 0, 633922
AgentMeter [baseline] (29.059 ms) : 0, 29059
AgentMeter [candidate] (29.3 ms) : 0, 29300
GlobalTracer [baseline] (256.284 ms) : 0, 256284
GlobalTracer [candidate] (259.175 ms) : 0, 259175
AppSec [baseline] (31.686 ms) : 0, 31686
AppSec [candidate] (31.888 ms) : 0, 31888
Debugger [baseline] (59.376 ms) : 0, 59376
Debugger [candidate] (59.984 ms) : 0, 59984
Remote Config [baseline] (585.895 µs) : 0, 586
Remote Config [candidate] (607.59 µs) : 0, 608
Telemetry [baseline] (8.059 ms) : 0, 8059
Telemetry [candidate] (8.153 ms) : 0, 8153
Flare Poller [baseline] (6.399 ms) : 0, 6399
Flare Poller [candidate] (8.101 ms) : 0, 8101
section iast
crashtracking [baseline] (1.22 ms) : 0, 1220
crashtracking [candidate] (1.203 ms) : 0, 1203
BytebuddyAgent [baseline] (807.337 ms) : 0, 807337
BytebuddyAgent [candidate] (796.966 ms) : 0, 796966
AgentMeter [baseline] (11.86 ms) : 0, 11860
AgentMeter [candidate] (11.343 ms) : 0, 11343
GlobalTracer [baseline] (249.944 ms) : 0, 249944
GlobalTracer [candidate] (248.261 ms) : 0, 248261
AppSec [baseline] (26.864 ms) : 0, 26864
AppSec [candidate] (26.456 ms) : 0, 26456
Debugger [baseline] (68.352 ms) : 0, 68352
Debugger [candidate] (68.038 ms) : 0, 68038
Remote Config [baseline] (516.505 µs) : 0, 517
Remote Config [candidate] (518.012 µs) : 0, 518
Telemetry [baseline] (10.661 ms) : 0, 10661
Telemetry [candidate] (10.729 ms) : 0, 10729
Flare Poller [baseline] (3.765 ms) : 0, 3765
Flare Poller [candidate] (3.798 ms) : 0, 3798
IAST [baseline] (25.691 ms) : 0, 25691
IAST [candidate] (25.27 ms) : 0, 25270
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058068
Total [baseline] (10.959 s) : 0, 10958879
Agent [candidate] (1.061 s) : 0, 1060581
Total [candidate] (11.018 s) : 0, 11017588
section appsec
Agent [baseline] (1.258 s) : 0, 1258193
Total [baseline] (11.31 s) : 0, 11309662
Agent [candidate] (1.245 s) : 0, 1244848
Total [candidate] (11.144 s) : 0, 11143741
section iast
Agent [baseline] (1.237 s) : 0, 1236905
Total [baseline] (11.391 s) : 0, 11390610
Agent [candidate] (1.24 s) : 0, 1240053
Total [candidate] (11.517 s) : 0, 11516844
section profiling
Agent [baseline] (1.184 s) : 0, 1184464
Total [baseline] (10.966 s) : 0, 10965584
Agent [candidate] (1.19 s) : 0, 1190155
Total [candidate] (11.125 s) : 0, 11125152
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.204 ms) : 0, 1204
crashtracking [candidate] (1.191 ms) : 0, 1191
BytebuddyAgent [baseline] (627.353 ms) : 0, 627353
BytebuddyAgent [candidate] (627.842 ms) : 0, 627842
AgentMeter [baseline] (29.017 ms) : 0, 29017
AgentMeter [candidate] (29.019 ms) : 0, 29019
GlobalTracer [baseline] (256.828 ms) : 0, 256828
GlobalTracer [candidate] (257.109 ms) : 0, 257109
AppSec [baseline] (31.683 ms) : 0, 31683
AppSec [candidate] (31.617 ms) : 0, 31617
Debugger [baseline] (60.068 ms) : 0, 60068
Debugger [candidate] (60.229 ms) : 0, 60229
Remote Config [baseline] (610.234 µs) : 0, 610
Remote Config [candidate] (592.115 µs) : 0, 592
Telemetry [baseline] (7.96 ms) : 0, 7960
Telemetry [candidate] (8.067 ms) : 0, 8067
Flare Poller [baseline] (7.278 ms) : 0, 7278
Flare Poller [candidate] (8.812 ms) : 0, 8812
section appsec
crashtracking [baseline] (1.205 ms) : 0, 1205
crashtracking [candidate] (1.205 ms) : 0, 1205
BytebuddyAgent [baseline] (663.456 ms) : 0, 663456
BytebuddyAgent [candidate] (656.749 ms) : 0, 656749
AgentMeter [baseline] (12.195 ms) : 0, 12195
AgentMeter [candidate] (12.05 ms) : 0, 12050
GlobalTracer [baseline] (261.118 ms) : 0, 261118
GlobalTracer [candidate] (258.068 ms) : 0, 258068
AppSec [baseline] (179.193 ms) : 0, 179193
AppSec [candidate] (177.507 ms) : 0, 177507
Debugger [baseline] (67.058 ms) : 0, 67058
Debugger [candidate] (66.34 ms) : 0, 66340
Remote Config [baseline] (633.931 µs) : 0, 634
Remote Config [candidate] (616.755 µs) : 0, 617
Telemetry [baseline] (8.481 ms) : 0, 8481
Telemetry [candidate] (8.243 ms) : 0, 8243
Flare Poller [baseline] (3.703 ms) : 0, 3703
Flare Poller [candidate] (3.605 ms) : 0, 3605
IAST [baseline] (24.579 ms) : 0, 24579
IAST [candidate] (24.123 ms) : 0, 24123
section iast
crashtracking [baseline] (1.203 ms) : 0, 1203
crashtracking [candidate] (1.21 ms) : 0, 1210
BytebuddyAgent [baseline] (801.855 ms) : 0, 801855
BytebuddyAgent [candidate] (804.553 ms) : 0, 804553
AgentMeter [baseline] (11.609 ms) : 0, 11609
AgentMeter [candidate] (11.658 ms) : 0, 11658
GlobalTracer [baseline] (249.031 ms) : 0, 249031
GlobalTracer [candidate] (249.113 ms) : 0, 249113
AppSec [baseline] (26.842 ms) : 0, 26842
AppSec [candidate] (26.777 ms) : 0, 26777
Debugger [baseline] (71.316 ms) : 0, 71316
Debugger [candidate] (70.768 ms) : 0, 70768
Remote Config [baseline] (527.093 µs) : 0, 527
Remote Config [candidate] (549.493 µs) : 0, 549
Telemetry [baseline] (9.216 ms) : 0, 9216
Telemetry [candidate] (9.846 ms) : 0, 9846
Flare Poller [baseline] (3.384 ms) : 0, 3384
Flare Poller [candidate] (3.572 ms) : 0, 3572
IAST [baseline] (25.6 ms) : 0, 25600
IAST [candidate] (25.586 ms) : 0, 25586
section profiling
ProfilingAgent [baseline] (93.522 ms) : 0, 93522
ProfilingAgent [candidate] (93.878 ms) : 0, 93878
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.176 ms) : 0, 1176
BytebuddyAgent [baseline] (683.969 ms) : 0, 683969
BytebuddyAgent [candidate] (687.385 ms) : 0, 687385
AgentMeter [baseline] (8.68 ms) : 0, 8680
AgentMeter [candidate] (8.741 ms) : 0, 8741
GlobalTracer [baseline] (216.052 ms) : 0, 216052
GlobalTracer [candidate] (217.071 ms) : 0, 217071
AppSec [baseline] (32.289 ms) : 0, 32289
AppSec [candidate] (32.458 ms) : 0, 32458
Debugger [baseline] (62.348 ms) : 0, 62348
Debugger [candidate] (64.035 ms) : 0, 64035
Remote Config [baseline] (578.413 µs) : 0, 578
Remote Config [candidate] (590.994 µs) : 0, 591
Telemetry [baseline] (10.748 ms) : 0, 10748
Telemetry [candidate] (9.998 ms) : 0, 9998
Flare Poller [baseline] (4.226 ms) : 0, 4226
Flare Poller [candidate] (3.541 ms) : 0, 3541
Profiling [baseline] (94.09 ms) : 0, 94090
Profiling [candidate] (94.431 ms) : 0, 94431
LoadParameters
See matching parameters
SummaryFound 5 performance improvements and 2 performance regressions! Performance is the same for 13 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section baseline
no_agent (1.189 ms) : 1177, 1201
. : milestone, 1189,
iast (3.176 ms) : 3134, 3218
. : milestone, 3176,
iast_FULL (6.125 ms) : 6060, 6190
. : milestone, 6125,
iast_GLOBAL (3.493 ms) : 3442, 3545
. : milestone, 3493,
profiling (2.367 ms) : 2343, 2390
. : milestone, 2367,
tracing (1.905 ms) : 1889, 1922
. : milestone, 1905,
section candidate
no_agent (1.19 ms) : 1178, 1202
. : milestone, 1190,
iast (3.247 ms) : 3204, 3291
. : milestone, 3247,
iast_FULL (5.865 ms) : 5806, 5924
. : milestone, 5865,
iast_GLOBAL (3.627 ms) : 3563, 3692
. : milestone, 3627,
profiling (2.008 ms) : 1990, 2026
. : milestone, 2008,
tracing (1.78 ms) : 1766, 1795
. : milestone, 1780,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section baseline
no_agent (18.218 ms) : 18033, 18404
. : milestone, 18218,
appsec (18.473 ms) : 18283, 18663
. : milestone, 18473,
code_origins (18.468 ms) : 18284, 18652
. : milestone, 18468,
iast (18.524 ms) : 18337, 18711
. : milestone, 18524,
profiling (18.302 ms) : 18121, 18483
. : milestone, 18302,
tracing (17.595 ms) : 17418, 17772
. : milestone, 17595,
section candidate
no_agent (18.323 ms) : 18131, 18515
. : milestone, 18323,
appsec (18.567 ms) : 18380, 18753
. : milestone, 18567,
code_origins (17.59 ms) : 17420, 17760
. : milestone, 17590,
iast (17.652 ms) : 17481, 17823
. : milestone, 17652,
profiling (20.579 ms) : 20376, 20781
. : milestone, 20579,
tracing (17.492 ms) : 17319, 17666
. : milestone, 17492,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section baseline
no_agent (15.027 s) : 15027000, 15027000
. : milestone, 15027000,
appsec (14.904 s) : 14904000, 14904000
. : milestone, 14904000,
iast (18.277 s) : 18277000, 18277000
. : milestone, 18277000,
iast_GLOBAL (17.953 s) : 17953000, 17953000
. : milestone, 17953000,
profiling (15.118 s) : 15118000, 15118000
. : milestone, 15118000,
tracing (14.796 s) : 14796000, 14796000
. : milestone, 14796000,
section candidate
no_agent (15.538 s) : 15538000, 15538000
. : milestone, 15538000,
appsec (14.645 s) : 14645000, 14645000
. : milestone, 14645000,
iast (18.51 s) : 18510000, 18510000
. : milestone, 18510000,
iast_GLOBAL (18.178 s) : 18178000, 18178000
. : milestone, 18178000,
profiling (14.896 s) : 14896000, 14896000
. : milestone, 14896000,
tracing (14.695 s) : 14695000, 14695000
. : milestone, 14695000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~7148b8205b, baseline=1.61.0-SNAPSHOT~95b7859b82
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (3.885 ms) : 3656, 4115
. : milestone, 3885,
iast (2.261 ms) : 2192, 2331
. : milestone, 2261,
iast_GLOBAL (2.301 ms) : 2231, 2370
. : milestone, 2301,
profiling (2.097 ms) : 2042, 2152
. : milestone, 2097,
tracing (2.063 ms) : 2009, 2116
. : milestone, 2063,
section candidate
no_agent (1.473 ms) : 1462, 1485
. : milestone, 1473,
appsec (3.724 ms) : 3510, 3938
. : milestone, 3724,
iast (2.251 ms) : 2182, 2321
. : milestone, 2251,
iast_GLOBAL (2.303 ms) : 2233, 2373
. : milestone, 2303,
profiling (2.516 ms) : 2351, 2680
. : milestone, 2516,
tracing (2.066 ms) : 2012, 2119
. : milestone, 2066,
|
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
stuart.mcculloch@datadoghq.com cancelled this merge request build |
Contributor
Author
|
/merge -c |
|
View all feedbacks in Devflow UI.
|
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
gh-worker-dd-mergequeue-cf854d
bot
deleted the
mcculls/otel-metric-registry
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Introduce shared
OtelMetricRegistryto track metrics by instrumentation scope.At the same time move any OTel support classes not tied to the OTel API to
otel-bootstrap- this avoids duplication of these classes (no need to have two copies for OTel drop-in vs OTel application) and also reduces the amount of helpers needing injection.Motivation
This will allow metrics produced by drop-in support and the OTel API to coexist within the same registry, rather than have two disconnected registries.
Additional Notes
To achieve a single metric registry we cannot use the specific
Attributesclass in the generic type of the map or certain method argument types. This is because theAttributestype for drop-in will be different to theAttributestype provided on the application class-path. We useObjectsince there is no other common type.At collection time we'll use the appropriate function to map either
Attributestype to the right OTLP content.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.