Lazily obtain feature discovery when starting client metrics #9548

amarziali · 2025-09-16T14:13:19Z

What Does This Do

Calling SharedCommunicationObjects.featureDiscovery forces a immediate call to the agent that causes I/O and premature initialisation of okhttp. This is hence causing I/O.

The agent is usually waiting for custom log manager / jmx initialisation before doing that but the client metrics aggregator escaped somehow to this logic

Also because of that, we notice an increased startup time in benchmarks that looks related to that I/O
Additionally the benchmark runs the application and the agent at same time and this can cause a downgrade to be detected and another forced info to be called.

This PR fixes that behaviour obtaining the feature discovery lazily. Also the discovery upon stats 404 is done asynchronously since there is no need to block for it.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

datadog-official · 2025-09-16T14:32:00Z

🎯 Code Coverage
• Patch Coverage: 82.35%
• Total Coverage: 63.07% (+3.26%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 36b62dd | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-09-16T14:44:26Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/metrics-lazy
git_commit_date	1758219687	1758267331
git_commit_sha	`1052f49`	`36b62dd`
release_version	1.54.0-SNAPSHOT~1052f4935d	1.54.0-SNAPSHOT~36b62dd6a7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1758269312	1758269312
ci_job_id	1137613978	1137613978
ci_pipeline_id	76950391	76950391
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-l6l2790i 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-l6l2790i 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 10 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1061962
Total [baseline] (10.656 s) : 0, 10655604
Agent [candidate] (1.076 s) : 0, 1075769
Total [candidate] (10.731 s) : 0, 10730515
section appsec
Agent [baseline] (1.244 s) : 0, 1244088
Total [baseline] (11.008 s) : 0, 11008005
Agent [candidate] (1.242 s) : 0, 1242446
Total [candidate] (11.108 s) : 0, 11108135
section iast
Agent [baseline] (1.191 s) : 0, 1191285
Total [baseline] (11.055 s) : 0, 11055322
Agent [candidate] (1.197 s) : 0, 1196532
Total [candidate] (11.045 s) : 0, 11045288
section profiling
Agent [baseline] (1.211 s) : 0, 1211027
Total [baseline] (10.854 s) : 0, 10853787
Agent [candidate] (1.219 s) : 0, 1219156
Total [candidate] (11.006 s) : 0, 11005542

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.062 s	-
Agent	appsec	1.244 s	182.125 ms (17.1%)
Agent	iast	1.191 s	129.322 ms (12.2%)
Agent	profiling	1.211 s	149.065 ms (14.0%)
Total	tracing	10.656 s	-
Total	appsec	11.008 s	352.401 ms (3.3%)
Total	iast	11.055 s	399.718 ms (3.8%)
Total	profiling	10.854 s	198.183 ms (1.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.076 s	-
Agent	appsec	1.242 s	166.676 ms (15.5%)
Agent	iast	1.197 s	120.762 ms (11.2%)
Agent	profiling	1.219 s	143.387 ms (13.3%)
Total	tracing	10.731 s	-
Total	appsec	11.108 s	377.62 ms (3.5%)
Total	iast	11.045 s	314.773 ms (2.9%)
Total	profiling	11.006 s	275.028 ms (2.6%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.474 ms) : 0, 1474
crashtracking [candidate] (1.486 ms) : 0, 1486
BytebuddyAgent [baseline] (732.734 ms) : 0, 732734
BytebuddyAgent [candidate] (742.502 ms) : 0, 742502
GlobalTracer [baseline] (252.65 ms) : 0, 252650
GlobalTracer [candidate] (255.965 ms) : 0, 255965
AppSec [baseline] (30.705 ms) : 0, 30705
AppSec [candidate] (31.044 ms) : 0, 31044
Debugger [baseline] (6.391 ms) : 0, 6391
Debugger [candidate] (6.452 ms) : 0, 6452
Remote Config [baseline] (699.097 µs) : 0, 699
Remote Config [candidate] (699.107 µs) : 0, 699
Telemetry [baseline] (16.296 ms) : 0, 16296
Telemetry [candidate] (16.407 ms) : 0, 16407
section appsec
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.449 ms) : 0, 1449
BytebuddyAgent [baseline] (762.646 ms) : 0, 762646
BytebuddyAgent [candidate] (759.056 ms) : 0, 759056
GlobalTracer [baseline] (247.556 ms) : 0, 247556
GlobalTracer [candidate] (248.892 ms) : 0, 248892
AppSec [baseline] (171.805 ms) : 0, 171805
AppSec [candidate] (171.723 ms) : 0, 171723
Debugger [baseline] (5.981 ms) : 0, 5981
Debugger [candidate] (6.029 ms) : 0, 6029
Remote Config [baseline] (621.869 µs) : 0, 622
Remote Config [candidate] (644.153 µs) : 0, 644
Telemetry [baseline] (8.503 ms) : 0, 8503
Telemetry [candidate] (9.372 ms) : 0, 9372
IAST [baseline] (24.142 ms) : 0, 24142
IAST [candidate] (24.006 ms) : 0, 24006
section iast
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.441 ms) : 0, 1441
BytebuddyAgent [baseline] (851.708 ms) : 0, 851708
BytebuddyAgent [candidate] (855.266 ms) : 0, 855266
GlobalTracer [baseline] (246.052 ms) : 0, 246052
GlobalTracer [candidate] (246.065 ms) : 0, 246065
AppSec [baseline] (25.815 ms) : 0, 25815
AppSec [candidate] (29.25 ms) : 0, 29250
Debugger [baseline] (6.163 ms) : 0, 6163
Debugger [candidate] (6.13 ms) : 0, 6130
Remote Config [baseline] (609.377 µs) : 0, 609
Remote Config [candidate] (597.983 µs) : 0, 598
Telemetry [baseline] (8.317 ms) : 0, 8317
Telemetry [candidate] (8.409 ms) : 0, 8409
IAST [baseline] (30.2 ms) : 0, 30200
IAST [candidate] (28.246 ms) : 0, 28246
section profiling
ProfilingAgent [baseline] (108.627 ms) : 0, 108627
ProfilingAgent [candidate] (109.883 ms) : 0, 109883
crashtracking [baseline] (1.443 ms) : 0, 1443
crashtracking [candidate] (1.446 ms) : 0, 1446
BytebuddyAgent [baseline] (761.846 ms) : 0, 761846
BytebuddyAgent [candidate] (767.105 ms) : 0, 767105
GlobalTracer [baseline] (233.241 ms) : 0, 233241
GlobalTracer [candidate] (235.08 ms) : 0, 235080
AppSec [baseline] (31.306 ms) : 0, 31306
AppSec [candidate] (30.388 ms) : 0, 30388
Debugger [baseline] (12.23 ms) : 0, 12230
Debugger [candidate] (14.528 ms) : 0, 14528
Remote Config [baseline] (733.615 µs) : 0, 734
Remote Config [candidate] (755.952 µs) : 0, 756
Telemetry [baseline] (10.291 ms) : 0, 10291
Telemetry [candidate] (8.638 ms) : 0, 8638
Profiling [baseline] (109.264 ms) : 0, 109264
Profiling [candidate] (110.525 ms) : 0, 110525

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1068250
Total [baseline] (8.685 s) : 0, 8685048
Agent [candidate] (1.064 s) : 0, 1063697
Total [candidate] (8.605 s) : 0, 8605388
section iast
Agent [baseline] (1.19 s) : 0, 1189864
Total [baseline] (9.25 s) : 0, 9249875
Agent [candidate] (1.19 s) : 0, 1190285
Total [candidate] (9.275 s) : 0, 9275188

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.068 s	-
Agent	iast	1.19 s	121.614 ms (11.4%)
Total	tracing	8.685 s	-
Total	iast	9.25 s	564.827 ms (6.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	iast	1.19 s	126.588 ms (11.9%)
Total	tracing	8.605 s	-
Total	iast	9.275 s	669.8 ms (7.8%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.469 ms) : 0, 1469
crashtracking [candidate] (1.449 ms) : 0, 1449
BytebuddyAgent [baseline] (737.884 ms) : 0, 737884
BytebuddyAgent [candidate] (733.465 ms) : 0, 733465
GlobalTracer [baseline] (253.524 ms) : 0, 253524
GlobalTracer [candidate] (253.547 ms) : 0, 253547
AppSec [baseline] (30.808 ms) : 0, 30808
AppSec [candidate] (30.763 ms) : 0, 30763
Debugger [baseline] (6.489 ms) : 0, 6489
Debugger [candidate] (6.395 ms) : 0, 6395
Remote Config [baseline] (705.398 µs) : 0, 705
Remote Config [candidate] (700.248 µs) : 0, 700
Telemetry [baseline] (16.31 ms) : 0, 16310
Telemetry [candidate] (16.407 ms) : 0, 16407
section iast
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (852.031 ms) : 0, 852031
BytebuddyAgent [candidate] (852.363 ms) : 0, 852363
GlobalTracer [baseline] (244.766 ms) : 0, 244766
GlobalTracer [candidate] (248.488 ms) : 0, 248488
AppSec [baseline] (27.037 ms) : 0, 27037
AppSec [candidate] (25.382 ms) : 0, 25382
Debugger [baseline] (6.019 ms) : 0, 6019
Debugger [candidate] (6.009 ms) : 0, 6009
Remote Config [baseline] (598.521 µs) : 0, 599
Remote Config [candidate] (603.729 µs) : 0, 604
Telemetry [baseline] (8.19 ms) : 0, 8190
Telemetry [candidate] (8.205 ms) : 0, 8205
IAST [baseline] (28.874 ms) : 0, 28874
IAST [candidate] (26.886 ms) : 0, 26886

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/metrics-lazy
git_commit_date	1758219687	1758267331
git_commit_sha	`1052f49`	`36b62dd`
release_version	1.54.0-SNAPSHOT~1052f4935d	1.54.0-SNAPSHOT~36b62dd6a7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1758268978	1758268978
ci_job_id	1137613979	1137613979
ci_pipeline_id	76950391	76950391
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-sdh9rpdu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-sdh9rpdu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 1 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:appsec:high_load	better [-2.243ms; -1.329ms] or [-4.560%; -2.703%]	unstable [-3.590op/s; +10.715op/s] or [-3.773%; +11.261%]	47.395ms	98.713op/s	49.181ms	95.150op/s
scenario:load:petclinic:no_agent:high_load	better [-2.887ms; -2.255ms] or [-7.575%; -5.914%]	unstable [+0.125op/s; +17.450op/s] or [+0.102%; +14.226%]	35.548ms	131.450op/s	38.119ms	122.662op/s
scenario:load:petclinic:iast:high_load	worse [+1.268ms; +2.099ms] or [+2.863%; +4.741%]	unstable [-11.250op/s; +3.475op/s] or [-10.647%; +3.289%]	45.966ms	101.775op/s	44.282ms	105.662op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.384 ms) : 4332, 4436
.   : milestone, 4384,
iast (9.741 ms) : 9580, 9903
.   : milestone, 9741,
iast_FULL (14.336 ms) : 14046, 14626
.   : milestone, 14336,
iast_GLOBAL (10.629 ms) : 10441, 10816
.   : milestone, 10629,
profiling (8.815 ms) : 8681, 8948
.   : milestone, 8815,
tracing (7.718 ms) : 7600, 7836
.   : milestone, 7718,
section candidate
no_agent (4.326 ms) : 4276, 4376
.   : milestone, 4326,
iast (9.381 ms) : 9226, 9536
.   : milestone, 9381,
iast_FULL (14.558 ms) : 14270, 14846
.   : milestone, 14558,
iast_GLOBAL (10.391 ms) : 10208, 10574
.   : milestone, 10391,
profiling (9.071 ms) : 8928, 9214
.   : milestone, 9071,
tracing (7.856 ms) : 7736, 7977
.   : milestone, 7856,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.384 ms [4.332 ms, 4.436 ms]	-
iast	9.741 ms [9.58 ms, 9.903 ms]	5.358 ms (122.2%)
iast_FULL	14.336 ms [14.046 ms, 14.626 ms]	9.952 ms (227.0%)
iast_GLOBAL	10.629 ms [10.441 ms, 10.816 ms]	6.245 ms (142.4%)
profiling	8.815 ms [8.681 ms, 8.948 ms]	4.431 ms (101.1%)
tracing	7.718 ms [7.6 ms, 7.836 ms]	3.334 ms (76.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.326 ms [4.276 ms, 4.376 ms]	-
iast	9.381 ms [9.226 ms, 9.536 ms]	5.054 ms (116.8%)
iast_FULL	14.558 ms [14.27 ms, 14.846 ms]	10.232 ms (236.5%)
iast_GLOBAL	10.391 ms [10.208 ms, 10.574 ms]	6.065 ms (140.2%)
profiling	9.071 ms [8.928 ms, 9.214 ms]	4.744 ms (109.7%)
tracing	7.856 ms [7.736 ms, 7.977 ms]	3.53 ms (81.6%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.119 ms) : 37815, 38424
.   : milestone, 38119,
appsec (49.181 ms) : 48754, 49608
.   : milestone, 49181,
code_origins (46.294 ms) : 45881, 46707
.   : milestone, 46294,
iast (44.282 ms) : 43891, 44674
.   : milestone, 44282,
profiling (49.305 ms) : 48825, 49786
.   : milestone, 49305,
tracing (43.874 ms) : 43487, 44262
.   : milestone, 43874,
section candidate
no_agent (35.548 ms) : 35265, 35832
.   : milestone, 35548,
appsec (47.395 ms) : 46973, 47817
.   : milestone, 47395,
code_origins (46.696 ms) : 46292, 47100
.   : milestone, 46696,
iast (45.966 ms) : 45585, 46347
.   : milestone, 45966,
profiling (49.657 ms) : 49223, 50092
.   : milestone, 49657,
tracing (44.915 ms) : 44527, 45303
.   : milestone, 44915,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.119 ms [37.815 ms, 38.424 ms]	-
appsec	49.181 ms [48.754 ms, 49.608 ms]	11.062 ms (29.0%)
code_origins	46.294 ms [45.881 ms, 46.707 ms]	8.175 ms (21.4%)
iast	44.282 ms [43.891 ms, 44.674 ms]	6.163 ms (16.2%)
profiling	49.305 ms [48.825 ms, 49.786 ms]	11.186 ms (29.3%)
tracing	43.874 ms [43.487 ms, 44.262 ms]	5.755 ms (15.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.548 ms [35.265 ms, 35.832 ms]	-
appsec	47.395 ms [46.973 ms, 47.817 ms]	11.847 ms (33.3%)
code_origins	46.696 ms [46.292 ms, 47.1 ms]	11.148 ms (31.4%)
iast	45.966 ms [45.585 ms, 46.347 ms]	10.418 ms (29.3%)
profiling	49.657 ms [49.223 ms, 50.092 ms]	14.109 ms (39.7%)
tracing	44.915 ms [44.527 ms, 45.303 ms]	9.367 ms (26.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/metrics-lazy
git_commit_date	1758219687	1758267331
git_commit_sha	`1052f49`	`36b62dd`
release_version	1.54.0-SNAPSHOT~1052f4935d	1.54.0-SNAPSHOT~36b62dd6a7

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1758269493	1758269493
ci_job_id	1137613980	1137613980
ci_pipeline_id	76950391	76950391
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-rbmvkx3t 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-rbmvkx3t 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:dacapo:tomcat:appsec	better [-1.407ms; -1.064ms] or [-37.441%; -28.316%]	2.523ms	3.758ms

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.486 ms) : 1474, 1497
.   : milestone, 1486,
appsec (3.758 ms) : 3540, 3977
.   : milestone, 3758,
iast (2.211 ms) : 2148, 2273
.   : milestone, 2211,
iast_GLOBAL (2.25 ms) : 2187, 2313
.   : milestone, 2250,
profiling (2.082 ms) : 2030, 2135
.   : milestone, 2082,
tracing (2.043 ms) : 1994, 2093
.   : milestone, 2043,
section candidate
no_agent (1.484 ms) : 1472, 1495
.   : milestone, 1484,
appsec (2.523 ms) : 2469, 2576
.   : milestone, 2523,
iast (2.207 ms) : 2145, 2270
.   : milestone, 2207,
iast_GLOBAL (2.254 ms) : 2191, 2317
.   : milestone, 2254,
profiling (2.053 ms) : 2002, 2103
.   : milestone, 2053,
tracing (2.028 ms) : 1978, 2077
.   : milestone, 2028,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.486 ms [1.474 ms, 1.497 ms]	-
appsec	3.758 ms [3.54 ms, 3.977 ms]	2.273 ms (153.0%)
iast	2.211 ms [2.148 ms, 2.273 ms]	724.834 µs (48.8%)
iast_GLOBAL	2.25 ms [2.187 ms, 2.313 ms]	764.185 µs (51.4%)
profiling	2.082 ms [2.03 ms, 2.135 ms]	596.327 µs (40.1%)
tracing	2.043 ms [1.994 ms, 2.093 ms]	557.649 µs (37.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.484 ms [1.472 ms, 1.495 ms]	-
appsec	2.523 ms [2.469 ms, 2.576 ms]	1.039 ms (70.0%)
iast	2.207 ms [2.145 ms, 2.27 ms]	723.665 µs (48.8%)
iast_GLOBAL	2.254 ms [2.191 ms, 2.317 ms]	770.358 µs (51.9%)
profiling	2.053 ms [2.002 ms, 2.103 ms]	568.989 µs (38.4%)
tracing	2.028 ms [1.978 ms, 2.077 ms]	543.971 µs (36.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~36b62dd6a7, baseline=1.54.0-SNAPSHOT~1052f4935d
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.76 s) : 14760000, 14760000
.   : milestone, 14760000,
appsec (15.082 s) : 15082000, 15082000
.   : milestone, 15082000,
iast (18.466 s) : 18466000, 18466000
.   : milestone, 18466000,
iast_GLOBAL (17.807 s) : 17807000, 17807000
.   : milestone, 17807000,
profiling (16.024 s) : 16024000, 16024000
.   : milestone, 16024000,
tracing (14.832 s) : 14832000, 14832000
.   : milestone, 14832000,
section candidate
no_agent (14.972 s) : 14972000, 14972000
.   : milestone, 14972000,
appsec (14.883 s) : 14883000, 14883000
.   : milestone, 14883000,
iast (18.542 s) : 18542000, 18542000
.   : milestone, 18542000,
iast_GLOBAL (18.432 s) : 18432000, 18432000
.   : milestone, 18432000,
profiling (15.263 s) : 15263000, 15263000
.   : milestone, 15263000,
tracing (15.04 s) : 15040000, 15040000
.   : milestone, 15040000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.76 s [14.76 s, 14.76 s]	-
appsec	15.082 s [15.082 s, 15.082 s]	322.0 ms (2.2%)
iast	18.466 s [18.466 s, 18.466 s]	3.706 s (25.1%)
iast_GLOBAL	17.807 s [17.807 s, 17.807 s]	3.047 s (20.6%)
profiling	16.024 s [16.024 s, 16.024 s]	1.264 s (8.6%)
tracing	14.832 s [14.832 s, 14.832 s]	72.0 ms (0.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.972 s [14.972 s, 14.972 s]	-
appsec	14.883 s [14.883 s, 14.883 s]	-89.0 ms (-0.6%)
iast	18.542 s [18.542 s, 18.542 s]	3.57 s (23.8%)
iast_GLOBAL	18.432 s [18.432 s, 18.432 s]	3.46 s (23.1%)
profiling	15.263 s [15.263 s, 15.263 s]	291.0 ms (1.9%)
tracing	15.04 s [15.04 s, 15.04 s]	68.0 ms (0.5%)

dd-trace-core/src/main/java/datadog/trace/common/metrics/ConflatingMetricsAggregator.java

amarziali · 2025-09-17T12:01:26Z

dd-trace-core/src/main/java/datadog/trace/common/metrics/ConflatingMetricsAggregator.java

  }

-  private boolean publish(CoreSpan<?> span, boolean isTopLevel) {
+  private boolean publish(CoreSpan<?> span, boolean isTopLevel, DDAgentFeaturesDiscovery features) {


I changed signatures to avoid accessing that volatile things multiple time when a single trace is published

bric3

Pre-approving, some suggestions though

dd-trace-core/src/main/java/datadog/trace/common/metrics/ConflatingMetricsAggregator.java

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java

bric3

Thanks for this improvement

PerfectSlayer

Looking good. Thanks for the fix

…9548)" This reverts commit 561eb5e.

* Revert "Lazily obtain feature discovery when starting client metrics (#9548)" This reverts commit 561eb5e. * Defer metrics aggregator classloading to save startup time

amarziali requested a review from a team as a code owner September 16, 2025 14:13

amarziali added the type: bug Bug report and fix label Sep 16, 2025

amarziali requested a review from dougqh September 16, 2025 14:13

amarziali added the comp: metrics Metrics label Sep 16, 2025

amarziali requested review from mcculls, bric3 and PerfectSlayer September 16, 2025 14:13

amarziali force-pushed the andrea.marziali/metrics-lazy branch from 83c6846 to eff49c7 Compare September 16, 2025 14:47

amarziali marked this pull request as draft September 16, 2025 15:26

amarziali force-pushed the andrea.marziali/metrics-lazy branch 3 times, most recently from ae02fdb to 09fd601 Compare September 17, 2025 10:05

amarziali marked this pull request as ready for review September 17, 2025 11:56

amarziali commented Sep 17, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/common/metrics/ConflatingMetricsAggregator.java Outdated Show resolved Hide resolved

amarziali commented Sep 17, 2025

View reviewed changes

bric3 approved these changes Sep 18, 2025

View reviewed changes

amarziali added 3 commits September 19, 2025 08:57

Lazily obtain feature discovery when starting client metrics

33a9126

Add volatile to features

059088c

Defer discovery as much as possible

2b1c379

amarziali force-pushed the andrea.marziali/metrics-lazy branch from 09fd601 to 8eb7096 Compare September 19, 2025 07:28

review and simplification

36b62dd

amarziali force-pushed the andrea.marziali/metrics-lazy branch from 8eb7096 to 36b62dd Compare September 19, 2025 07:35

bric3 approved these changes Sep 19, 2025

View reviewed changes

PerfectSlayer approved these changes Sep 19, 2025

View reviewed changes

amarziali enabled auto-merge (squash) September 19, 2025 08:26

amarziali merged commit 561eb5e into master Sep 19, 2025
504 checks passed

amarziali deleted the andrea.marziali/metrics-lazy branch September 19, 2025 08:30

github-actions bot added this to the 1.54.0 milestone Sep 19, 2025

amarziali added a commit that referenced this pull request Sep 23, 2025

Revert "Lazily obtain feature discovery when starting client metrics (#…

cb4b55d

…9548)" This reverts commit 561eb5e.

amarziali mentioned this pull request Sep 23, 2025

Defer metrics aggregator classloading to save startup time #9590

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Lazily obtain feature discovery when starting client metrics #9548

Lazily obtain feature discovery when starting client metrics #9548

Uh oh!

amarziali commented Sep 16, 2025 •

edited

Loading

Uh oh!

datadog-official bot commented Sep 16, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Sep 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

amarziali Sep 17, 2025

Uh oh!

bric3 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bric3 left a comment

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Uh oh!

Lazily obtain feature discovery when starting client metrics #9548

Lazily obtain feature discovery when starting client metrics #9548

Uh oh!

Conversation

amarziali commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official bot commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

amarziali Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

bric3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bric3 left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

amarziali commented Sep 16, 2025 •

edited

Loading

datadog-official bot commented Sep 16, 2025 •

edited

Loading

pr-commenter bot commented Sep 16, 2025 •

edited

Loading