🍒 9935 - set unified service tags on llm obs span start #9988

gary-huang · 2025-11-17T19:21:50Z

Backport #9935 to release/v1.55.x

(cherry picked from commit cd26b9a)

(cherry picked from commit 859ad01)

(cherry picked from commit 4569fd7)

datadog-official · 2025-11-17T19:48:15Z

🎯 Code Coverage
• Patch Coverage: 25.00%
• Total Coverage: 59.71% (-0.01%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c1d0d79 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

pr-commenter · 2025-11-17T20:14:10Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary.huang/backport-pr-9935
git_commit_date	1763406135	1763407306
git_commit_sha	`952c455`	`c1d0d79`
release_version	1.56.0-SNAPSHOT~952c455dc0	1.56.0-SNAPSHOT~c1d0d7910e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1763409377	1763409377
ci_job_id	1240103609	1240103609
ci_pipeline_id	82784026	82784026
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-2t5mmcjp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-2t5mmcjp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 11 performance improvements and 11 performance regressions! Performance is the same for 30 metrics, 13 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:iast:Agent.start	better [-86.902ms; -33.132ms] or [-7.013%; -2.674%]	1.179s	1.239s
scenario:startup:insecure-bank:iast:Debugger	better [-54.874ms; -53.176ms] or [-91.164%; -88.343%]	6.168ms	60.193ms
scenario:startup:insecure-bank:iast:Telemetry	worse [+0.671ms; +1.412ms] or [+8.777%; +18.468%]	8.688ms	7.647ms
scenario:startup:insecure-bank:iast:Flare Poller	worse [+623.501µs; +928.228µs] or [+18.107%; +26.956%]	4.219ms	3.443ms
scenario:startup:insecure-bank:tracing:Agent.start	better [-74.827ms; -66.151ms] or [-6.796%; -6.008%]	1.031s	1.101s
scenario:startup:insecure-bank:tracing:Debugger	better [-58.216ms; -56.932ms] or [-91.058%; -89.051%]	6.358ms	63.933ms
scenario:startup:insecure-bank:tracing:Remote Config	worse [+31.794µs; +71.099µs] or [+5.022%; +11.232%]	684.477µs	633.030µs
scenario:startup:petclinic:appsec:Agent.start	better [-79.323ms; -69.786ms] or [-6.199%; -5.454%]	1.205s	1.280s
scenario:startup:petclinic:appsec:Debugger	better [-55.986ms; -54.327ms] or [-91.665%; -88.949%]	5.920ms	61.076ms
scenario:startup:petclinic:iast:Agent.start	better [-79.963ms; -70.876ms] or [-6.457%; -5.723%]	1.163s	1.238s
scenario:startup:petclinic:iast:Debugger	better [-55.108ms; -54.248ms] or [-90.620%; -89.206%]	6.134ms	60.812ms
scenario:startup:petclinic:iast:Remote Config	worse [+37.490µs; +70.229µs] or [+6.966%; +13.050%]	592.024µs	538.164µs
scenario:startup:petclinic:iast:Telemetry	worse [+558.454µs; +836.196µs] or [+7.239%; +10.839%]	8.412ms	7.715ms
scenario:startup:petclinic:iast:Flare Poller	worse [+531.851µs; +758.052µs] or [+15.109%; +21.535%]	4.165ms	3.520ms
scenario:startup:petclinic:profiling:ProfilingAgent	worse [+10.984ms; +15.662ms] or [+11.250%; +16.041%]	110.958ms	97.635ms
scenario:startup:petclinic:profiling:Debugger	better [-53.122ms; -48.203ms] or [-84.159%; -76.366%]	12.459ms	63.121ms
scenario:startup:petclinic:profiling:Remote Config	worse [+27.716µs; +76.171µs] or [+4.252%; +11.686%]	703.768µs	651.824µs
scenario:startup:petclinic:profiling:Flare Poller	worse [+280.089µs; +553.233µs] or [+7.396%; +14.608%]	4.204ms	3.787ms
scenario:startup:petclinic:profiling:Profiling	worse [+11.097ms; +15.742ms] or [+11.298%; +16.026%]	111.645ms	98.226ms
scenario:startup:petclinic:tracing:Agent.start	better [-72.159ms; -63.106ms] or [-6.570%; -5.746%]	1.031s	1.098s
scenario:startup:petclinic:tracing:Debugger	better [-58.472ms; -57.033ms] or [-91.128%; -88.885%]	6.413ms	64.165ms
scenario:startup:petclinic:tracing:Remote Config	worse [+29.767µs; +57.826µs] or [+4.748%; +9.224%]	670.675µs	626.879µs

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.098 s) : 0, 1098328
Total [baseline] (10.765 s) : 0, 10764840
Agent [candidate] (1.031 s) : 0, 1030696
Total [candidate] (10.87 s) : 0, 10870058
section appsec
Agent [baseline] (1.28 s) : 0, 1279550
Total [baseline] (11.089 s) : 0, 11088723
Agent [candidate] (1.205 s) : 0, 1204995
Total [candidate] (10.983 s) : 0, 10983418
section iast
Agent [baseline] (1.238 s) : 0, 1238473
Total [baseline] (11.232 s) : 0, 11232265
Agent [candidate] (1.163 s) : 0, 1163054
Total [candidate] (11.168 s) : 0, 11167964
section profiling
Agent [baseline] (1.229 s) : 0, 1229227
Total [baseline] (11.107 s) : 0, 11107444
Agent [candidate] (1.192 s) : 0, 1191662
Total [candidate] (10.964 s) : 0, 10963540

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.098 s	-
Agent	appsec	1.28 s	181.221 ms (16.5%)
Agent	iast	1.238 s	140.145 ms (12.8%)
Agent	profiling	1.229 s	130.899 ms (11.9%)
Total	tracing	10.765 s	-
Total	appsec	11.089 s	323.883 ms (3.0%)
Total	iast	11.232 s	467.425 ms (4.3%)
Total	profiling	11.107 s	342.604 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	appsec	1.205 s	174.299 ms (16.9%)
Agent	iast	1.163 s	132.358 ms (12.8%)
Agent	profiling	1.192 s	160.967 ms (15.6%)
Total	tracing	10.87 s	-
Total	appsec	10.983 s	113.36 ms (1.0%)
Total	iast	11.168 s	297.906 ms (2.7%)
Total	profiling	10.964 s	93.481 ms (0.9%)

gantt
    title petclinic - break down per module: candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (705.139 ms) : 0, 705139
BytebuddyAgent [candidate] (702.448 ms) : 0, 702448
GlobalTracer [baseline] (248.023 ms) : 0, 248023
GlobalTracer [candidate] (245.155 ms) : 0, 245155
AppSec [baseline] (32.219 ms) : 0, 32219
AppSec [candidate] (32.319 ms) : 0, 32319
Debugger [baseline] (64.165 ms) : 0, 64165
Debugger [candidate] (6.413 ms) : 0, 6413
Remote Config [baseline] (626.879 µs) : 0, 627
Remote Config [candidate] (670.675 µs) : 0, 671
Telemetry [baseline] (8.116 ms) : 0, 8116
Telemetry [candidate] (14.365 ms) : 0, 14365
Flare Poller [baseline] (3.716 ms) : 0, 3716
Flare Poller [candidate] (6.564 ms) : 0, 6564
section appsec
crashtracking [baseline] (1.491 ms) : 0, 1491
crashtracking [candidate] (1.468 ms) : 0, 1468
BytebuddyAgent [baseline] (729.891 ms) : 0, 729891
BytebuddyAgent [candidate] (727.322 ms) : 0, 727322
GlobalTracer [baseline] (240.475 ms) : 0, 240475
GlobalTracer [candidate] (236.404 ms) : 0, 236404
AppSec [baseline] (173.992 ms) : 0, 173992
AppSec [candidate] (174.535 ms) : 0, 174535
Debugger [baseline] (61.076 ms) : 0, 61076
Debugger [candidate] (5.92 ms) : 0, 5920
Remote Config [baseline] (665.038 µs) : 0, 665
Remote Config [candidate] (622.203 µs) : 0, 622
Telemetry [baseline] (8.28 ms) : 0, 8280
Telemetry [candidate] (8.419 ms) : 0, 8419
Flare Poller [baseline] (3.892 ms) : 0, 3892
Flare Poller [candidate] (3.889 ms) : 0, 3889
IAST [baseline] (24.599 ms) : 0, 24599
IAST [candidate] (25.204 ms) : 0, 25204
section iast
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.489 ms) : 0, 1489
BytebuddyAgent [baseline] (829.778 ms) : 0, 829778
BytebuddyAgent [candidate] (825.176 ms) : 0, 825176
GlobalTracer [baseline] (238.203 ms) : 0, 238203
GlobalTracer [candidate] (234.548 ms) : 0, 234548
AppSec [baseline] (33.186 ms) : 0, 33186
AppSec [candidate] (28.761 ms) : 0, 28761
Debugger [baseline] (60.812 ms) : 0, 60812
Debugger [candidate] (6.134 ms) : 0, 6134
Remote Config [baseline] (538.164 µs) : 0, 538
Remote Config [candidate] (592.024 µs) : 0, 592
Telemetry [baseline] (7.715 ms) : 0, 7715
Telemetry [candidate] (8.412 ms) : 0, 8412
Flare Poller [baseline] (3.52 ms) : 0, 3520
Flare Poller [candidate] (4.165 ms) : 0, 4165
IAST [baseline] (28.183 ms) : 0, 28183
IAST [candidate] (32.455 ms) : 0, 32455
section profiling
ProfilingAgent [baseline] (97.635 ms) : 0, 97635
ProfilingAgent [candidate] (110.958 ms) : 0, 110958
crashtracking [baseline] (1.429 ms) : 0, 1429
crashtracking [candidate] (1.48 ms) : 0, 1480
BytebuddyAgent [baseline] (730.313 ms) : 0, 730313
BytebuddyAgent [candidate] (738.661 ms) : 0, 738661
GlobalTracer [baseline] (222.325 ms) : 0, 222325
GlobalTracer [candidate] (223.295 ms) : 0, 223295
AppSec [baseline] (32.175 ms) : 0, 32175
AppSec [candidate] (33.814 ms) : 0, 33814
Debugger [baseline] (63.121 ms) : 0, 63121
Debugger [candidate] (12.459 ms) : 0, 12459
Remote Config [baseline] (651.824 µs) : 0, 652
Remote Config [candidate] (703.768 µs) : 0, 704
Telemetry [baseline] (8.061 ms) : 0, 8061
Telemetry [candidate] (9.776 ms) : 0, 9776
Flare Poller [baseline] (3.787 ms) : 0, 3787
Flare Poller [candidate] (4.204 ms) : 0, 4204
Profiling [baseline] (98.226 ms) : 0, 98226
Profiling [candidate] (111.645 ms) : 0, 111645

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.101 s) : 0, 1101064
Total [baseline] (8.847 s) : 0, 8846519
Agent [candidate] (1.031 s) : 0, 1030576
Total [candidate] (8.661 s) : 0, 8660992
section iast
Agent [baseline] (1.239 s) : 0, 1239160
Total [baseline] (9.58 s) : 0, 9579612
Agent [candidate] (1.179 s) : 0, 1179143
Total [candidate] (9.398 s) : 0, 9397815

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.101 s	-
Agent	iast	1.239 s	138.096 ms (12.5%)
Total	tracing	8.847 s	-
Total	iast	9.58 s	733.094 ms (8.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	iast	1.179 s	148.568 ms (14.4%)
Total	tracing	8.661 s	-
Total	iast	9.398 s	736.823 ms (8.5%)

gantt
    title insecure-bank - break down per module: candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.611 ms) : 0, 1611
BytebuddyAgent [baseline] (706.97 ms) : 0, 706970
BytebuddyAgent [candidate] (702.512 ms) : 0, 702512
GlobalTracer [baseline] (248.731 ms) : 0, 248731
GlobalTracer [candidate] (245.064 ms) : 0, 245064
AppSec [baseline] (32.418 ms) : 0, 32418
AppSec [candidate] (32.358 ms) : 0, 32358
Debugger [baseline] (63.933 ms) : 0, 63933
Debugger [candidate] (6.358 ms) : 0, 6358
Remote Config [baseline] (633.03 µs) : 0, 633
Remote Config [candidate] (684.477 µs) : 0, 684
Telemetry [baseline] (8.266 ms) : 0, 8266
Telemetry [candidate] (15.101 ms) : 0, 15101
Flare Poller [baseline] (3.748 ms) : 0, 3748
Flare Poller [candidate] (5.727 ms) : 0, 5727
section iast
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.491 ms) : 0, 1491
BytebuddyAgent [baseline] (830.796 ms) : 0, 830796
BytebuddyAgent [candidate] (837.619 ms) : 0, 837619
GlobalTracer [baseline] (238.654 ms) : 0, 238654
GlobalTracer [candidate] (236.234 ms) : 0, 236234
IAST [baseline] (29.387 ms) : 0, 29387
IAST [candidate] (31.441 ms) : 0, 31441
AppSec [baseline] (32.083 ms) : 0, 32083
AppSec [candidate] (30.984 ms) : 0, 30984
Debugger [baseline] (60.193 ms) : 0, 60193
Debugger [candidate] (6.168 ms) : 0, 6168
Remote Config [baseline] (557.808 µs) : 0, 558
Remote Config [candidate] (625.848 µs) : 0, 626
Telemetry [baseline] (7.647 ms) : 0, 7647
Telemetry [candidate] (8.688 ms) : 0, 8688
Flare Poller [baseline] (3.443 ms) : 0, 3443
Flare Poller [candidate] (4.219 ms) : 0, 4219

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary.huang/backport-pr-9935
git_commit_date	1763406135	1763407306
git_commit_sha	`952c455`	`c1d0d79`
release_version	1.56.0-SNAPSHOT~952c455dc0	1.56.0-SNAPSHOT~c1d0d7910e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1763409903	1763409903
ci_job_id	1240103611	1240103611
ci_pipeline_id	82784026	82784026
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-9zrxnec8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-9zrxnec8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 4 performance improvements and 2 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	better [-154.371µs; -39.409µs] or [-9.680%; -2.471%]	unstable [-1112.031µs; -201.125µs] or [-23.680%; -4.283%]	unstable [-53.417op/s; +486.917op/s] or [-2.340%; +21.331%]	1.498ms	4.039ms	2499.469op/s	1.595ms	4.696ms	2282.719op/s
scenario:load:insecure-bank:profiling:high_load	better [-216.553µs; -98.730µs] or [-12.616%; -5.752%]	unstable [-1332.703µs; -442.630µs] or [-26.206%; -8.704%]	unstable [+13.496op/s; +526.317op/s] or [+0.638%; +24.871%]	1.559ms	4.198ms	2386.125op/s	1.717ms	5.086ms	2116.219op/s
scenario:load:insecure-bank:iast:high_load	better [-211.882µs; -132.142µs] or [-8.525%; -5.317%]	unsure [-501.942µs; -113.813µs] or [-6.878%; -1.560%]	unstable [-57.801op/s; +229.176op/s] or [-4.046%; +16.041%]	2.313ms	6.990ms	1514.344op/s	2.486ms	7.297ms	1428.656op/s
scenario:load:petclinic:no_agent:high_load	better [-1.838ms; -0.429ms] or [-10.350%; -2.413%]	unstable [-2.884ms; +0.254ms] or [-9.630%; +0.847%]	unstable [-10.293op/s; +43.168op/s] or [-4.030%; +16.900%]	16.629ms	28.636ms	271.875op/s	17.762ms	29.951ms	255.438op/s
scenario:load:petclinic:tracing:high_load	worse [+1.677ms; +2.115ms] or [+9.865%; +12.445%]	worse [+1.442ms; +2.760ms] or [+5.120%; +9.799%]	unstable [-47.641op/s; +0.141op/s] or [-17.822%; +0.053%]	18.891ms	30.269ms	243.562op/s	16.995ms	28.167ms	267.312op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.172 ms) : 1160, 1183
.   : milestone, 1172,
iast (3.202 ms) : 3160, 3245
.   : milestone, 3202,
iast_FULL (5.628 ms) : 5572, 5684
.   : milestone, 5628,
iast_GLOBAL (3.543 ms) : 3490, 3596
.   : milestone, 3543,
profiling (2.138 ms) : 2118, 2158
.   : milestone, 2138,
tracing (1.977 ms) : 1959, 1994
.   : milestone, 1977,
section candidate
no_agent (1.17 ms) : 1158, 1181
.   : milestone, 1170,
iast (3.017 ms) : 2977, 3057
.   : milestone, 3017,
iast_FULL (5.699 ms) : 5642, 5756
.   : milestone, 5699,
iast_GLOBAL (3.678 ms) : 3624, 3733
.   : milestone, 3678,
profiling (1.888 ms) : 1871, 1905
.   : milestone, 1888,
tracing (1.799 ms) : 1784, 1814
.   : milestone, 1799,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.172 ms [1.16 ms, 1.183 ms]	-
iast	3.202 ms [3.16 ms, 3.245 ms]	2.031 ms (173.3%)
iast_FULL	5.628 ms [5.572 ms, 5.684 ms]	4.456 ms (380.3%)
iast_GLOBAL	3.543 ms [3.49 ms, 3.596 ms]	2.372 ms (202.4%)
profiling	2.138 ms [2.118 ms, 2.158 ms]	966.233 µs (82.5%)
tracing	1.977 ms [1.959 ms, 1.994 ms]	804.787 µs (68.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.17 ms [1.158 ms, 1.181 ms]	-
iast	3.017 ms [2.977 ms, 3.057 ms]	1.848 ms (158.0%)
iast_FULL	5.699 ms [5.642 ms, 5.756 ms]	4.529 ms (387.3%)
iast_GLOBAL	3.678 ms [3.624 ms, 3.733 ms]	2.509 ms (214.5%)
profiling	1.888 ms [1.871 ms, 1.905 ms]	718.445 µs (61.4%)
tracing	1.799 ms [1.784 ms, 1.814 ms]	629.365 µs (53.8%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.269 ms) : 18083, 18455
.   : milestone, 18269,
appsec (18.774 ms) : 18585, 18964
.   : milestone, 18774,
code_origins (17.912 ms) : 17732, 18092
.   : milestone, 17912,
iast (17.793 ms) : 17615, 17970
.   : milestone, 17793,
profiling (19.522 ms) : 19319, 19724
.   : milestone, 19522,
tracing (17.455 ms) : 17281, 17630
.   : milestone, 17455,
section candidate
no_agent (17.161 ms) : 16987, 17336
.   : milestone, 17161,
appsec (18.438 ms) : 18250, 18627
.   : milestone, 18438,
code_origins (18.184 ms) : 17999, 18369
.   : milestone, 18184,
iast (18.414 ms) : 18227, 18600
.   : milestone, 18414,
profiling (20.472 ms) : 20263, 20681
.   : milestone, 20472,
tracing (19.162 ms) : 18970, 19355
.   : milestone, 19162,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.269 ms [18.083 ms, 18.455 ms]	-
appsec	18.774 ms [18.585 ms, 18.964 ms]	505.528 µs (2.8%)
code_origins	17.912 ms [17.732 ms, 18.092 ms]	-356.796 µs (-2.0%)
iast	17.793 ms [17.615 ms, 17.97 ms]	-476.044 µs (-2.6%)
profiling	19.522 ms [19.319 ms, 19.724 ms]	1.253 ms (6.9%)
tracing	17.455 ms [17.281 ms, 17.63 ms]	-813.308 µs (-4.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.161 ms [16.987 ms, 17.336 ms]	-
appsec	18.438 ms [18.25 ms, 18.627 ms]	1.277 ms (7.4%)
code_origins	18.184 ms [17.999 ms, 18.369 ms]	1.023 ms (6.0%)
iast	18.414 ms [18.227 ms, 18.6 ms]	1.252 ms (7.3%)
profiling	20.472 ms [20.263 ms, 20.681 ms]	3.311 ms (19.3%)
tracing	19.162 ms [18.97 ms, 19.355 ms]	2.001 ms (11.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary.huang/backport-pr-9935
git_commit_date	1763406135	1763407306
git_commit_sha	`952c455`	`c1d0d79`
release_version	1.56.0-SNAPSHOT~952c455dc0	1.56.0-SNAPSHOT~c1d0d7910e

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1763409549	1763409549
ci_job_id	1240103614	1240103614
ci_pipeline_id	82784026	82784026
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-1r2hpl49 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-1r2hpl49 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.048 s) : 15048000, 15048000
.   : milestone, 15048000,
appsec (14.613 s) : 14613000, 14613000
.   : milestone, 14613000,
iast (18.183 s) : 18183000, 18183000
.   : milestone, 18183000,
iast_GLOBAL (17.835 s) : 17835000, 17835000
.   : milestone, 17835000,
profiling (15.053 s) : 15053000, 15053000
.   : milestone, 15053000,
tracing (14.554 s) : 14554000, 14554000
.   : milestone, 14554000,
section candidate
no_agent (15.032 s) : 15032000, 15032000
.   : milestone, 15032000,
appsec (15.11 s) : 15110000, 15110000
.   : milestone, 15110000,
iast (18.544 s) : 18544000, 18544000
.   : milestone, 18544000,
iast_GLOBAL (17.674 s) : 17674000, 17674000
.   : milestone, 17674000,
profiling (15.407 s) : 15407000, 15407000
.   : milestone, 15407000,
tracing (15.449 s) : 15449000, 15449000
.   : milestone, 15449000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.048 s [15.048 s, 15.048 s]	-
appsec	14.613 s [14.613 s, 14.613 s]	-435.0 ms (-2.9%)
iast	18.183 s [18.183 s, 18.183 s]	3.135 s (20.8%)
iast_GLOBAL	17.835 s [17.835 s, 17.835 s]	2.787 s (18.5%)
profiling	15.053 s [15.053 s, 15.053 s]	5.0 ms (0.0%)
tracing	14.554 s [14.554 s, 14.554 s]	-494.0 ms (-3.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.032 s [15.032 s, 15.032 s]	-
appsec	15.11 s [15.11 s, 15.11 s]	78.0 ms (0.5%)
iast	18.544 s [18.544 s, 18.544 s]	3.512 s (23.4%)
iast_GLOBAL	17.674 s [17.674 s, 17.674 s]	2.642 s (17.6%)
profiling	15.407 s [15.407 s, 15.407 s]	375.0 ms (2.5%)
tracing	15.449 s [15.449 s, 15.449 s]	417.0 ms (2.8%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.56.0-SNAPSHOT~c1d0d7910e, baseline=1.56.0-SNAPSHOT~952c455dc0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.468 ms) : 1456, 1479
.   : milestone, 1468,
appsec (2.442 ms) : 2391, 2492
.   : milestone, 2442,
iast (2.206 ms) : 2142, 2270
.   : milestone, 2206,
iast_GLOBAL (2.24 ms) : 2176, 2304
.   : milestone, 2240,
profiling (2.047 ms) : 1996, 2098
.   : milestone, 2047,
tracing (2.016 ms) : 1966, 2065
.   : milestone, 2016,
section candidate
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.443 ms) : 2392, 2494
.   : milestone, 2443,
iast (2.204 ms) : 2140, 2267
.   : milestone, 2204,
iast_GLOBAL (2.248 ms) : 2184, 2312
.   : milestone, 2248,
profiling (2.052 ms) : 2000, 2103
.   : milestone, 2052,
tracing (2.015 ms) : 1966, 2064
.   : milestone, 2015,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.468 ms [1.456 ms, 1.479 ms]	-
appsec	2.442 ms [2.391 ms, 2.492 ms]	974.06 µs (66.4%)
iast	2.206 ms [2.142 ms, 2.27 ms]	738.751 µs (50.3%)
iast_GLOBAL	2.24 ms [2.176 ms, 2.304 ms]	772.376 µs (52.6%)
profiling	2.047 ms [1.996 ms, 2.098 ms]	579.888 µs (39.5%)
tracing	2.016 ms [1.966 ms, 2.065 ms]	548.313 µs (37.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.482 ms]	-
appsec	2.443 ms [2.392 ms, 2.494 ms]	972.435 µs (66.1%)
iast	2.204 ms [2.14 ms, 2.267 ms]	733.225 µs (49.9%)
iast_GLOBAL	2.248 ms [2.184 ms, 2.312 ms]	777.63 µs (52.9%)
profiling	2.052 ms [2.0 ms, 2.103 ms]	581.23 µs (39.5%)
tracing	2.015 ms [1.966 ms, 2.064 ms]	544.509 µs (37.0%)

PerfectSlayer · 2025-11-21T12:00:06Z

Is it still need after release 1.56.0?

gary-huang added 3 commits November 17, 2025 14:21

set unified service tags on llm obs span start

e210c54

(cherry picked from commit cd26b9a)

tests and clarify comment

0086e19

(cherry picked from commit 859ad01)

dont need asserts in then

c1d0d79

(cherry picked from commit 4569fd7)

gary-huang added type: bug Bug report and fix comp: mlobs ML Observability (LLMObs) labels Nov 17, 2025

gary-huang requested a review from a team as a code owner November 17, 2025 19:21

gary-huang added type: bug Bug report and fix comp: mlobs ML Observability (LLMObs) labels Nov 17, 2025

ygree approved these changes Nov 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🍒 9935 - set unified service tags on llm obs span start #9988

🍒 9935 - set unified service tags on llm obs span start #9988

gary-huang commented Nov 17, 2025

Uh oh!

datadog-official bot commented Nov 17, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Nov 17, 2025

Uh oh!

PerfectSlayer commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

🍒 9935 - set unified service tags on llm obs span start #9988

Are you sure you want to change the base?

🍒 9935 - set unified service tags on llm obs span start #9988

Conversation

gary-huang commented Nov 17, 2025

Uh oh!

datadog-official bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Nov 17, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

PerfectSlayer commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

datadog-official bot commented Nov 17, 2025 •

edited

Loading