Skip to content

Make test span serialization idempotent #9456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 19, 2025
Merged

Make test span serialization idempotent #9456

merged 2 commits into from
Nov 19, 2025

Conversation

@daniel-mohedano
Copy link
Contributor

@daniel-mohedano daniel-mohedano commented Sep 3, 2025 •
edited
Loading

What Does This Do

  • Makes datadog.trace.civisibility.writer.ddintake.CiTestCycleMapperV1#map idempotent by not removing the top level tags test_session_id, test_module_id, test_suite_id and itr_correlation_id from the test span on mapping.
  • An additional unit test has been created to validate this behavior.

Motivation

  • If a buffer overflow is encountered while mapping a test span in datadog.communication.serialization.msgpack.MsgPackWriter#format, the mapping will be attempted again after flushing the buffer. Given that the test span was being modified inside the method, it meant that successive calls on the same span would yield different results. This materialized as test spans being sent to the backend without the ID tags.

Contributor Checklist

Jira ticket: SDTEST-2590

@datadog-datadog-prod-us1
Copy link
Contributor

datadog-datadog-prod-us1 bot commented Sep 3, 2025 •
edited by datadog-official bot
Loading

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 100.00% (+40.32%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ba3963c | Docs | Was this helpful? Give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Sep 3, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testcycle-mapper-fix
git_commit_date 1761646523 1761646587
git_commit_sha 2413c50 ba3963c
release_version 1.55.0-SNAPSHOT~2413c501d7 1.53.0-SNAPSHOT~ba3963c2e1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1761648634 1761648634
ci_job_id 1201498890 1201498890
ci_pipeline_id 80467677 80467677
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-7kersxn7 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-7kersxn7 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1036267
Total [baseline] (8.65 s) : 0, 8649663
Agent [candidate] (1.032 s) : 0, 1032037
Total [candidate] (8.669 s) : 0, 8669173
section iast
Agent [baseline] (1.17 s) : 0, 1169836
Total [baseline] (9.43 s) : 0, 9429656
Agent [candidate] (1.162 s) : 0, 1161835
Total [candidate] (9.336 s) : 0, 9336116
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.036 s -
Agent iast 1.17 s 133.569 ms (12.9%)
Total tracing 8.65 s -
Total iast 9.43 s 779.993 ms (9.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.032 s -
Agent iast 1.162 s 129.798 ms (12.6%)
Total tracing 8.669 s -
Total iast 9.336 s 666.942 ms (7.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.478 ms) : 0, 1478
crashtracking [candidate] (1.476 ms) : 0, 1476
BytebuddyAgent [baseline] (706.45 ms) : 0, 706450
BytebuddyAgent [candidate] (704.077 ms) : 0, 704077
GlobalTracer [baseline] (246.063 ms) : 0, 246063
GlobalTracer [candidate] (245.11 ms) : 0, 245110
AppSec [baseline] (32.619 ms) : 0, 32619
AppSec [candidate] (32.47 ms) : 0, 32470
Debugger [baseline] (6.466 ms) : 0, 6466
Debugger [candidate] (6.392 ms) : 0, 6392
Remote Config [baseline] (687.883 µs) : 0, 688
Remote Config [candidate] (673.637 µs) : 0, 674
Telemetry [baseline] (12.433 ms) : 0, 12433
Telemetry [candidate] (15.737 ms) : 0, 15737
Flare Poller [baseline] (8.856 ms) : 0, 8856
Flare Poller [candidate] (5.044 ms) : 0, 5044
section iast
crashtracking [baseline] (1.484 ms) : 0, 1484
crashtracking [candidate] (1.469 ms) : 0, 1469
BytebuddyAgent [baseline] (830.458 ms) : 0, 830458
BytebuddyAgent [candidate] (824.391 ms) : 0, 824391
GlobalTracer [baseline] (235.147 ms) : 0, 235147
GlobalTracer [candidate] (233.663 ms) : 0, 233663
AppSec [baseline] (28.348 ms) : 0, 28348
AppSec [candidate] (30.896 ms) : 0, 30896
Debugger [baseline] (6.147 ms) : 0, 6147
Debugger [candidate] (6.099 ms) : 0, 6099
Remote Config [baseline] (627.448 µs) : 0, 627
Remote Config [candidate] (601.809 µs) : 0, 602
Telemetry [baseline] (8.45 ms) : 0, 8450
Telemetry [candidate] (8.53 ms) : 0, 8530
Flare Poller [baseline] (4.147 ms) : 0, 4147
Flare Poller [candidate] (4.188 ms) : 0, 4188
IAST [baseline] (33.545 ms) : 0, 33545
IAST [candidate] (30.729 ms) : 0, 30729
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.033 s) : 0, 1032649
Total [baseline] (10.768 s) : 0, 10768430
Agent [candidate] (1.032 s) : 0, 1031752
Total [candidate] (10.84 s) : 0, 10840201
section appsec
Agent [baseline] (1.204 s) : 0, 1203574
Total [baseline] (10.943 s) : 0, 10943437
Agent [candidate] (1.205 s) : 0, 1204880
Total [candidate] (11.011 s) : 0, 11010796
section iast
Agent [baseline] (1.167 s) : 0, 1166546
Total [baseline] (11.089 s) : 0, 11088751
Agent [candidate] (1.173 s) : 0, 1172624
Total [candidate] (11.095 s) : 0, 11095296
section profiling
Agent [baseline] (1.172 s) : 0, 1172226
Total [baseline] (10.885 s) : 0, 10884784
Agent [candidate] (1.18 s) : 0, 1180436
Total [candidate] (11.05 s) : 0, 11049693
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.033 s -
Agent appsec 1.204 s 170.926 ms (16.6%)
Agent iast 1.167 s 133.898 ms (13.0%)
Agent profiling 1.172 s 139.578 ms (13.5%)
Total tracing 10.768 s -
Total appsec 10.943 s 175.006 ms (1.6%)
Total iast 11.089 s 320.321 ms (3.0%)
Total profiling 10.885 s 116.354 ms (1.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.032 s -
Agent appsec 1.205 s 173.128 ms (16.8%)
Agent iast 1.173 s 140.872 ms (13.7%)
Agent profiling 1.18 s 148.684 ms (14.4%)
Total tracing 10.84 s -
Total appsec 11.011 s 170.595 ms (1.6%)
Total iast 11.095 s 255.095 ms (2.4%)
Total profiling 11.05 s 209.492 ms (1.9%) 
gantt
    title petclinic - break down per module: candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (704.181 ms) : 0, 704181
BytebuddyAgent [candidate] (703.504 ms) : 0, 703504
GlobalTracer [baseline] (245.276 ms) : 0, 245276
GlobalTracer [candidate] (245.175 ms) : 0, 245175
AppSec [baseline] (32.386 ms) : 0, 32386
AppSec [candidate] (32.418 ms) : 0, 32418
Debugger [baseline] (6.406 ms) : 0, 6406
Debugger [candidate] (6.389 ms) : 0, 6389
Remote Config [baseline] (678.155 µs) : 0, 678
Remote Config [candidate] (677.46 µs) : 0, 677
Telemetry [baseline] (15.938 ms) : 0, 15938
Telemetry [candidate] (15.943 ms) : 0, 15943
Flare Poller [baseline] (5.074 ms) : 0, 5074
Flare Poller [candidate] (4.968 ms) : 0, 4968
section appsec
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.457 ms) : 0, 1457
BytebuddyAgent [baseline] (726.922 ms) : 0, 726922
BytebuddyAgent [candidate] (727.254 ms) : 0, 727254
GlobalTracer [baseline] (236.256 ms) : 0, 236256
GlobalTracer [candidate] (236.269 ms) : 0, 236269
AppSec [baseline] (174.033 ms) : 0, 174033
AppSec [candidate] (174.636 ms) : 0, 174636
Debugger [baseline] (5.887 ms) : 0, 5887
Debugger [candidate] (5.94 ms) : 0, 5940
Remote Config [baseline] (623.373 µs) : 0, 623
Remote Config [candidate] (628.512 µs) : 0, 629
Telemetry [baseline] (8.376 ms) : 0, 8376
Telemetry [candidate] (8.483 ms) : 0, 8483
Flare Poller [baseline] (3.889 ms) : 0, 3889
Flare Poller [candidate] (3.923 ms) : 0, 3923
IAST [baseline] (24.995 ms) : 0, 24995
IAST [candidate] (25.154 ms) : 0, 25154
section iast
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.476 ms) : 0, 1476
BytebuddyAgent [baseline] (827.912 ms) : 0, 827912
BytebuddyAgent [candidate] (833.012 ms) : 0, 833012
GlobalTracer [baseline] (234.763 ms) : 0, 234763
GlobalTracer [candidate] (235.524 ms) : 0, 235524
AppSec [baseline] (28.954 ms) : 0, 28954
AppSec [candidate] (29.046 ms) : 0, 29046
Debugger [baseline] (6.173 ms) : 0, 6173
Debugger [candidate] (6.187 ms) : 0, 6187
Remote Config [baseline] (601.317 µs) : 0, 601
Remote Config [candidate] (588.87 µs) : 0, 589
Telemetry [baseline] (8.435 ms) : 0, 8435
Telemetry [candidate] (8.502 ms) : 0, 8502
Flare Poller [baseline] (4.119 ms) : 0, 4119
Flare Poller [candidate] (4.108 ms) : 0, 4108
IAST [baseline] (32.709 ms) : 0, 32709
IAST [candidate] (32.693 ms) : 0, 32693
section profiling
ProfilingAgent [baseline] (109.298 ms) : 0, 109298
ProfilingAgent [candidate] (110.809 ms) : 0, 110809
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (726.058 ms) : 0, 726058
BytebuddyAgent [candidate] (731.003 ms) : 0, 731003
GlobalTracer [baseline] (219.833 ms) : 0, 219833
GlobalTracer [candidate] (221.214 ms) : 0, 221214
AppSec [baseline] (32.379 ms) : 0, 32379
AppSec [candidate] (32.463 ms) : 0, 32463
Debugger [baseline] (12.281 ms) : 0, 12281
Debugger [candidate] (13.105 ms) : 0, 13105
Remote Config [baseline] (719.66 µs) : 0, 720
Remote Config [candidate] (1.491 ms) : 0, 1491
Telemetry [baseline] (10.705 ms) : 0, 10705
Telemetry [candidate] (9.07 ms) : 0, 9070
Flare Poller [baseline] (4.091 ms) : 0, 4091
Flare Poller [candidate] (4.111 ms) : 0, 4111
Profiling [baseline] (109.939 ms) : 0, 109939
Profiling [candidate] (111.468 ms) : 0, 111468
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testcycle-mapper-fix
git_commit_date 1761646523 1761646587
git_commit_sha 2413c50 ba3963c
release_version 1.55.0-SNAPSHOT~2413c501d7 1.53.0-SNAPSHOT~ba3963c2e1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1761648973 1761648973
ci_job_id 1201498891 1201498891
ci_pipeline_id 80467677 80467677
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-3-udschwxo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-3-udschwxo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:iast_FULL:high_load better
[-262.411µs; -141.748µs] or [-4.552%; -2.459%]		 unstable
[-47.867op/s; +105.804op/s] or [-5.970%; +13.195%]		 5.563ms 830.812op/s 5.765ms 801.844op/s
scenario:load:petclinic:tracing:high_load worse
[+1.071ms; +1.477ms] or [+5.889%; +8.123%]		 unstable
[-45.845op/s; +12.345op/s] or [-17.867%; +4.811%]		 19.460ms 239.844op/s 18.186ms 256.594op/s
scenario:load:petclinic:appsec:high_load worse
[+643.483µs; +1082.981µs] or [+3.305%; +5.563%]		 unstable
[-37.401op/s; +16.964op/s] or [-15.592%; +7.072%]		 20.330ms 229.656op/s 19.467ms 239.875op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.862 ms) : 18667, 19056
.   : milestone, 18862,
appsec (19.467 ms) : 19266, 19669
.   : milestone, 19467,
code_origins (19.508 ms) : 19309, 19706
.   : milestone, 19508,
iast (19.488 ms) : 19292, 19684
.   : milestone, 19488,
profiling (19.107 ms) : 18912, 19302
.   : milestone, 19107,
tracing (18.186 ms) : 18004, 18368
.   : milestone, 18186,
section candidate
no_agent (18.904 ms) : 18712, 19095
.   : milestone, 18904,
appsec (20.33 ms) : 20124, 20537
.   : milestone, 20330,
code_origins (19.25 ms) : 19060, 19441
.   : milestone, 19250,
iast (19.824 ms) : 19623, 20026
.   : milestone, 19824,
profiling (19.271 ms) : 19078, 19464
.   : milestone, 19271,
tracing (19.46 ms) : 19265, 19655
.   : milestone, 19460,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.862 ms [18.667 ms, 19.056 ms] -
appsec 19.467 ms [19.266 ms, 19.669 ms] 605.428 µs (3.2%)
code_origins 19.508 ms [19.309 ms, 19.706 ms] 645.721 µs (3.4%)
iast 19.488 ms [19.292 ms, 19.684 ms] 625.909 µs (3.3%)
profiling 19.107 ms [18.912 ms, 19.302 ms] 244.784 µs (1.3%)
tracing 18.186 ms [18.004 ms, 18.368 ms] -676.025 µs (-3.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.904 ms [18.712 ms, 19.095 ms] -
appsec 20.33 ms [20.124 ms, 20.537 ms] 1.427 ms (7.5%)
code_origins 19.25 ms [19.06 ms, 19.441 ms] 346.661 µs (1.8%)
iast 19.824 ms [19.623 ms, 20.026 ms] 920.736 µs (4.9%)
profiling 19.271 ms [19.078 ms, 19.464 ms] 367.359 µs (1.9%)
tracing 19.46 ms [19.265 ms, 19.655 ms] 556.224 µs (2.9%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.198 ms) : 1186, 1210
.   : milestone, 1198,
iast (3.234 ms) : 3193, 3275
.   : milestone, 3234,
iast_FULL (5.765 ms) : 5707, 5822
.   : milestone, 5765,
iast_GLOBAL (3.524 ms) : 3466, 3582
.   : milestone, 3524,
profiling (1.943 ms) : 1927, 1959
.   : milestone, 1943,
tracing (1.81 ms) : 1795, 1825
.   : milestone, 1810,
section candidate
no_agent (1.193 ms) : 1181, 1204
.   : milestone, 1193,
iast (3.236 ms) : 3191, 3280
.   : milestone, 3236,
iast_FULL (5.563 ms) : 5508, 5617
.   : milestone, 5563,
iast_GLOBAL (3.553 ms) : 3501, 3606
.   : milestone, 3553,
profiling (1.898 ms) : 1882, 1913
.   : milestone, 1898,
tracing (1.854 ms) : 1838, 1871
.   : milestone, 1854,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.198 ms [1.186 ms, 1.21 ms] -
iast 3.234 ms [3.193 ms, 3.275 ms] 2.036 ms (170.0%)
iast_FULL 5.765 ms [5.707 ms, 5.822 ms] 4.567 ms (381.3%)
iast_GLOBAL 3.524 ms [3.466 ms, 3.582 ms] 2.326 ms (194.2%)
profiling 1.943 ms [1.927 ms, 1.959 ms] 745.0 µs (62.2%)
tracing 1.81 ms [1.795 ms, 1.825 ms] 612.085 µs (51.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.193 ms [1.181 ms, 1.204 ms] -
iast 3.236 ms [3.191 ms, 3.28 ms] 2.043 ms (171.3%)
iast_FULL 5.563 ms [5.508 ms, 5.617 ms] 4.37 ms (366.5%)
iast_GLOBAL 3.553 ms [3.501 ms, 3.606 ms] 2.361 ms (197.9%)
profiling 1.898 ms [1.882 ms, 1.913 ms] 705.376 µs (59.2%)
tracing 1.854 ms [1.838 ms, 1.871 ms] 661.708 µs (55.5%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testcycle-mapper-fix
git_commit_date 1761646523 1761646587
git_commit_sha 2413c50 ba3963c
release_version 1.55.0-SNAPSHOT~2413c501d7 1.53.0-SNAPSHOT~ba3963c2e1
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1761648795 1761648795
ci_job_id 1201498893 1201498893
ci_pipeline_id 80467677 80467677
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-4-5jy3nxu3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-4-5jy3nxu3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.449 ms) : 2398, 2500
.   : milestone, 2449,
iast (2.199 ms) : 2135, 2262
.   : milestone, 2199,
iast_GLOBAL (2.233 ms) : 2169, 2296
.   : milestone, 2233,
profiling (2.045 ms) : 1994, 2097
.   : milestone, 2045,
tracing (2.027 ms) : 1978, 2077
.   : milestone, 2027,
section candidate
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (3.652 ms) : 3437, 3867
.   : milestone, 3652,
iast (2.205 ms) : 2142, 2269
.   : milestone, 2205,
iast_GLOBAL (2.255 ms) : 2191, 2320
.   : milestone, 2255,
profiling (2.055 ms) : 2002, 2107
.   : milestone, 2055,
tracing (2.022 ms) : 1972, 2071
.   : milestone, 2022,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.458 ms, 1.481 ms] -
appsec 2.449 ms [2.398 ms, 2.5 ms] 979.683 µs (66.7%)
iast 2.199 ms [2.135 ms, 2.262 ms] 729.348 µs (49.6%)
iast_GLOBAL 2.233 ms [2.169 ms, 2.296 ms] 763.548 µs (52.0%)
profiling 2.045 ms [1.994 ms, 2.097 ms] 575.823 µs (39.2%)
tracing 2.027 ms [1.978 ms, 2.077 ms] 558.116 µs (38.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.458 ms, 1.481 ms] -
appsec 3.652 ms [3.437 ms, 3.867 ms] 2.183 ms (148.5%)
iast 2.205 ms [2.142 ms, 2.269 ms] 735.747 µs (50.1%)
iast_GLOBAL 2.255 ms [2.191 ms, 2.32 ms] 785.91 µs (53.5%)
profiling 2.055 ms [2.002 ms, 2.107 ms] 585.165 µs (39.8%)
tracing 2.022 ms [1.972 ms, 2.071 ms] 552.301 µs (37.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~ba3963c2e1, baseline=1.55.0-SNAPSHOT~2413c501d7
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.944 s) : 14944000, 14944000
.   : milestone, 14944000,
appsec (15.024 s) : 15024000, 15024000
.   : milestone, 15024000,
iast (18.247 s) : 18247000, 18247000
.   : milestone, 18247000,
iast_GLOBAL (17.986 s) : 17986000, 17986000
.   : milestone, 17986000,
profiling (15.3 s) : 15300000, 15300000
.   : milestone, 15300000,
tracing (15.351 s) : 15351000, 15351000
.   : milestone, 15351000,
section candidate
no_agent (15.553 s) : 15553000, 15553000
.   : milestone, 15553000,
appsec (14.761 s) : 14761000, 14761000
.   : milestone, 14761000,
iast (18.523 s) : 18523000, 18523000
.   : milestone, 18523000,
iast_GLOBAL (18.129 s) : 18129000, 18129000
.   : milestone, 18129000,
profiling (15.275 s) : 15275000, 15275000
.   : milestone, 15275000,
tracing (15.328 s) : 15328000, 15328000
.   : milestone, 15328000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.944 s [14.944 s, 14.944 s] -
appsec 15.024 s [15.024 s, 15.024 s] 80.0 ms (0.5%)
iast 18.247 s [18.247 s, 18.247 s] 3.303 s (22.1%)
iast_GLOBAL 17.986 s [17.986 s, 17.986 s] 3.042 s (20.4%)
profiling 15.3 s [15.3 s, 15.3 s] 356.0 ms (2.4%)
tracing 15.351 s [15.351 s, 15.351 s] 407.0 ms (2.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.553 s [15.553 s, 15.553 s] -
appsec 14.761 s [14.761 s, 14.761 s] -792.0 ms (-5.1%)
iast 18.523 s [18.523 s, 18.523 s] 2.97 s (19.1%)
iast_GLOBAL 18.129 s [18.129 s, 18.129 s] 2.576 s (16.6%)
profiling 15.275 s [15.275 s, 15.275 s] -278.0 ms (-1.8%)
tracing 15.328 s [15.328 s, 15.328 s] -225.0 ms (-1.4%)

@daniel-mohedano daniel-mohedano added type: bug Bug report and fix comp: ci visibility Continuous Integration Visibility labels Sep 4, 2025
@daniel-mohedano daniel-mohedano changed the title fix: dont remove id tags and instead ignore in consumer Make test span serialization idempotent Sep 4, 2025
@daniel-mohedano daniel-mohedano marked this pull request as ready for review September 4, 2025 08:05
@daniel-mohedano daniel-mohedano requested a review from a team as a code owner September 4, 2025 08:05
nikita-tkachenko-datadog
nikita-tkachenko-datadog reviewed Sep 4, 2025
View reviewed changes
dd-trace-core/src/main/java/datadog/trace/civisibility/writer/ddintake/CiTestCycleMapperV1.java

@Override
public void accept(Metadata metadata) {
TagMap tags = metadata.getTags().copy();
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we try and estimate the overhead of this (with a test-environment run, for example)? Creating a copy of the tags map for every serialized span could be a big deal. I know there's an implementation of TagMap that is optimized for copying, but it isn't used by default still.

Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it will be less overhead to skip the DEFAULT_TOP_LEVEL_TAGS when iterating over the tag entries below.

Copy link
Contributor Author

@daniel-mohedano daniel-mohedano Sep 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, I'll test the two implementations on test environment to see how it impacts performance 👍

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - sbt-scalatest

Job Status: success

Scenario Overhead (%)
agent 57.01
agentEvpProxy 57.35

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - nebula-release-plugin

Job Status: success

Scenario Overhead (%)
agent 36.07
agentless 36.33
agentlessCodeCoverage 43.94
agentlessLineCoverage 73.50

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - pass4s

Job Status: success

Scenario Overhead (%)
agent 9.38
agentless 12.61
agentlessCodeCoverage 19.29

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - netflix-zuul

Job Status: success

Scenario Overhead (%)
agent 90.00
agentless 84.02
agentlessCodeCoverage 99.75
agentlessLineCoverage 112.63

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - reactive-streams-jvm

Job Status: success

Scenario Overhead (%)
agent 21.93
agentless 18.13
agentlessCodeCoverage 19.95
agentlessLineCoverage 29.74

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - sonar-kotlin

Job Status: success

Scenario Overhead (%)
agent 13.69
agentless 12.39
agentlessCodeCoverage 16.30
agentlessLineCoverage 20.51

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - sonar-java

Job Status: success

Scenario Overhead (%)
agent 28.58
agentless -2.72
agentlessCodeCoverage 57.97
agentlessLineCoverage 109.12

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - jolokia

Job Status: success

Scenario Overhead (%)
agent 92.10
agentless 88.19
agentlessCodeCoverage 97.80
agentlessLineCoverage 97.26

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - okhttp

Job Status: success

Scenario Overhead (%)
agent 17.47
agentless 15.10
agentlessCodeCoverage 16.37
agentlessLineCoverage 33.93

@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 28, 2025

Test Environment - spring_boot

Job Status: success

Scenario Overhead (%)
agent 12.33
agentless 6.22
agentlessCodeCoverage 10.07
agentlessLineCoverage 28.74

@daniel-mohedano
Copy link
Contributor Author

daniel-mohedano commented Oct 29, 2025

@nikita-tkachenko-datadog coming back to this fix, now that the test environment is in a more stable state, the performance comparison between this approach and #9462 is the following:

Copy + Remove Implementation Skip Tags Implementation Baseline p5 Baseline p95
sbt-scalatest agent 57.10 🟩 56.30 51.2 57.7
agentEvpProxy 57.35 🟩 56.23 51.2 57.7
nebula-release-plugin agent 🟩 36.07 37.11 35.7 39.5
agentless 🟩 36.33 38.25 35.0 38.7
agentlessCodeCoverage 43.94 🟩 43.51 42.7 45.4
agentlessLineCoverage 🟩 73.50 73.23 73.3 77.9
pass4s agent 🟩 9.38 11.55 4.3 14.8
agentless 🟩 12.61 13.03 5.4 13.9
agentlessCodeCoverage 19.29 🟩 15.60 10.3 19.6
netflix-zuul agent 90.00 🟩 89.64 87.8 91.4
agentless 🟩 84.02 🟩 84.02 81.1 84.4
agentlessCodeCoverage 99.75 🟩 98.49 95.1 99.0
agentlessLineCoverage 🟩 112.63 114.14 111.6 116.2
reactive-streams-jvm agent 🟩 21.93 22.12 19.2 22.5
agentless 🟩 18.13 19.41 16.4 19.6
agentlessCodeCoverage 🟩 19.95 21.16 17.7 20.8
agentlessLineCoverage 🟩 29.74 30.63 27.5 31.0
sonar-kotlin agent 🟩 13.69 15.02 11.9 15.1
agentless 🟩 12.39 13.46 9.9 14.8
agentlessCodeCoverage 🟩 16.30 17.05 13.1 17.4
agentlessLineCoverage 🟩 20.51 21.29 17.4 21.7
jolokia agent 🟡 92.10 93.01 86.1 91.4
agentless 🟡 88.19 88.41 82.7 87.8
agentlessCodeCoverage 🟡 97.80 98.23 91.4 97.0
agentlessLineCoverage 🟩 97.26 97.98 93.2 99.0
okhttp agent 17.40 🟩 16.59 14.8 18.4
agentless 🟩 15.10 16.69 14.8 17.7
agentlessCodeCoverage 🟩 16.37 20.40 18.1 21.7
agentlessLineCoverage 🟩 33.93 35.22 32.3 36.4
spring_boot agent 12.33 🟩 12.17 11.4 12.9
agentless 6.22 🟩 6.17 5.6 6.8
agentlessCodeCoverage 🟩 10.07 10.24 9.3 11.4
agentlessLineCoverage 🟩 28.74 29.60 28.1 30.4

The green markers indicate the lower overhead implementation, with the yellow markers indicating that the overhead still surpassed the p95 average. The results per implementation are for individual runs.

@daniel-mohedano daniel-mohedano merged commit 5adec51 into master Nov 19, 2025
539 checks passed
@daniel-mohedano daniel-mohedano deleted the daniel.mohedano/testcycle-mapper-fix branch November 19, 2025 12:58
@github-actions github-actions bot added this to the 1.56.0 milestone Nov 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nikita-tkachenko-datadog nikita-tkachenko-datadog nikita-tkachenko-datadog approved these changes

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility type: bug Bug report and fix

Projects

None yet

Milestone

1.56.0

Development

Successfully merging this pull request may close these issues.

3 participants

@daniel-mohedano @nikita-tkachenko-datadog