Skip to content

Fix field-injection of ForkJoinTask on Java 25 #10084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mcculls merged 3 commits into from
Dec 4, 2025
Merged

Fix field-injection of ForkJoinTask on Java 25 #10084

mcculls merged 3 commits into from
Dec 4, 2025
+10 −1

Conversation

@mcculls
Copy link
Contributor

@mcculls mcculls commented Dec 4, 2025 •
edited
Loading

What Does This Do

Turning off crash-tracking in FieldInjectionSmokeTest was hiding a bug on Java 25:

https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-java/builds/1270792914

By default crash-tracking executes a task on the dd-task-scheduler thread before any instrumentations are installed. When AgentTaskScheduler is called for the first time without an initial delay we prepare the work queue (an instance of DelayQueue), execute the task, and then block on workQueue.take().

On Java 25 calling DelayQueue.take() when the DelayQueue is empty leads to an await call without a timeout:

https://github.com/openjdk/jdk/blob/jdk-25%2B0/src/java.base/share/classes/java/util/concurrent/DelayQueue.java#L243

this in turn leads to the ForkJoinPool class being initialized:

https://github.com/openjdk/jdk/blob/jdk-25%2B0/src/java.base/share/classes/java/util/concurrent/locks/AbstractQueuedSynchronizer.java#L1751

which in turn leads to the ForkJoinTask class being loaded:

https://github.com/openjdk/jdk/blob/jdk-25%2B0/src/java.base/share/classes/java/util/concurrent/ForkJoinPool.java#L3977

and because this happens before any instrumentations are installed we lose the chance to field-inject ForkJoinTask. This results in us using the global weak map to track async context across instances of ForkJoinTask which is not memory efficient.

The simplest solution is to add a no-op, one-shot task far enough in the future when preparing the work queue. The take() call will now call awaitNanos(delay) which doesn't result in either of the ForkJoinPool or ForkJoinTask classes being initialized.

Adding this no-op, one-shot task doesn't affect startup performance.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@mcculls mcculls added type: bug Bug report and fix inst: java Core Java language instrumentation labels Dec 4, 2025
@mcculls
Preload a future no-op task to ensure workQueue.take() will use await…
a73fd0b 
… with timeout during

premain - otherwise on Java 25 it will load ForkJoinPool which in turn loads ForkJoinTask,
which then means we lose the chance to field-inject context into ForkJoinTask instances
@mcculls mcculls force-pushed the mcculls/fix-fieldinject-smoke-test-on-java25 branch from cebf1c7 to a73fd0b Compare December 4, 2025 17:58
@mcculls mcculls changed the title WIP: Fix field-injection of ForkJoinTask on Java 25 Fix field-injection of ForkJoinTask on Java 25 Dec 4, 2025
@mcculls mcculls added comp: core Tracer core and removed inst: java Core Java language instrumentation labels Dec 4, 2025
@pr-commenter
Copy link

pr-commenter bot commented Dec 4, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/fix-fieldinject-smoke-test-on-java25
git_commit_date 1764867093 1764873561
git_commit_sha b690a79 c4beb14
release_version 1.57.0-SNAPSHOT~b690a79bd7 1.57.0-SNAPSHOT~c4beb14f04
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1764878002 1764878002
ci_job_id 1271144144 1271144144
ci_pipeline_id 84929293 84929293
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-2x9ospfa 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-2x9ospfa 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 18 performance improvements and 6 performance regressions! Performance is the same for 27 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast:crashtracking better
[-328.324µs; -277.962µs] or [-22.055%; -18.672%]		 1.186ms 1.489ms
scenario:startup:insecure-bank:iast:BytebuddyAgent better
[-50.127ms; -36.011ms] or [-6.015%; -4.321%]		 790.269ms 833.338ms
scenario:startup:insecure-bank:iast:GlobalTracer worse
[+16.642ms; +25.834ms] or [+7.004%; +10.872%]		 258.868ms 237.630ms
scenario:startup:insecure-bank:iast:Flare Poller better
[-6.925ms; -6.455ms] or [-64.018%; -59.679%]		 4.127ms 10.817ms
scenario:startup:insecure-bank:tracing:crashtracking better
[-299.403µs; -275.368µs] or [-20.309%; -18.679%]		 1.187ms 1.474ms
scenario:startup:insecure-bank:tracing:BytebuddyAgent better
[-45.445ms; -40.247ms] or [-6.587%; -5.834%]		 647.051ms 689.897ms
scenario:startup:insecure-bank:tracing:GlobalTracer worse
[+17.584ms; +20.238ms] or [+6.669%; +7.675%]		 282.586ms 263.675ms
scenario:startup:petclinic:appsec:crashtracking better
[-302.051µs; -250.377µs] or [-20.506%; -16.998%]		 1.197ms 1.473ms
scenario:startup:petclinic:appsec:BytebuddyAgent better
[-48.462ms; -44.136ms] or [-6.615%; -6.025%]		 686.280ms 732.579ms
scenario:startup:petclinic:appsec:GlobalTracer worse
[+17.213ms; +19.314ms] or [+7.137%; +8.008%]		 259.428ms 241.165ms
scenario:startup:petclinic:appsec:Flare Poller worse
[+170.669µs; +319.614µs] or [+4.307%; +8.066%]		 4.208ms 3.962ms
scenario:startup:petclinic:iast:crashtracking better
[-326.466µs; -262.772µs] or [-21.919%; -17.642%]		 1.195ms 1.489ms
scenario:startup:petclinic:iast:BytebuddyAgent better
[-62.753ms; -30.708ms] or [-7.470%; -3.656%]		 793.305ms 840.036ms
scenario:startup:petclinic:iast:GlobalTracer worse
[+14.424ms; +18.224ms] or [+6.034%; +7.623%]		 255.385ms 239.061ms
scenario:startup:petclinic:iast:Remote Config better
[-38.299µs; -16.220µs] or [-6.213%; -2.631%]		 589.192µs 616.452µs
scenario:startup:petclinic:iast:Flare Poller better
[-6.837ms; -6.521ms] or [-62.899%; -59.995%]		 4.190ms 10.869ms
scenario:startup:petclinic:profiling:Agent.start better
[-58.357ms; -24.214ms] or [-4.834%; -2.006%]		 1.166s 1.207s
scenario:startup:petclinic:profiling:crashtracking better
[-280.673µs; -217.759µs] or [-19.459%; -15.098%]		 1.193ms 1.442ms
scenario:startup:petclinic:profiling:BytebuddyAgent better
[-49.317ms; -27.555ms] or [-6.664%; -3.724%]		 701.578ms 740.015ms
scenario:startup:petclinic:tracing:Agent.start better
[-30.274ms; -24.263ms] or [-2.872%; -2.301%]		 1.027s 1.054s
scenario:startup:petclinic:tracing:crashtracking better
[-308.220µs; -246.260µs] or [-20.785%; -16.607%]		 1.206ms 1.483ms
scenario:startup:petclinic:tracing:BytebuddyAgent better
[-47.893ms; -43.313ms] or [-6.913%; -6.252%]		 647.181ms 692.785ms
scenario:startup:petclinic:tracing:GlobalTracer worse
[+16.906ms; +18.579ms] or [+6.383%; +7.014%]		 282.618ms 264.875ms
scenario:startup:petclinic:tracing:Telemetry better
[-7.638ms; -7.074ms] or [-46.038%; -42.637%]		 9.235ms 16.591ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054247
Total [baseline] (10.942 s) : 0, 10942466
Agent [candidate] (1.027 s) : 0, 1026978
Total [candidate] (10.926 s) : 0, 10925591
section appsec
Agent [baseline] (1.23 s) : 0, 1230391
Total [baseline] (10.943 s) : 0, 10942681
Agent [candidate] (1.202 s) : 0, 1202372
Total [candidate] (10.917 s) : 0, 10917066
section iast
Agent [baseline] (1.204 s) : 0, 1204422
Total [baseline] (11.249 s) : 0, 11249194
Agent [candidate] (1.167 s) : 0, 1166935
Total [candidate] (11.169 s) : 0, 11169205
section profiling
Agent [baseline] (1.207 s) : 0, 1207196
Total [baseline] (10.961 s) : 0, 10960590
Agent [candidate] (1.166 s) : 0, 1165910
Total [candidate] (10.903 s) : 0, 10903124
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent appsec 1.23 s 176.144 ms (16.7%)
Agent iast 1.204 s 150.175 ms (14.2%)
Agent profiling 1.207 s 152.949 ms (14.5%)
Total tracing 10.942 s -
Total appsec 10.943 s 215.053 µs (0.0%)
Total iast 11.249 s 306.728 ms (2.8%)
Total profiling 10.961 s 18.124 ms (0.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.027 s -
Agent appsec 1.202 s 175.394 ms (17.1%)
Agent iast 1.167 s 139.957 ms (13.6%)
Agent profiling 1.166 s 138.932 ms (13.5%)
Total tracing 10.926 s -
Total appsec 10.917 s -8.525 ms (-0.1%)
Total iast 11.169 s 243.614 ms (2.2%)
Total profiling 10.903 s -22.467 ms (-0.2%) 
gantt
    title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.483 ms) : 0, 1483
crashtracking [candidate] (1.206 ms) : 0, 1206
BytebuddyAgent [baseline] (692.785 ms) : 0, 692785
BytebuddyAgent [candidate] (647.181 ms) : 0, 647181
GlobalTracer [baseline] (264.875 ms) : 0, 264875
GlobalTracer [candidate] (282.618 ms) : 0, 282618
AppSec [baseline] (31.998 ms) : 0, 31998
AppSec [candidate] (32.108 ms) : 0, 32108
Debugger [baseline] (6.351 ms) : 0, 6351
Debugger [candidate] (6.314 ms) : 0, 6314
Remote Config [baseline] (675.255 µs) : 0, 675
Remote Config [candidate] (668.192 µs) : 0, 668
Telemetry [baseline] (16.591 ms) : 0, 16591
Telemetry [candidate] (9.235 ms) : 0, 9235
Flare Poller [baseline] (4.351 ms) : 0, 4351
Flare Poller [candidate] (11.959 ms) : 0, 11959
section appsec
crashtracking [baseline] (1.473 ms) : 0, 1473
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (732.579 ms) : 0, 732579
BytebuddyAgent [candidate] (686.28 ms) : 0, 686280
GlobalTracer [baseline] (241.165 ms) : 0, 241165
GlobalTracer [candidate] (259.428 ms) : 0, 259428
IAST [baseline] (24.633 ms) : 0, 24633
IAST [candidate] (24.67 ms) : 0, 24670
AppSec [baseline] (174.703 ms) : 0, 174703
AppSec [candidate] (174.011 ms) : 0, 174011
Debugger [baseline] (6.348 ms) : 0, 6348
Debugger [candidate] (6.516 ms) : 0, 6516
Remote Config [baseline] (710.773 µs) : 0, 711
Remote Config [candidate] (727.251 µs) : 0, 727
Telemetry [baseline] (9.725 ms) : 0, 9725
Telemetry [candidate] (9.958 ms) : 0, 9958
Flare Poller [baseline] (3.962 ms) : 0, 3962
Flare Poller [candidate] (4.208 ms) : 0, 4208
section iast
crashtracking [baseline] (1.489 ms) : 0, 1489
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (840.036 ms) : 0, 840036
BytebuddyAgent [candidate] (793.305 ms) : 0, 793305
GlobalTracer [baseline] (239.061 ms) : 0, 239061
GlobalTracer [candidate] (255.385 ms) : 0, 255385
IAST [baseline] (30.478 ms) : 0, 30478
IAST [candidate] (26.702 ms) : 0, 26702
AppSec [baseline] (31.705 ms) : 0, 31705
AppSec [candidate] (35.373 ms) : 0, 35373
Debugger [baseline] (6.109 ms) : 0, 6109
Debugger [candidate] (6.067 ms) : 0, 6067
Remote Config [baseline] (616.451 µs) : 0, 616
Remote Config [candidate] (589.192 µs) : 0, 589
Telemetry [baseline] (8.643 ms) : 0, 8643
Telemetry [candidate] (8.672 ms) : 0, 8672
Flare Poller [baseline] (10.869 ms) : 0, 10869
Flare Poller [candidate] (4.19 ms) : 0, 4190
section profiling
crashtracking [baseline] (1.442 ms) : 0, 1442
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (740.015 ms) : 0, 740015
BytebuddyAgent [candidate] (701.578 ms) : 0, 701578
GlobalTracer [baseline] (223.541 ms) : 0, 223541
GlobalTracer [candidate] (221.275 ms) : 0, 221275
AppSec [baseline] (32.488 ms) : 0, 32488
AppSec [candidate] (31.976 ms) : 0, 31976
Debugger [baseline] (7.468 ms) : 0, 7468
Debugger [candidate] (6.674 ms) : 0, 6674
Remote Config [baseline] (1.45 ms) : 0, 1450
Remote Config [candidate] (679.301 µs) : 0, 679
Telemetry [baseline] (15.222 ms) : 0, 15222
Telemetry [candidate] (9.836 ms) : 0, 9836
Flare Poller [baseline] (4.118 ms) : 0, 4118
Flare Poller [candidate] (10.328 ms) : 0, 10328
ProfilingAgent [baseline] (111.514 ms) : 0, 111514
ProfilingAgent [candidate] (112.549 ms) : 0, 112549
Profiling [baseline] (112.16 ms) : 0, 112160
Profiling [candidate] (113.212 ms) : 0, 113212
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1049973
Total [baseline] (8.738 s) : 0, 8738025
Agent [candidate] (1.025 s) : 0, 1024791
Total [candidate] (8.714 s) : 0, 8713806
section iast
Agent [baseline] (1.195 s) : 0, 1195101
Total [baseline] (9.351 s) : 0, 9351011
Agent [candidate] (1.169 s) : 0, 1168635
Total [candidate] (9.355 s) : 0, 9354793
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.05 s -
Agent iast 1.195 s 145.128 ms (13.8%)
Total tracing 8.738 s -
Total iast 9.351 s 612.987 ms (7.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.025 s -
Agent iast 1.169 s 143.843 ms (14.0%)
Total tracing 8.714 s -
Total iast 9.355 s 640.987 ms (7.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.474 ms) : 0, 1474
crashtracking [candidate] (1.187 ms) : 0, 1187
BytebuddyAgent [baseline] (689.897 ms) : 0, 689897
BytebuddyAgent [candidate] (647.051 ms) : 0, 647051
GlobalTracer [baseline] (263.675 ms) : 0, 263675
GlobalTracer [candidate] (282.586 ms) : 0, 282586
AppSec [baseline] (31.891 ms) : 0, 31891
AppSec [candidate] (32.355 ms) : 0, 32355
Debugger [baseline] (6.317 ms) : 0, 6317
Debugger [candidate] (6.376 ms) : 0, 6376
Remote Config [baseline] (658.288 µs) : 0, 658
Remote Config [candidate] (666.195 µs) : 0, 666
Telemetry [baseline] (15.119 ms) : 0, 15119
Telemetry [candidate] (9.287 ms) : 0, 9287
Flare Poller [baseline] (5.839 ms) : 0, 5839
Flare Poller [candidate] (9.637 ms) : 0, 9637
section iast
crashtracking [baseline] (1.489 ms) : 0, 1489
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (833.338 ms) : 0, 833338
BytebuddyAgent [candidate] (790.269 ms) : 0, 790269
GlobalTracer [baseline] (237.63 ms) : 0, 237630
GlobalTracer [candidate] (258.868 ms) : 0, 258868
IAST [baseline] (34.463 ms) : 0, 34463
IAST [candidate] (28.031 ms) : 0, 28031
AppSec [baseline] (27.026 ms) : 0, 27026
AppSec [candidate] (33.271 ms) : 0, 33271
Debugger [baseline] (6.036 ms) : 0, 6036
Debugger [candidate] (7.953 ms) : 0, 7953
Remote Config [baseline] (628.731 µs) : 0, 629
Remote Config [candidate] (626.766 µs) : 0, 627
Telemetry [baseline] (8.489 ms) : 0, 8489
Telemetry [candidate] (8.902 ms) : 0, 8902
Flare Poller [baseline] (10.817 ms) : 0, 10817
Flare Poller [candidate] (4.127 ms) : 0, 4127
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/fix-fieldinject-smoke-test-on-java25
git_commit_date 1764867093 1764873561
git_commit_sha b690a79 c4beb14
release_version 1.57.0-SNAPSHOT~b690a79bd7 1.57.0-SNAPSHOT~c4beb14f04
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1764875790 1764875790
ci_job_id 1271025014 1271025014
ci_pipeline_id 84929293 84929293
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-zuogonxw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-zuogonxw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast:high_load better
[-134.393µs; -56.123µs] or [-5.223%; -2.181%]		 same
[-220.399µs; +205.663µs] or [-2.941%; +2.745%]		 unstable
[-132.317op/s; +172.817op/s] or [-9.495%; +12.401%]		 2.478ms 7.486ms 1413.844op/s 2.573ms 7.493ms 1393.594op/s
scenario:load:petclinic:tracing:high_load worse
[+0.876ms; +1.540ms] or [+5.095%; +8.960%]		 worse
[+0.834ms; +1.925ms] or [+2.938%; +6.782%]		 unstable
[-42.477op/s; +10.227op/s] or [-15.973%; +3.846%]		 18.401ms 29.764ms 249.812op/s 17.193ms 28.385ms 265.938op/s
scenario:load:petclinic:profiling:high_load better
[-1404.342µs; -403.301µs] or [-7.366%; -2.115%]		 unsure
[-1.940ms; -0.127ms] or [-6.285%; -0.411%]		 unstable
[-14.403op/s; +37.153op/s] or [-5.986%; +15.440%]		 18.161ms 29.827ms 252.000op/s 19.065ms 30.861ms 240.625op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.195 ms) : 1184, 1207
.   : milestone, 1195,
iast (3.286 ms) : 3244, 3328
.   : milestone, 3286,
iast_FULL (5.742 ms) : 5685, 5799
.   : milestone, 5742,
iast_GLOBAL (3.681 ms) : 3627, 3736
.   : milestone, 3681,
profiling (2.015 ms) : 1996, 2034
.   : milestone, 2015,
tracing (1.783 ms) : 1769, 1798
.   : milestone, 1783,
section candidate
no_agent (1.231 ms) : 1219, 1243
.   : milestone, 1231,
iast (3.236 ms) : 3193, 3279
.   : milestone, 3236,
iast_FULL (5.618 ms) : 5563, 5674
.   : milestone, 5618,
iast_GLOBAL (3.738 ms) : 3683, 3793
.   : milestone, 3738,
profiling (2.011 ms) : 1992, 2030
.   : milestone, 2011,
tracing (1.845 ms) : 1828, 1861
.   : milestone, 1845,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.195 ms [1.184 ms, 1.207 ms] -
iast 3.286 ms [3.244 ms, 3.328 ms] 2.091 ms (174.9%)
iast_FULL 5.742 ms [5.685 ms, 5.799 ms] 4.547 ms (380.4%)
iast_GLOBAL 3.681 ms [3.627 ms, 3.736 ms] 2.486 ms (208.0%)
profiling 2.015 ms [1.996 ms, 2.034 ms] 819.451 µs (68.6%)
tracing 1.783 ms [1.769 ms, 1.798 ms] 587.914 µs (49.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.231 ms [1.219 ms, 1.243 ms] -
iast 3.236 ms [3.193 ms, 3.279 ms] 2.005 ms (162.9%)
iast_FULL 5.618 ms [5.563 ms, 5.674 ms] 4.387 ms (356.4%)
iast_GLOBAL 3.738 ms [3.683 ms, 3.793 ms] 2.507 ms (203.7%)
profiling 2.011 ms [1.992 ms, 2.03 ms] 780.237 µs (63.4%)
tracing 1.845 ms [1.828 ms, 1.861 ms] 613.645 µs (49.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.315 ms) : 18127, 18504
.   : milestone, 18315,
appsec (18.518 ms) : 18331, 18705
.   : milestone, 18518,
code_origins (17.795 ms) : 17617, 17973
.   : milestone, 17795,
iast (17.861 ms) : 17685, 18038
.   : milestone, 17861,
profiling (19.399 ms) : 19200, 19599
.   : milestone, 19399,
tracing (17.547 ms) : 17372, 17723
.   : milestone, 17547,
section candidate
no_agent (18.933 ms) : 18742, 19125
.   : milestone, 18933,
appsec (18.684 ms) : 18492, 18875
.   : milestone, 18684,
code_origins (17.699 ms) : 17525, 17872
.   : milestone, 17699,
iast (17.679 ms) : 17508, 17851
.   : milestone, 17679,
profiling (18.52 ms) : 18332, 18709
.   : milestone, 18520,
tracing (18.686 ms) : 18500, 18871
.   : milestone, 18686,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.315 ms [18.127 ms, 18.504 ms] -
appsec 18.518 ms [18.331 ms, 18.705 ms] 202.554 µs (1.1%)
code_origins 17.795 ms [17.617 ms, 17.973 ms] -520.531 µs (-2.8%)
iast 17.861 ms [17.685 ms, 18.038 ms] -454.143 µs (-2.5%)
profiling 19.399 ms [19.2 ms, 19.599 ms] 1.084 ms (5.9%)
tracing 17.547 ms [17.372 ms, 17.723 ms] -768.318 µs (-4.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.933 ms [18.742 ms, 19.125 ms] -
appsec 18.684 ms [18.492 ms, 18.875 ms] -249.866 µs (-1.3%)
code_origins 17.699 ms [17.525 ms, 17.872 ms] -1.235 ms (-6.5%)
iast 17.679 ms [17.508 ms, 17.851 ms] -1.254 ms (-6.6%)
profiling 18.52 ms [18.332 ms, 18.709 ms] -413.247 µs (-2.2%)
tracing 18.686 ms [18.5 ms, 18.871 ms] -247.914 µs (-1.3%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/fix-fieldinject-smoke-test-on-java25
git_commit_date 1764867093 1764873561
git_commit_sha b690a79 c4beb14
release_version 1.57.0-SNAPSHOT~b690a79bd7 1.57.0-SNAPSHOT~c4beb14f04
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1764875456 1764875456
ci_job_id 1271025015 1271025015
ci_pipeline_id 84929293 84929293
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-zdsu2lt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-zdsu2lt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1489
.   : milestone, 1477,
appsec (3.69 ms) : 3473, 3907
.   : milestone, 3690,
iast (2.211 ms) : 2146, 2276
.   : milestone, 2211,
iast_GLOBAL (2.258 ms) : 2193, 2324
.   : milestone, 2258,
profiling (2.055 ms) : 2002, 2107
.   : milestone, 2055,
tracing (2.047 ms) : 1996, 2099
.   : milestone, 2047,
section candidate
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (3.694 ms) : 3477, 3910
.   : milestone, 3694,
iast (2.211 ms) : 2146, 2276
.   : milestone, 2211,
iast_GLOBAL (2.256 ms) : 2190, 2321
.   : milestone, 2256,
profiling (2.09 ms) : 2035, 2145
.   : milestone, 2090,
tracing (2.042 ms) : 1991, 2093
.   : milestone, 2042,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.477 ms [1.465 ms, 1.489 ms] -
appsec 3.69 ms [3.473 ms, 3.907 ms] 2.213 ms (149.8%)
iast 2.211 ms [2.146 ms, 2.276 ms] 734.152 µs (49.7%)
iast_GLOBAL 2.258 ms [2.193 ms, 2.324 ms] 781.509 µs (52.9%)
profiling 2.055 ms [2.002 ms, 2.107 ms] 577.529 µs (39.1%)
tracing 2.047 ms [1.996 ms, 2.099 ms] 570.419 µs (38.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.462 ms, 1.485 ms] -
appsec 3.694 ms [3.477 ms, 3.91 ms] 2.221 ms (150.7%)
iast 2.211 ms [2.146 ms, 2.276 ms] 737.876 µs (50.1%)
iast_GLOBAL 2.256 ms [2.19 ms, 2.321 ms] 782.884 µs (53.1%)
profiling 2.09 ms [2.035 ms, 2.145 ms] 617.13 µs (41.9%)
tracing 2.042 ms [1.991 ms, 2.093 ms] 568.571 µs (38.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~c4beb14f04, baseline=1.57.0-SNAPSHOT~b690a79bd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.608 s) : 15608000, 15608000
.   : milestone, 15608000,
appsec (14.927 s) : 14927000, 14927000
.   : milestone, 14927000,
iast (18.197 s) : 18197000, 18197000
.   : milestone, 18197000,
iast_GLOBAL (18.206 s) : 18206000, 18206000
.   : milestone, 18206000,
profiling (14.582 s) : 14582000, 14582000
.   : milestone, 14582000,
tracing (14.658 s) : 14658000, 14658000
.   : milestone, 14658000,
section candidate
no_agent (15.618 s) : 15618000, 15618000
.   : milestone, 15618000,
appsec (14.504 s) : 14504000, 14504000
.   : milestone, 14504000,
iast (17.974 s) : 17974000, 17974000
.   : milestone, 17974000,
iast_GLOBAL (17.861 s) : 17861000, 17861000
.   : milestone, 17861000,
profiling (14.668 s) : 14668000, 14668000
.   : milestone, 14668000,
tracing (14.534 s) : 14534000, 14534000
.   : milestone, 14534000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.608 s [15.608 s, 15.608 s] -
appsec 14.927 s [14.927 s, 14.927 s] -681.0 ms (-4.4%)
iast 18.197 s [18.197 s, 18.197 s] 2.589 s (16.6%)
iast_GLOBAL 18.206 s [18.206 s, 18.206 s] 2.598 s (16.6%)
profiling 14.582 s [14.582 s, 14.582 s] -1.026 s (-6.6%)
tracing 14.658 s [14.658 s, 14.658 s] -950.0 ms (-6.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.618 s [15.618 s, 15.618 s] -
appsec 14.504 s [14.504 s, 14.504 s] -1.114 s (-7.1%)
iast 17.974 s [17.974 s, 17.974 s] 2.356 s (15.1%)
iast_GLOBAL 17.861 s [17.861 s, 17.861 s] 2.243 s (14.4%)
profiling 14.668 s [14.668 s, 14.668 s] -950.0 ms (-6.1%)
tracing 14.534 s [14.534 s, 14.534 s] -1.084 s (-6.9%)

@mcculls mcculls marked this pull request as ready for review December 4, 2025 19:30
@mcculls mcculls requested a review from a team as a code owner December 4, 2025 19:30
amarziali
amarziali approved these changes Dec 4, 2025
View reviewed changes
Copy link
Contributor

@amarziali amarziali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix and for adding the test case. I’m wondering whether, in the long run, we should introduce a very lightweight scheduler - something that won’t impact load but would still provide an opportunity to perform field injection when classes are redefined.

@mcculls
Copy link
Contributor Author

mcculls commented Dec 4, 2025

Thanks for the fix and for adding the test case. I’m wondering whether, in the long run, we should introduce a very lightweight scheduler - something that won’t impact load but would still provide an opportunity to perform field injection when classes are redefined.

That was the idea behind AgentTaskScheduler :)

When I joined the tracer used a ScheduledThreadPoolExecutor but that led to all sorts of odd behaviour once we started expanding our java.lang.concurrent instrumentation.

So we replaced it with a simple mechanism built on top of a DelayQeueue: #1861 - i.e. decoupled from any of the executor implementation classes.

It's only recently that Oracle made the internals behind AbstractQueuedSynchronizer adaptive, in the sense that if the thread belongs to a fork-join pool then it adjusts the blocking behaviour. This change, coupled with another change in the static initializer of ForkJoinPool led it to it potentially initializing the ForkJoinTask class in a specific scenario, even if the thread has nothing to do with fork-join.

Any solution that uses AbstractQueuedSynchronizer will suffer from the same issue. Likewise for other java.lang.concurrent building blocks that attempt to be adaptive.

While we could potentially develop our own DelayQueue implementation from scratch, it won't be necessary in the near future because instead we'll be doing field injection separately from all the other instrumentation - and this will happen really early on before any scheduling.

So this is a pragmatic short-term fix which soon won't be necessary.

@mcculls mcculls merged commit 4e48384 into master Dec 4, 2025
548 checks passed
@mcculls mcculls deleted the mcculls/fix-fieldinject-smoke-test-on-java25 branch December 4, 2025 22:50
@github-actions github-actions bot added this to the 1.57.0 milestone Dec 4, 2025
@PerfectSlayer
Copy link
Contributor

PerfectSlayer commented Dec 5, 2025

Thanks for all the context and details explanation 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amarziali amarziali amarziali approved these changes

@smola smola Awaiting requested review from smola smola is a code owner automatically assigned from DataDog/apm-java

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer

Assignees

No one assigned

Labels

comp: core Tracer core type: bug Bug report and fix

Projects

None yet

Milestone

1.57.0

Development

Successfully merging this pull request may close these issues.

4 participants

@mcculls @PerfectSlayer @amarziali