🪞 9979 - Record detailed instrumentation errors during tests#10300
Merged
Conversation
Contributor
Author
|
Created this to mainly run CI checks - there are a couple of things to clean up before this can be merged. |
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
mcculls
added
type: enhancement
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 10 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1063218
Total [baseline] (11.029 s) : 0, 11029280
Agent [candidate] (1.063 s) : 0, 1063351
Total [candidate] (10.957 s) : 0, 10956896
section appsec
Agent [baseline] (1.269 s) : 0, 1268649
Total [baseline] (11.194 s) : 0, 11194304
Agent [candidate] (1.278 s) : 0, 1278266
Total [candidate] (11.186 s) : 0, 11185602
section iast
Agent [baseline] (1.248 s) : 0, 1247693
Total [baseline] (11.302 s) : 0, 11302294
Agent [candidate] (1.244 s) : 0, 1244388
Total [candidate] (11.28 s) : 0, 11280398
section profiling
Agent [baseline] (1.184 s) : 0, 1184084
Total [baseline] (10.96 s) : 0, 10960443
Agent [candidate] (1.188 s) : 0, 1188126
Total [candidate] (10.994 s) : 0, 10994445
gantt
title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.234 ms) : 0, 1234
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (635.068 ms) : 0, 635068
BytebuddyAgent [candidate] (635.958 ms) : 0, 635958
AgentMeter [baseline] (29.396 ms) : 0, 29396
AgentMeter [candidate] (29.492 ms) : 0, 29492
GlobalTracer [baseline] (248.529 ms) : 0, 248529
GlobalTracer [candidate] (248.266 ms) : 0, 248266
AppSec [baseline] (32.652 ms) : 0, 32652
AppSec [candidate] (32.655 ms) : 0, 32655
Debugger [baseline] (60.439 ms) : 0, 60439
Debugger [candidate] (60.252 ms) : 0, 60252
Remote Config [baseline] (604.943 µs) : 0, 605
Remote Config [candidate] (592.344 µs) : 0, 592
Telemetry [baseline] (8.384 ms) : 0, 8384
Telemetry [candidate] (9.179 ms) : 0, 9179
Flare Poller [baseline] (10.839 ms) : 0, 10839
Flare Poller [candidate] (9.746 ms) : 0, 9746
section appsec
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (675.691 ms) : 0, 675691
BytebuddyAgent [candidate] (681.012 ms) : 0, 681012
AgentMeter [baseline] (12.291 ms) : 0, 12291
AgentMeter [candidate] (12.307 ms) : 0, 12307
GlobalTracer [baseline] (250.109 ms) : 0, 250109
GlobalTracer [candidate] (252.12 ms) : 0, 252120
IAST [baseline] (24.895 ms) : 0, 24895
IAST [candidate] (25.054 ms) : 0, 25054
AppSec [baseline] (185.8 ms) : 0, 185800
AppSec [candidate] (186.488 ms) : 0, 186488
Debugger [baseline] (65.39 ms) : 0, 65390
Debugger [candidate] (66.004 ms) : 0, 66004
Remote Config [baseline] (590.825 µs) : 0, 591
Remote Config [candidate] (589.909 µs) : 0, 590
Telemetry [baseline] (8.032 ms) : 0, 8032
Telemetry [candidate] (7.96 ms) : 0, 7960
Flare Poller [baseline] (8.006 ms) : 0, 8006
Flare Poller [candidate] (9.275 ms) : 0, 9275
section iast
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (826.047 ms) : 0, 826047
BytebuddyAgent [candidate] (823.658 ms) : 0, 823658
AgentMeter [baseline] (11.479 ms) : 0, 11479
AgentMeter [candidate] (11.282 ms) : 0, 11282
GlobalTracer [baseline] (237.884 ms) : 0, 237884
GlobalTracer [candidate] (237.35 ms) : 0, 237350
IAST [baseline] (28.94 ms) : 0, 28940
IAST [candidate] (26.487 ms) : 0, 26487
AppSec [baseline] (30.626 ms) : 0, 30626
AppSec [candidate] (32.976 ms) : 0, 32976
Debugger [baseline] (63.299 ms) : 0, 63299
Debugger [candidate] (63.262 ms) : 0, 63262
Remote Config [baseline] (535.637 µs) : 0, 536
Remote Config [candidate] (523.572 µs) : 0, 524
Telemetry [baseline] (7.99 ms) : 0, 7990
Telemetry [candidate] (8.064 ms) : 0, 8064
Flare Poller [baseline] (3.462 ms) : 0, 3462
Flare Poller [candidate] (3.514 ms) : 0, 3514
section profiling
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.19 ms) : 0, 1190
BytebuddyAgent [baseline] (690.213 ms) : 0, 690213
BytebuddyAgent [candidate] (692.196 ms) : 0, 692196
AgentMeter [baseline] (8.855 ms) : 0, 8855
AgentMeter [candidate] (8.871 ms) : 0, 8871
GlobalTracer [baseline] (207.711 ms) : 0, 207711
GlobalTracer [candidate] (208.762 ms) : 0, 208762
AppSec [baseline] (32.675 ms) : 0, 32675
AppSec [candidate] (32.747 ms) : 0, 32747
Debugger [baseline] (65.84 ms) : 0, 65840
Debugger [candidate] (66.22 ms) : 0, 66220
Remote Config [baseline] (582.215 µs) : 0, 582
Remote Config [candidate] (575.073 µs) : 0, 575
Telemetry [baseline] (8.079 ms) : 0, 8079
Telemetry [candidate] (8.157 ms) : 0, 8157
Flare Poller [baseline] (3.505 ms) : 0, 3505
Flare Poller [candidate] (3.59 ms) : 0, 3590
ProfilingAgent [baseline] (94.052 ms) : 0, 94052
ProfilingAgent [candidate] (94.365 ms) : 0, 94365
Profiling [baseline] (94.608 ms) : 0, 94608
Profiling [candidate] (94.932 ms) : 0, 94932
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1063193
Total [baseline] (8.832 s) : 0, 8831808
Agent [candidate] (1.06 s) : 0, 1059674
Total [candidate] (8.826 s) : 0, 8826354
section iast
Agent [baseline] (1.242 s) : 0, 1242362
Total [baseline] (9.524 s) : 0, 9523573
Agent [candidate] (1.25 s) : 0, 1249709
Total [candidate] (9.546 s) : 0, 9546446
gantt
title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.226 ms) : 0, 1226
crashtracking [candidate] (1.242 ms) : 0, 1242
BytebuddyAgent [baseline] (634.574 ms) : 0, 634574
BytebuddyAgent [candidate] (633.885 ms) : 0, 633885
AgentMeter [baseline] (29.419 ms) : 0, 29419
AgentMeter [candidate] (29.319 ms) : 0, 29319
GlobalTracer [baseline] (248.627 ms) : 0, 248627
GlobalTracer [candidate] (247.699 ms) : 0, 247699
AppSec [baseline] (32.766 ms) : 0, 32766
AppSec [candidate] (32.627 ms) : 0, 32627
Debugger [baseline] (60.02 ms) : 0, 60020
Debugger [candidate] (59.446 ms) : 0, 59446
Remote Config [baseline] (599.911 µs) : 0, 600
Remote Config [candidate] (594.395 µs) : 0, 594
Telemetry [baseline] (9.968 ms) : 0, 9968
Telemetry [candidate] (9.07 ms) : 0, 9070
Flare Poller [baseline] (9.987 ms) : 0, 9987
Flare Poller [candidate] (9.906 ms) : 0, 9906
section iast
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (823.36 ms) : 0, 823360
BytebuddyAgent [candidate] (826.926 ms) : 0, 826926
AgentMeter [baseline] (11.255 ms) : 0, 11255
AgentMeter [candidate] (11.559 ms) : 0, 11559
GlobalTracer [baseline] (238.728 ms) : 0, 238728
GlobalTracer [candidate] (239.109 ms) : 0, 239109
IAST [baseline] (26.49 ms) : 0, 26490
IAST [candidate] (29.26 ms) : 0, 29260
AppSec [baseline] (31.211 ms) : 0, 31211
AppSec [candidate] (30.893 ms) : 0, 30893
Debugger [baseline] (62.316 ms) : 0, 62316
Debugger [candidate] (62.685 ms) : 0, 62685
Remote Config [baseline] (526.513 µs) : 0, 527
Remote Config [candidate] (526.789 µs) : 0, 527
Telemetry [baseline] (7.87 ms) : 0, 7870
Telemetry [candidate] (7.959 ms) : 0, 7959
Flare Poller [baseline] (3.347 ms) : 0, 3347
Flare Poller [candidate] (3.385 ms) : 0, 3385
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 2 performance regressions! Performance is the same for 19 metrics, 15 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section baseline
no_agent (19.22 ms) : 19026, 19415
. : milestone, 19220,
appsec (18.691 ms) : 18499, 18883
. : milestone, 18691,
code_origins (17.767 ms) : 17591, 17943
. : milestone, 17767,
iast (17.845 ms) : 17672, 18018
. : milestone, 17845,
profiling (18.111 ms) : 17932, 18289
. : milestone, 18111,
tracing (17.788 ms) : 17615, 17961
. : milestone, 17788,
section candidate
no_agent (19.272 ms) : 19076, 19468
. : milestone, 19272,
appsec (19.86 ms) : 19657, 20063
. : milestone, 19860,
code_origins (17.88 ms) : 17700, 18060
. : milestone, 17880,
iast (17.894 ms) : 17717, 18071
. : milestone, 17894,
profiling (18.46 ms) : 18277, 18643
. : milestone, 18460,
tracing (17.691 ms) : 17513, 17869
. : milestone, 17691,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section baseline
no_agent (1.248 ms) : 1236, 1260
. : milestone, 1248,
iast (3.366 ms) : 3315, 3416
. : milestone, 3366,
iast_FULL (5.984 ms) : 5924, 6044
. : milestone, 5984,
iast_GLOBAL (3.761 ms) : 3696, 3826
. : milestone, 3761,
profiling (2.335 ms) : 2314, 2357
. : milestone, 2335,
tracing (1.908 ms) : 1892, 1924
. : milestone, 1908,
section candidate
no_agent (1.252 ms) : 1240, 1265
. : milestone, 1252,
iast (3.361 ms) : 3308, 3415
. : milestone, 3361,
iast_FULL (6.065 ms) : 6004, 6126
. : milestone, 6065,
iast_GLOBAL (3.762 ms) : 3694, 3830
. : milestone, 3762,
profiling (2.294 ms) : 2273, 2316
. : milestone, 2294,
tracing (1.9 ms) : 1885, 1916
. : milestone, 1900,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section baseline
no_agent (1.49 ms) : 1479, 1502
. : milestone, 1490,
appsec (3.846 ms) : 3622, 4070
. : milestone, 3846,
iast (2.273 ms) : 2204, 2342
. : milestone, 2273,
iast_GLOBAL (2.317 ms) : 2246, 2387
. : milestone, 2317,
profiling (2.103 ms) : 2048, 2158
. : milestone, 2103,
tracing (2.08 ms) : 2026, 2133
. : milestone, 2080,
section candidate
no_agent (1.489 ms) : 1477, 1501
. : milestone, 1489,
appsec (3.825 ms) : 3601, 4049
. : milestone, 3825,
iast (2.289 ms) : 2219, 2359
. : milestone, 2289,
iast_GLOBAL (2.328 ms) : 2258, 2399
. : milestone, 2328,
profiling (2.107 ms) : 2052, 2162
. : milestone, 2107,
tracing (2.083 ms) : 2029, 2137
. : milestone, 2083,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~b406b8c6cd2, baseline=1.62.0-SNAPSHOT~da5e7c3da7
dateFormat X
axisFormat %s
section baseline
no_agent (15.039 s) : 15039000, 15039000
. : milestone, 15039000,
appsec (14.851 s) : 14851000, 14851000
. : milestone, 14851000,
iast (18.803 s) : 18803000, 18803000
. : milestone, 18803000,
iast_GLOBAL (17.919 s) : 17919000, 17919000
. : milestone, 17919000,
profiling (14.883 s) : 14883000, 14883000
. : milestone, 14883000,
tracing (14.939 s) : 14939000, 14939000
. : milestone, 14939000,
section candidate
no_agent (15.581 s) : 15581000, 15581000
. : milestone, 15581000,
appsec (14.98 s) : 14980000, 14980000
. : milestone, 14980000,
iast (18.603 s) : 18603000, 18603000
. : milestone, 18603000,
iast_GLOBAL (17.72 s) : 17720000, 17720000
. : milestone, 17720000,
profiling (15.502 s) : 15502000, 15502000
. : milestone, 15502000,
tracing (14.9 s) : 14900000, 14900000
. : milestone, 14900000,
|
amarziali
force-pushed
the
community-pr-9979
branch
from
7e9999d to
f696e30
Compare
This puts the details front and center in the test summary and is much easier than hunting back through log messages. Note: In InstrumentationErrors, errors are only recorded when enableRecordingAndReset has been called. This is only done from the InstrumentationSpecification test specification, so we don't need to worry about the ArrayList growing without bounds in production. I considered adding a limit to the ArrayList, but opted not to for simplicity. I also wanted to avoid silently discarding some errors.
If there is a blocked instrumentation, the test will fail and the failure message will include the same information that is logged from MuzzleCheck.
amarziali
force-pushed
the
community-pr-9979
branch
from
f696e30 to
b406b8c
Compare
This comment has been minimized.
This comment has been minimized.
* Can be toggled on/off with feature-flag (on for InstrumentationSpecification) * Zero-impact when the detailed instrumentation error feature is off * Allow recording of errors indirectly via InstrumentationSpecification * Reverted change to trace asserts (we shouldn't mix concerns in asserts)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8666b1a064
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
This PR enhances instrumentation test failure visibility by recording instrumentation/muzzle/bytecode-handler errors and surfacing them directly in test assertions, with an opt-in mode to include detailed stack traces.
Changes:
- Introduces a new config flag (
detailed.instrumentation.errors) and wires it throughTraceInstrumentationConfig→InstrumenterConfig. - Records instrumentation errors centrally via
InstrumentationErrors(optionally capturing stack traces) and assertsnoErrors()during/after instrumentation-test execution. - Improves developer-facing error messaging when
InstrumentationContext.get(...)calls were not rewritten as expected.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| metadata/supported-configurations.json | Adds the DD_DETAILED_INSTRUMENTATION_ERRORS supported configuration entry. |
| internal-api/src/main/java/datadog/trace/api/InstrumenterConfig.java | Loads and exposes the detailed.instrumentation.errors flag. |
| dd-trace-api/src/main/java/datadog/trace/api/config/TraceInstrumentationConfig.java | Defines the new DETAILED_INSTRUMENTATION_ERRORS config key constant. |
| dd-java-agent/instrumentation-testing/src/main/java/datadog/trace/agent/test/ClassFileTransformerListener.java | Records Byte Buddy listener errors into InstrumentationErrors instead of a local counter. |
| dd-java-agent/instrumentation-testing/src/main/java/datadog/trace/agent/test/AbstractInstrumentationTest.java | Enables detailed instrumentation errors in JUnit tests and asserts no instrumentation errors at key phases. |
| dd-java-agent/instrumentation-testing/src/main/groovy/datadog/trace/agent/test/InstrumentationSpecification.groovy | Enables detailed errors in Spock tests and asserts no instrumentation errors during install/cleanup. |
| dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/bytebuddy/ExceptionHandlers.java | Records instrumentation-advice exceptions via InstrumentationErrors, optionally with stack traces. |
| dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstrumentationErrors.java | Adds detailed error recording, reset, and human-readable descriptions for test failures. |
| dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstrumentationContext.java | Expands exception text to better explain why context calls were not rewritten. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
amarziali
reviewed
May 22, 2026
| false); | ||
| mv.visitLabel(skipRecord); | ||
| if (frames) { | ||
| mv.visitFrame(Opcodes.F_SAME1, 0, null, 1, new Object[] {"java/lang/Throwable"}); |
Contributor
There was a problem hiding this comment.
@mcculls I was pretty sure that was required. Thanks for having catched up so
amarziali
approved these changes
May 22, 2026
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
mcculls
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR mirrors the changes from the original community contribution to enable CI testing with maintainer privileges.
Original PR: #9979
Original Author: @deejgregor
Original Branch: deejgregor/dd-trace-java:feature-instrumentation-test-error-visibility
Closes #9979
This is an automated mirror created to run CI checks. See tooling/mirror-community-pull-request.sh for details.