Fix flaky InProduct enablement smoke test #8627
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR aims to fix a flaky InProduct enablement smoke test by ensuring the feature is fully started before sending requests and by improving logging clarity.
- Removed the @flaky annotation from one of the tests after adding a more precise waitForFeatureStarted callback
- Adjusted log messages in both test and agent code to indicate the correct startup state
- Simplified diagnostic output in the ServerDebuggerTestApplication
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| dd-smoke-tests/debugger-integration-tests/src/test/java/datadog/smoketest/InProductEnablementIntegrationTest.java | Removed @flaky annotation and updated the message in waitForFeatureStarted to accurately reflect the state |
| dd-smoke-tests/debugger-integration-tests/src/main/java/datadog/smoketest/debugger/ServerDebuggerTestApplication.java | Modified the console output in waitForInstrumentation to remove an unnecessary reference |
| dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/DebuggerAgent.java | Added INFO-level log messages to indicate when dynamic instrumentation, exception replay, and code origin features have started |
Comments suppressed due to low confidence (1)
dd-smoke-tests/debugger-integration-tests/src/test/java/datadog/smoketest/InProductEnablementIntegrationTest.java:54
- Removal of the @flaky annotation may lead to unstable test behavior if the new wait logic does not fully mitigate the race condition. Verify that the waitForFeatureStarted method reliably prevents test flakiness under all conditions.
@Flaky
|
|
||
| protected void waitForInstrumentation(String className) { | ||
| System.out.println("waitForInstrumentation on " + className + " from: " + lastMatchedLine); | ||
| System.out.println("waitForInstrumentation on " + className); |
There was a problem hiding this comment.
[nitpick] Removing the reference to lastMatchedLine from the output message may reduce useful diagnostic details when troubleshooting instrumentation issues. Consider logging lastMatchedLine at a debug level to provide additional context without cluttering the standard output.
Suggested change
| System.out.println("waitForInstrumentation on " + className); | |
| logger.info("waitForInstrumentation on " + className); |
evanchooly
approved these changes
Mar 27, 2025
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Wait for feature started before sending request to avoid racy startup
jpbempel
force-pushed
the
jpbempel/fix-flaky-smoke-test
branch
from
4c6f773 to
b33402c
Compare
Debugger benchmarksParameters
See matching parameters
SummaryFound 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.
See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (311.077 µs) : 288, 334
. : milestone, 311,
basic (309.965 µs) : 301, 318
. : milestone, 310,
loop (10.92 ms) : 10850, 10990
. : milestone, 10920,
section candidate
noprobe (313.316 µs) : 283, 343
. : milestone, 313,
basic (315.507 µs) : 307, 324
. : milestone, 316,
loop (10.132 ms) : 10092, 10172
. : milestone, 10132,
|
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054447
Total [baseline] (10.516 s) : 0, 10515823
Agent [candidate] (1.065 s) : 0, 1065136
Total [candidate] (10.468 s) : 0, 10467633
section appsec
Agent [baseline] (1.195 s) : 0, 1195089
Total [baseline] (10.772 s) : 0, 10771778
Agent [candidate] (1.197 s) : 0, 1196748
Total [candidate] (10.736 s) : 0, 10736131
section iast
Agent [baseline] (1.179 s) : 0, 1179319
Total [baseline] (11.04 s) : 0, 11040400
Agent [candidate] (1.178 s) : 0, 1177664
Total [candidate] (11.022 s) : 0, 11022149
section profiling
Agent [baseline] (1.274 s) : 0, 1274447
Total [baseline] (10.82 s) : 0, 10819937
Agent [candidate] (1.288 s) : 0, 1288157
Total [candidate] (10.873 s) : 0, 10873177
gantt
title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (721.789 ms) : 0, 721789
BytebuddyAgent [candidate] (727.423 ms) : 0, 727423
GlobalTracer [baseline] (240.112 ms) : 0, 240112
GlobalTracer [candidate] (242.177 ms) : 0, 242177
AppSec [baseline] (54.803 ms) : 0, 54803
AppSec [candidate] (55.555 ms) : 0, 55555
Debugger [baseline] (4.486 ms) : 0, 4486
Debugger [candidate] (4.541 ms) : 0, 4541
Remote Config [baseline] (691.34 µs) : 0, 691
Remote Config [candidate] (710.318 µs) : 0, 710
Telemetry [baseline] (12.8 ms) : 0, 12800
Telemetry [candidate] (14.828 ms) : 0, 14828
section appsec
BytebuddyAgent [baseline] (738.708 ms) : 0, 738708
BytebuddyAgent [candidate] (740.251 ms) : 0, 740251
GlobalTracer [baseline] (235.945 ms) : 0, 235945
GlobalTracer [candidate] (236.205 ms) : 0, 236205
AppSec [baseline] (176.771 ms) : 0, 176771
AppSec [candidate] (176.185 ms) : 0, 176185
Debugger [baseline] (4.289 ms) : 0, 4289
Debugger [candidate] (4.302 ms) : 0, 4302
Remote Config [baseline] (641.243 µs) : 0, 641
Remote Config [candidate] (650.721 µs) : 0, 651
Telemetry [baseline] (8.178 ms) : 0, 8178
Telemetry [candidate] (8.544 ms) : 0, 8544
IAST [baseline] (21.68 ms) : 0, 21680
IAST [candidate] (21.639 ms) : 0, 21639
section iast
BytebuddyAgent [baseline] (837.716 ms) : 0, 837716
BytebuddyAgent [candidate] (836.655 ms) : 0, 836655
GlobalTracer [baseline] (229.5 ms) : 0, 229500
GlobalTracer [candidate] (228.843 ms) : 0, 228843
AppSec [baseline] (55.915 ms) : 0, 55915
AppSec [candidate] (55.925 ms) : 0, 55925
Debugger [baseline] (4.221 ms) : 0, 4221
Debugger [candidate] (4.206 ms) : 0, 4206
Remote Config [baseline] (614.213 µs) : 0, 614
Remote Config [candidate] (609.63 µs) : 0, 610
Telemetry [baseline] (8.894 ms) : 0, 8894
Telemetry [candidate] (8.81 ms) : 0, 8810
IAST [baseline] (22.82 ms) : 0, 22820
IAST [candidate] (23.002 ms) : 0, 23002
section profiling
ProfilingAgent [baseline] (102.621 ms) : 0, 102621
ProfilingAgent [candidate] (103.445 ms) : 0, 103445
BytebuddyAgent [baseline] (709.691 ms) : 0, 709691
BytebuddyAgent [candidate] (719.959 ms) : 0, 719959
GlobalTracer [baseline] (349.083 ms) : 0, 349083
GlobalTracer [candidate] (352.149 ms) : 0, 352149
AppSec [baseline] (54.299 ms) : 0, 54299
AppSec [candidate] (53.222 ms) : 0, 53222
Debugger [baseline] (4.28 ms) : 0, 4280
Debugger [candidate] (4.269 ms) : 0, 4269
Remote Config [baseline] (683.554 µs) : 0, 684
Remote Config [candidate] (680.075 µs) : 0, 680
Telemetry [baseline] (8.941 ms) : 0, 8941
Telemetry [candidate] (8.858 ms) : 0, 8858
Profiling [baseline] (102.647 ms) : 0, 102647
Profiling [candidate] (103.47 ms) : 0, 103470
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1061368
Total [baseline] (8.713 s) : 0, 8713404
Agent [candidate] (1.049 s) : 0, 1048895
Total [candidate] (8.668 s) : 0, 8668182
section iast
Agent [baseline] (1.176 s) : 0, 1176369
Total [baseline] (9.264 s) : 0, 9263814
Agent [candidate] (1.178 s) : 0, 1178216
Total [candidate] (9.235 s) : 0, 9234903
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.179 s) : 0, 1179380
Total [baseline] (9.208 s) : 0, 9208004
Agent [candidate] (1.175 s) : 0, 1174910
Total [candidate] (9.243 s) : 0, 9243219
section iast_TELEMETRY_OFF
Agent [baseline] (1.176 s) : 0, 1176277
Total [baseline] (9.247 s) : 0, 9247464
Agent [candidate] (1.177 s) : 0, 1176537
Total [candidate] (9.246 s) : 0, 9245673
gantt
title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (725.581 ms) : 0, 725581
BytebuddyAgent [candidate] (718.317 ms) : 0, 718317
GlobalTracer [baseline] (241.449 ms) : 0, 241449
GlobalTracer [candidate] (239.101 ms) : 0, 239101
AppSec [baseline] (54.977 ms) : 0, 54977
AppSec [candidate] (54.512 ms) : 0, 54512
Debugger [baseline] (4.475 ms) : 0, 4475
Debugger [candidate] (5.146 ms) : 0, 5146
Remote Config [baseline] (699.93 µs) : 0, 700
Remote Config [candidate] (694.603 µs) : 0, 695
Telemetry [baseline] (14.236 ms) : 0, 14236
Telemetry [candidate] (11.414 ms) : 0, 11414
section iast
BytebuddyAgent [baseline] (835.386 ms) : 0, 835386
BytebuddyAgent [candidate] (837.096 ms) : 0, 837096
GlobalTracer [baseline] (228.612 ms) : 0, 228612
GlobalTracer [candidate] (229.058 ms) : 0, 229058
AppSec [baseline] (56.271 ms) : 0, 56271
AppSec [candidate] (55.897 ms) : 0, 55897
Debugger [baseline] (4.212 ms) : 0, 4212
Debugger [candidate] (4.212 ms) : 0, 4212
Remote Config [baseline] (610.096 µs) : 0, 610
Remote Config [candidate] (608.516 µs) : 0, 609
Telemetry [baseline] (8.719 ms) : 0, 8719
Telemetry [candidate] (8.791 ms) : 0, 8791
IAST [baseline] (22.925 ms) : 0, 22925
IAST [candidate] (22.844 ms) : 0, 22844
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.869 ms) : 0, 837869
BytebuddyAgent [candidate] (834.58 ms) : 0, 834580
GlobalTracer [baseline] (229.266 ms) : 0, 229266
GlobalTracer [candidate] (228.077 ms) : 0, 228077
AppSec [baseline] (56.139 ms) : 0, 56139
AppSec [candidate] (56.11 ms) : 0, 56110
Debugger [baseline] (4.179 ms) : 0, 4179
Debugger [candidate] (4.231 ms) : 0, 4231
Remote Config [baseline] (632.756 µs) : 0, 633
Remote Config [candidate] (614.856 µs) : 0, 615
Telemetry [baseline] (8.81 ms) : 0, 8810
Telemetry [candidate] (8.836 ms) : 0, 8836
IAST [baseline] (22.782 ms) : 0, 22782
IAST [candidate] (22.827 ms) : 0, 22827
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (835.999 ms) : 0, 835999
BytebuddyAgent [candidate] (835.768 ms) : 0, 835768
GlobalTracer [baseline] (228.985 ms) : 0, 228985
GlobalTracer [candidate] (229.367 ms) : 0, 229367
AppSec [baseline] (55.947 ms) : 0, 55947
AppSec [candidate] (55.996 ms) : 0, 55996
Debugger [baseline] (4.145 ms) : 0, 4145
Debugger [candidate] (4.137 ms) : 0, 4137
Remote Config [baseline] (600.624 µs) : 0, 601
Remote Config [candidate] (595.379 µs) : 0, 595
Telemetry [baseline] (8.605 ms) : 0, 8605
Telemetry [candidate] (8.709 ms) : 0, 8709
IAST [baseline] (22.33 ms) : 0, 22330
IAST [candidate] (22.31 ms) : 0, 22310
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 17 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section baseline
no_agent (383.011 µs) : 363, 403
. : milestone, 383,
iast (510.943 µs) : 489, 533
. : milestone, 511,
iast_FULL (737.298 µs) : 715, 759
. : milestone, 737,
iast_GLOBAL (561.456 µs) : 540, 583
. : milestone, 561,
iast_HARDCODED_SECRET_DISABLED (519.5 µs) : 498, 541
. : milestone, 520,
iast_INACTIVE (463.998 µs) : 443, 485
. : milestone, 464,
iast_TELEMETRY_OFF (501.192 µs) : 480, 523
. : milestone, 501,
tracing (456.565 µs) : 436, 477
. : milestone, 457,
section candidate
no_agent (379.354 µs) : 360, 399
. : milestone, 379,
iast (516.135 µs) : 494, 538
. : milestone, 516,
iast_FULL (733.7 µs) : 712, 756
. : milestone, 734,
iast_GLOBAL (561.856 µs) : 540, 583
. : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (520.015 µs) : 498, 542
. : milestone, 520,
iast_INACTIVE (465.293 µs) : 443, 487
. : milestone, 465,
iast_TELEMETRY_OFF (505.374 µs) : 483, 528
. : milestone, 505,
tracing (464.527 µs) : 443, 486
. : milestone, 465,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section baseline
no_agent (1.367 ms) : 1347, 1387
. : milestone, 1367,
appsec (1.737 ms) : 1714, 1761
. : milestone, 1737,
appsec_no_iast (1.749 ms) : 1726, 1772
. : milestone, 1749,
code_origins (1.695 ms) : 1668, 1722
. : milestone, 1695,
iast (1.503 ms) : 1479, 1528
. : milestone, 1503,
profiling (1.521 ms) : 1498, 1544
. : milestone, 1521,
tracing (1.506 ms) : 1482, 1531
. : milestone, 1506,
section candidate
no_agent (1.361 ms) : 1342, 1381
. : milestone, 1361,
appsec (1.73 ms) : 1706, 1754
. : milestone, 1730,
appsec_no_iast (1.742 ms) : 1719, 1765
. : milestone, 1742,
code_origins (1.695 ms) : 1668, 1721
. : milestone, 1695,
iast (1.532 ms) : 1508, 1555
. : milestone, 1532,
profiling (1.561 ms) : 1537, 1585
. : milestone, 1561,
tracing (1.501 ms) : 1477, 1525
. : milestone, 1501,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (2.358 ms) : 2314, 2403
. : milestone, 2358,
iast (2.133 ms) : 2076, 2190
. : milestone, 2133,
iast_GLOBAL (2.173 ms) : 2117, 2230
. : milestone, 2173,
profiling (2.448 ms) : 2270, 2626
. : milestone, 2448,
tracing (1.96 ms) : 1917, 2003
. : milestone, 1960,
section candidate
no_agent (1.473 ms) : 1461, 1484
. : milestone, 1473,
appsec (2.352 ms) : 2307, 2397
. : milestone, 2352,
iast (2.139 ms) : 2082, 2196
. : milestone, 2139,
iast_GLOBAL (2.172 ms) : 2115, 2228
. : milestone, 2172,
profiling (2.003 ms) : 1957, 2049
. : milestone, 2003,
tracing (1.964 ms) : 1920, 2007
. : milestone, 1964,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~b33402c729, baseline=1.48.0-SNAPSHOT~4a5461cf3e
dateFormat X
axisFormat %s
section baseline
no_agent (14.975 s) : 14975000, 14975000
. : milestone, 14975000,
appsec (15.154 s) : 15154000, 15154000
. : milestone, 15154000,
iast (18.802 s) : 18802000, 18802000
. : milestone, 18802000,
iast_GLOBAL (17.869 s) : 17869000, 17869000
. : milestone, 17869000,
profiling (15.178 s) : 15178000, 15178000
. : milestone, 15178000,
tracing (15.169 s) : 15169000, 15169000
. : milestone, 15169000,
section candidate
no_agent (14.889 s) : 14889000, 14889000
. : milestone, 14889000,
appsec (14.661 s) : 14661000, 14661000
. : milestone, 14661000,
iast (19.077 s) : 19077000, 19077000
. : milestone, 19077000,
iast_GLOBAL (17.769 s) : 17769000, 17769000
. : milestone, 17769000,
profiling (15.246 s) : 15246000, 15246000
. : milestone, 15246000,
tracing (15.091 s) : 15091000, 15091000
. : milestone, 15091000,
|
smola
reviewed
Mar 28, 2025
| } else { | ||
| LOGGER.debug("No configuration poller available from SharedCommunicationObjects"); | ||
| } | ||
| LOGGER.info("Started Dynamic Instrumentation"); |
Member
There was a problem hiding this comment.
Might want to do debug level here? (and enable debug level in some tests)
Member
Author
There was a problem hiding this comment.
No I think this is fine, it would not spam anyway
jpbempel
added
comp: debugger
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Wait for feature started before sending request to avoid racy startup
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: DEBUG-3653