Fix test status propagation in headless mode #9478
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-tkachenko-datadog
added
the
comp: ci visibility
| * @param durationMillis duration of current test execution in milliseconds | ||
| */ | ||
| void registerExecution(TestStatus status, long durationMillis); | ||
| ExecutionOutcome registerExecution(TestStatus status, long durationMillis); |
There was a problem hiding this comment.
The idea is to force the users to register their executions to be able to query up-to-date state
|
🎯 Code Coverage 🔗 Commit SHA: ae0456a | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 13 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053281
Total [baseline] (10.827 s) : 0, 10827462
Agent [candidate] (1.052 s) : 0, 1052315
Total [candidate] (10.817 s) : 0, 10817213
section appsec
Agent [baseline] (1.225 s) : 0, 1224977
Total [baseline] (10.904 s) : 0, 10903987
Agent [candidate] (1.228 s) : 0, 1227707
Total [candidate] (10.87 s) : 0, 10870213
section iast
Agent [baseline] (1.184 s) : 0, 1184268
Total [baseline] (10.974 s) : 0, 10973869
Agent [candidate] (1.195 s) : 0, 1194509
Total [candidate] (11.093 s) : 0, 11092528
section profiling
Agent [baseline] (1.202 s) : 0, 1201889
Total [baseline] (10.89 s) : 0, 10890206
Agent [candidate] (1.21 s) : 0, 1210128
Total [candidate] (10.983 s) : 0, 10983109
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.46 ms) : 0, 1460
BytebuddyAgent [baseline] (736.147 ms) : 0, 736147
BytebuddyAgent [candidate] (734.772 ms) : 0, 734772
GlobalTracer [baseline] (244.529 ms) : 0, 244529
GlobalTracer [candidate] (243.901 ms) : 0, 243901
AppSec [baseline] (30.51 ms) : 0, 30510
AppSec [candidate] (30.32 ms) : 0, 30320
Debugger [baseline] (6.15 ms) : 0, 6150
Debugger [candidate] (6.099 ms) : 0, 6099
Remote Config [baseline] (701.242 µs) : 0, 701
Remote Config [candidate] (693.513 µs) : 0, 694
Telemetry [baseline] (12.461 ms) : 0, 12461
Telemetry [candidate] (13.863 ms) : 0, 13863
section appsec
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (755.94 ms) : 0, 755940
BytebuddyAgent [candidate] (758.063 ms) : 0, 758063
GlobalTracer [baseline] (235.968 ms) : 0, 235968
GlobalTracer [candidate] (236.431 ms) : 0, 236431
IAST [baseline] (23.663 ms) : 0, 23663
IAST [candidate] (23.604 ms) : 0, 23604
AppSec [baseline] (168.875 ms) : 0, 168875
AppSec [candidate] (169.063 ms) : 0, 169063
Debugger [baseline] (7.329 ms) : 0, 7329
Debugger [candidate] (7.949 ms) : 0, 7949
Remote Config [baseline] (628.674 µs) : 0, 629
Remote Config [candidate] (628.535 µs) : 0, 629
Telemetry [baseline] (10.026 ms) : 0, 10026
Telemetry [candidate] (9.294 ms) : 0, 9294
section iast
crashtracking [baseline] (1.459 ms) : 0, 1459
crashtracking [candidate] (1.471 ms) : 0, 1471
BytebuddyAgent [baseline] (854.37 ms) : 0, 854370
BytebuddyAgent [candidate] (862.177 ms) : 0, 862177
GlobalTracer [baseline] (234.272 ms) : 0, 234272
GlobalTracer [candidate] (235.789 ms) : 0, 235789
IAST [baseline] (29.328 ms) : 0, 29328
IAST [candidate] (30.571 ms) : 0, 30571
AppSec [baseline] (27.015 ms) : 0, 27015
AppSec [candidate] (28.384 ms) : 0, 28384
Debugger [baseline] (7.625 ms) : 0, 7625
Debugger [candidate] (5.899 ms) : 0, 5899
Remote Config [baseline] (617.913 µs) : 0, 618
Remote Config [candidate] (607.794 µs) : 0, 608
Telemetry [baseline] (8.399 ms) : 0, 8399
Telemetry [candidate] (8.394 ms) : 0, 8394
section profiling
crashtracking [baseline] (1.434 ms) : 0, 1434
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (763.568 ms) : 0, 763568
BytebuddyAgent [candidate] (768.748 ms) : 0, 768748
GlobalTracer [baseline] (223.71 ms) : 0, 223710
GlobalTracer [candidate] (224.715 ms) : 0, 224715
AppSec [baseline] (30.609 ms) : 0, 30609
AppSec [candidate] (30.776 ms) : 0, 30776
Debugger [baseline] (6.285 ms) : 0, 6285
Debugger [candidate] (6.311 ms) : 0, 6311
Remote Config [baseline] (724.067 µs) : 0, 724
Remote Config [candidate] (713.908 µs) : 0, 714
Telemetry [baseline] (16.539 ms) : 0, 16539
Telemetry [candidate] (15.861 ms) : 0, 15861
ProfilingAgent [baseline] (108.379 ms) : 0, 108379
ProfilingAgent [candidate] (110.675 ms) : 0, 110675
Profiling [baseline] (109.038 ms) : 0, 109038
Profiling [candidate] (111.344 ms) : 0, 111344
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064332
Total [baseline] (8.696 s) : 0, 8696422
Agent [candidate] (1.051 s) : 0, 1051416
Total [candidate] (8.664 s) : 0, 8663526
section iast
Agent [baseline] (1.18 s) : 0, 1179957
Total [baseline] (9.404 s) : 0, 9403686
Agent [candidate] (1.188 s) : 0, 1188110
Total [candidate] (9.38 s) : 0, 9380409
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.484 ms) : 0, 1484
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (743.297 ms) : 0, 743297
BytebuddyAgent [candidate] (735.367 ms) : 0, 735367
GlobalTracer [baseline] (245.875 ms) : 0, 245875
GlobalTracer [candidate] (244.057 ms) : 0, 244057
AppSec [baseline] (30.648 ms) : 0, 30648
AppSec [candidate] (30.355 ms) : 0, 30355
Debugger [baseline] (6.19 ms) : 0, 6190
Debugger [candidate] (6.129 ms) : 0, 6129
Remote Config [baseline] (710.217 µs) : 0, 710
Remote Config [candidate] (683.192 µs) : 0, 683
Telemetry [baseline] (14.807 ms) : 0, 14807
Telemetry [candidate] (12.199 ms) : 0, 12199
section iast
crashtracking [baseline] (1.47 ms) : 0, 1470
crashtracking [candidate] (1.474 ms) : 0, 1474
BytebuddyAgent [baseline] (852.179 ms) : 0, 852179
BytebuddyAgent [candidate] (857.247 ms) : 0, 857247
GlobalTracer [baseline] (232.918 ms) : 0, 232918
GlobalTracer [candidate] (234.655 ms) : 0, 234655
IAST [baseline] (30.784 ms) : 0, 30784
IAST [candidate] (30.644 ms) : 0, 30644
AppSec [baseline] (26.042 ms) : 0, 26042
AppSec [candidate] (27.101 ms) : 0, 27101
Debugger [baseline] (6.665 ms) : 0, 6665
Debugger [candidate] (6.747 ms) : 0, 6747
Remote Config [baseline] (605.174 µs) : 0, 605
Remote Config [candidate] (602.341 µs) : 0, 602
Telemetry [baseline] (8.296 ms) : 0, 8296
Telemetry [candidate] (8.365 ms) : 0, 8365
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 2 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section baseline
no_agent (36.46 ms) : 36166, 36753
. : milestone, 36460,
appsec (47.177 ms) : 46779, 47575
. : milestone, 47177,
code_origins (45.851 ms) : 45449, 46253
. : milestone, 45851,
iast (46.303 ms) : 45900, 46706
. : milestone, 46303,
profiling (48.207 ms) : 47729, 48684
. : milestone, 48207,
tracing (44.224 ms) : 43839, 44608
. : milestone, 44224,
section candidate
no_agent (37.441 ms) : 37136, 37747
. : milestone, 37441,
appsec (48.205 ms) : 47788, 48623
. : milestone, 48205,
code_origins (44.255 ms) : 43879, 44631
. : milestone, 44255,
iast (45.095 ms) : 44704, 45485
. : milestone, 45095,
profiling (50.081 ms) : 49625, 50538
. : milestone, 50081,
tracing (43.039 ms) : 42675, 43402
. : milestone, 43039,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section baseline
no_agent (4.252 ms) : 4195, 4309
. : milestone, 4252,
iast (9.464 ms) : 9306, 9623
. : milestone, 9464,
iast_FULL (13.697 ms) : 13423, 13972
. : milestone, 13697,
iast_GLOBAL (10.265 ms) : 10073, 10458
. : milestone, 10265,
profiling (9.28 ms) : 9134, 9426
. : milestone, 9280,
tracing (7.553 ms) : 7439, 7668
. : milestone, 7553,
section candidate
no_agent (4.437 ms) : 4386, 4488
. : milestone, 4437,
iast (9.355 ms) : 9190, 9520
. : milestone, 9355,
iast_FULL (13.867 ms) : 13590, 14143
. : milestone, 13867,
iast_GLOBAL (10.168 ms) : 9990, 10346
. : milestone, 10168,
profiling (8.467 ms) : 8336, 8598
. : milestone, 8467,
tracing (7.55 ms) : 7434, 7665
. : milestone, 7550,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section baseline
no_agent (1.482 ms) : 1470, 1493
. : milestone, 1482,
appsec (3.674 ms) : 3456, 3892
. : milestone, 3674,
iast (2.201 ms) : 2138, 2263
. : milestone, 2201,
iast_GLOBAL (2.258 ms) : 2195, 2322
. : milestone, 2258,
profiling (2.056 ms) : 2005, 2108
. : milestone, 2056,
tracing (2.014 ms) : 1966, 2063
. : milestone, 2014,
section candidate
no_agent (1.482 ms) : 1470, 1493
. : milestone, 1482,
appsec (2.473 ms) : 2421, 2526
. : milestone, 2473,
iast (2.211 ms) : 2148, 2274
. : milestone, 2211,
iast_GLOBAL (2.251 ms) : 2187, 2314
. : milestone, 2251,
profiling (2.04 ms) : 1990, 2090
. : milestone, 2040,
tracing (2.044 ms) : 1995, 2093
. : milestone, 2044,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~ae0456a231, baseline=1.54.0-SNAPSHOT~4abe3ff9a9
dateFormat X
axisFormat %s
section baseline
no_agent (15.093 s) : 15093000, 15093000
. : milestone, 15093000,
appsec (14.922 s) : 14922000, 14922000
. : milestone, 14922000,
iast (18.767 s) : 18767000, 18767000
. : milestone, 18767000,
iast_GLOBAL (18.099 s) : 18099000, 18099000
. : milestone, 18099000,
profiling (15.576 s) : 15576000, 15576000
. : milestone, 15576000,
tracing (14.756 s) : 14756000, 14756000
. : milestone, 14756000,
section candidate
no_agent (15.504 s) : 15504000, 15504000
. : milestone, 15504000,
appsec (15.104 s) : 15104000, 15104000
. : milestone, 15104000,
iast (18.344 s) : 18344000, 18344000
. : milestone, 18344000,
iast_GLOBAL (17.95 s) : 17950000, 17950000
. : milestone, 17950000,
profiling (15.216 s) : 15216000, 15216000
. : milestone, 15216000,
tracing (15.137 s) : 15137000, 15137000
. : milestone, 15137000,
|
daniel-mohedano
approved these changes
Sep 8, 2025
nikita-tkachenko-datadog
deleted the
nikita-tkachenko/fix-headless-status-propagation
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR fixes how test status propagation works in headless mode.
Testing frameworks may report suite/session status as successful even if there are failed tests (this behaviour is not consistent across frameworks).
When instrumenting build systems (Maven/Gradle) we get accurate statuses for modules and sessions from them.
However, for determining the statuses of test suites or modules/sessions in headless mode, we have to propagate test failures to their parent spans (and from the parents to parents of the parents and so on).
This generally works fine, but does not take into account that a test failure may be suppressed (e.g. if the test is quarantined or is going to be retried), in which case it has no impact on the status of its parent suite/module/session.
This PR fixes propagation logic: if a test failure is suppressed, its parent spans' statuses will not be suppressed.
Motivation
Reporting incorrect statuses in headless mode interferes with Test Health metrics: e.g. for logs backend we have 0 savings from ATR since any test session that contains a test that failed and was retried is considered failed, and ATR savings are only calculated for successful sessions.
Additional Notes
There's some refactoring to (hopefully) simplify the execution history/policy logic.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]