Add session level metrics for Failed Test Replay #9531
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daniel-mohedano
added
type: enhancement
|
🎯 Code Coverage 🔗 Commit SHA: a40166a | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 12 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1062048
Total [baseline] (10.73 s) : 0, 10730236
Agent [candidate] (1.069 s) : 0, 1068732
Total [candidate] (10.687 s) : 0, 10686780
section appsec
Agent [baseline] (1.235 s) : 0, 1235115
Total [baseline] (11.035 s) : 0, 11034834
Agent [candidate] (1.233 s) : 0, 1233044
Total [candidate] (10.819 s) : 0, 10818904
section iast
Agent [baseline] (1.189 s) : 0, 1189329
Total [baseline] (10.968 s) : 0, 10967883
Agent [candidate] (1.198 s) : 0, 1197750
Total [candidate] (10.979 s) : 0, 10978795
section profiling
Agent [baseline] (1.21 s) : 0, 1209506
Total [baseline] (10.889 s) : 0, 10889483
Agent [candidate] (1.206 s) : 0, 1206361
Total [candidate] (10.829 s) : 0, 10828758
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.471 ms) : 0, 1471
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (733.18 ms) : 0, 733180
BytebuddyAgent [candidate] (738.533 ms) : 0, 738533
GlobalTracer [baseline] (252.305 ms) : 0, 252305
GlobalTracer [candidate] (253.976 ms) : 0, 253976
AppSec [baseline] (30.554 ms) : 0, 30554
AppSec [candidate] (30.763 ms) : 0, 30763
Debugger [baseline] (6.349 ms) : 0, 6349
Debugger [candidate] (6.42 ms) : 0, 6420
Remote Config [baseline] (711.224 µs) : 0, 711
Remote Config [candidate] (693.317 µs) : 0, 693
Telemetry [baseline] (16.368 ms) : 0, 16368
Telemetry [candidate] (15.633 ms) : 0, 15633
section appsec
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.442 ms) : 0, 1442
BytebuddyAgent [baseline] (756.398 ms) : 0, 756398
BytebuddyAgent [candidate] (755.886 ms) : 0, 755886
GlobalTracer [baseline] (245.686 ms) : 0, 245686
GlobalTracer [candidate] (244.96 ms) : 0, 244960
IAST [baseline] (23.768 ms) : 0, 23768
IAST [candidate] (23.755 ms) : 0, 23755
AppSec [baseline] (171.402 ms) : 0, 171402
AppSec [candidate] (170.847 ms) : 0, 170847
Debugger [baseline] (6.028 ms) : 0, 6028
Debugger [candidate] (5.993 ms) : 0, 5993
Remote Config [baseline] (631.559 µs) : 0, 632
Remote Config [candidate] (630.043 µs) : 0, 630
Telemetry [baseline] (8.526 ms) : 0, 8526
Telemetry [candidate] (8.382 ms) : 0, 8382
section iast
crashtracking [baseline] (1.453 ms) : 0, 1453
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (851.679 ms) : 0, 851679
BytebuddyAgent [candidate] (858.873 ms) : 0, 858873
GlobalTracer [baseline] (246.884 ms) : 0, 246884
GlobalTracer [candidate] (247.224 ms) : 0, 247224
IAST [baseline] (27.116 ms) : 0, 27116
IAST [candidate] (29.551 ms) : 0, 29551
AppSec [baseline] (26.392 ms) : 0, 26392
AppSec [candidate] (24.694 ms) : 0, 24694
Debugger [baseline] (6.034 ms) : 0, 6034
Debugger [candidate] (6.048 ms) : 0, 6048
Remote Config [baseline] (586.072 µs) : 0, 586
Remote Config [candidate] (599.952 µs) : 0, 600
Telemetry [baseline] (8.15 ms) : 0, 8150
Telemetry [candidate] (8.175 ms) : 0, 8175
section profiling
crashtracking [baseline] (1.436 ms) : 0, 1436
crashtracking [candidate] (1.419 ms) : 0, 1419
BytebuddyAgent [baseline] (761.254 ms) : 0, 761254
BytebuddyAgent [candidate] (759.883 ms) : 0, 759883
GlobalTracer [baseline] (233.311 ms) : 0, 233311
GlobalTracer [candidate] (233.09 ms) : 0, 233090
AppSec [baseline] (30.462 ms) : 0, 30462
AppSec [candidate] (30.325 ms) : 0, 30325
Debugger [baseline] (9.024 ms) : 0, 9024
Debugger [candidate] (8.925 ms) : 0, 8925
Remote Config [baseline] (752.273 µs) : 0, 752
Remote Config [candidate] (710.927 µs) : 0, 711
Telemetry [baseline] (14.196 ms) : 0, 14196
Telemetry [candidate] (14.004 ms) : 0, 14004
ProfilingAgent [baseline] (108.173 ms) : 0, 108173
ProfilingAgent [candidate] (106.976 ms) : 0, 106976
Profiling [baseline] (108.847 ms) : 0, 108847
Profiling [candidate] (107.653 ms) : 0, 107653
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058638
Total [baseline] (8.646 s) : 0, 8645650
Agent [candidate] (1.061 s) : 0, 1061467
Total [candidate] (8.618 s) : 0, 8617607
section iast
Agent [baseline] (1.187 s) : 0, 1187385
Total [baseline] (9.309 s) : 0, 9309211
Agent [candidate] (1.191 s) : 0, 1190652
Total [candidate] (9.268 s) : 0, 9267673
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.449 ms) : 0, 1449
BytebuddyAgent [baseline] (731.192 ms) : 0, 731192
BytebuddyAgent [candidate] (732.556 ms) : 0, 732556
GlobalTracer [baseline] (252.002 ms) : 0, 252002
GlobalTracer [candidate] (252.233 ms) : 0, 252233
AppSec [baseline] (30.492 ms) : 0, 30492
AppSec [candidate] (30.747 ms) : 0, 30747
Debugger [baseline] (6.375 ms) : 0, 6375
Debugger [candidate] (6.403 ms) : 0, 6403
Remote Config [baseline] (696.915 µs) : 0, 697
Remote Config [candidate] (693.622 µs) : 0, 694
Telemetry [baseline] (15.472 ms) : 0, 15472
Telemetry [candidate] (16.373 ms) : 0, 16373
section iast
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (850.419 ms) : 0, 850419
BytebuddyAgent [candidate] (852.971 ms) : 0, 852971
GlobalTracer [baseline] (245.644 ms) : 0, 245644
GlobalTracer [candidate] (247.275 ms) : 0, 247275
IAST [baseline] (28.971 ms) : 0, 28971
IAST [candidate] (27.913 ms) : 0, 27913
AppSec [baseline] (24.533 ms) : 0, 24533
AppSec [candidate] (25.305 ms) : 0, 25305
Debugger [baseline] (5.984 ms) : 0, 5984
Debugger [candidate] (6.053 ms) : 0, 6053
Remote Config [baseline] (589.794 µs) : 0, 590
Remote Config [candidate] (600.463 µs) : 0, 600
Telemetry [baseline] (8.087 ms) : 0, 8087
Telemetry [candidate] (8.19 ms) : 0, 8190
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 3 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section baseline
no_agent (37.651 ms) : 37342, 37960
. : milestone, 37651,
appsec (48.679 ms) : 48254, 49104
. : milestone, 48679,
code_origins (44.299 ms) : 43924, 44674
. : milestone, 44299,
iast (44.225 ms) : 43846, 44604
. : milestone, 44225,
profiling (47.413 ms) : 46979, 47848
. : milestone, 47413,
tracing (46.119 ms) : 45723, 46516
. : milestone, 46119,
section candidate
no_agent (36.611 ms) : 36318, 36905
. : milestone, 36611,
appsec (48.88 ms) : 48452, 49308
. : milestone, 48880,
code_origins (45.668 ms) : 45285, 46050
. : milestone, 45668,
iast (44.685 ms) : 44308, 45062
. : milestone, 44685,
profiling (50.119 ms) : 49666, 50572
. : milestone, 50119,
tracing (45.482 ms) : 45104, 45860
. : milestone, 45482,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section baseline
no_agent (4.358 ms) : 4309, 4407
. : milestone, 4358,
iast (10.055 ms) : 9874, 10236
. : milestone, 10055,
iast_FULL (13.972 ms) : 13696, 14247
. : milestone, 13972,
iast_GLOBAL (10.473 ms) : 10288, 10659
. : milestone, 10473,
profiling (9.249 ms) : 9094, 9404
. : milestone, 9249,
tracing (7.573 ms) : 7467, 7680
. : milestone, 7573,
section candidate
no_agent (4.325 ms) : 4277, 4374
. : milestone, 4325,
iast (9.831 ms) : 9660, 10001
. : milestone, 9831,
iast_FULL (14.686 ms) : 14397, 14975
. : milestone, 14686,
iast_GLOBAL (10.805 ms) : 10612, 10998
. : milestone, 10805,
profiling (9.108 ms) : 8938, 9278
. : milestone, 9108,
tracing (7.701 ms) : 7585, 7816
. : milestone, 7701,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section baseline
no_agent (14.922 s) : 14922000, 14922000
. : milestone, 14922000,
appsec (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
iast (18.291 s) : 18291000, 18291000
. : milestone, 18291000,
iast_GLOBAL (17.892 s) : 17892000, 17892000
. : milestone, 17892000,
profiling (15.496 s) : 15496000, 15496000
. : milestone, 15496000,
tracing (14.904 s) : 14904000, 14904000
. : milestone, 14904000,
section candidate
no_agent (15.479 s) : 15479000, 15479000
. : milestone, 15479000,
appsec (14.802 s) : 14802000, 14802000
. : milestone, 14802000,
iast (18.625 s) : 18625000, 18625000
. : milestone, 18625000,
iast_GLOBAL (17.869 s) : 17869000, 17869000
. : milestone, 17869000,
profiling (15.393 s) : 15393000, 15393000
. : milestone, 15393000,
tracing (14.861 s) : 14861000, 14861000
. : milestone, 14861000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~a40166adff, baseline=1.54.0-SNAPSHOT~dacc5a50ca
dateFormat X
axisFormat %s
section baseline
no_agent (1.479 ms) : 1468, 1491
. : milestone, 1479,
appsec (3.721 ms) : 3503, 3939
. : milestone, 3721,
iast (2.211 ms) : 2148, 2273
. : milestone, 2211,
iast_GLOBAL (2.252 ms) : 2189, 2315
. : milestone, 2252,
profiling (2.068 ms) : 2016, 2120
. : milestone, 2068,
tracing (2.026 ms) : 1977, 2075
. : milestone, 2026,
section candidate
no_agent (1.48 ms) : 1468, 1491
. : milestone, 1480,
appsec (2.522 ms) : 2469, 2575
. : milestone, 2522,
iast (2.21 ms) : 2147, 2272
. : milestone, 2210,
iast_GLOBAL (2.243 ms) : 2180, 2305
. : milestone, 2243,
profiling (2.05 ms) : 1999, 2101
. : milestone, 2050,
tracing (2.028 ms) : 1979, 2077
. : milestone, 2028,
|
daniel-mohedano
commented
Sep 15, 2025
Comment on lines
-191
to
+203
| return Collections.emptyList(); | ||
| Set<TagValue> tags = new HashSet<>(); | ||
| if (Constants.EFD_ABORT_REASON_FAULTY.equals(span.getTag(Tags.TEST_EARLY_FLAKE_ABORT_REASON))) { | ||
| tags.add(EarlyFlakeDetectionAbortReason.FAULTY); | ||
| } | ||
| if (span.getTag(DDTags.TEST_HAS_FAILED_TEST_REPLAY) != null) { | ||
| tags.add(FailedTestReplayEnabled.SessionMetric.TRUE); | ||
| } | ||
| return tags; |
There was a problem hiding this comment.
Seems like we always want to follow the same logic in HeadlessTestSession and BuildSystemSessionImpl. Was this previously implemented in overriden methods to avoid performing the logic in ManualApiTestSession?
There was a problem hiding this comment.
I cannot recall now, but it's fine to have this in logic in the parent class
nikita-tkachenko-datadog
approved these changes
Sep 15, 2025
Comment on lines
-191
to
+203
| return Collections.emptyList(); | ||
| Set<TagValue> tags = new HashSet<>(); | ||
| if (Constants.EFD_ABORT_REASON_FAULTY.equals(span.getTag(Tags.TEST_EARLY_FLAKE_ABORT_REASON))) { | ||
| tags.add(EarlyFlakeDetectionAbortReason.FAULTY); | ||
| } | ||
| if (span.getTag(DDTags.TEST_HAS_FAILED_TEST_REPLAY) != null) { | ||
| tags.add(FailedTestReplayEnabled.SessionMetric.TRUE); | ||
| } | ||
| return tags; |
There was a problem hiding this comment.
I cannot recall now, but it's fine to have this in logic in the parent class
mhlidd
pushed a commit
that referenced
this pull request
Sep 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
has_failed_test_replayfield inevent_finishedmetrics when the session contained at least one test with Failed Test Replay debug info captured.Motivation
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]