Implement flaky tests retry for Cucumber JUnit 5 #6339

nikita-tkachenko-datadog · 2023-12-10T16:11:09Z

What Does This Do

Implements retrying flaky Cucumber Junit 5 tests.

Motivation

Adding flaky test retries feature to CI Visibility.

Jira ticket: CIVIS-8243

pr-commenter · 2023-12-10T17:09:13Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/retry-flaky-tests-cucumber-junit-5
git_commit_date	1702310882	1702325677
git_commit_sha	`163ba8a`	`777177d`
release_version	1.26.0-SNAPSHOT~163ba8a996	1.26.0-SNAPSHOT~777177d886

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702328729	1702328729
ci_job_id	387087754	387087754
ci_pipeline_id	24836285	24836285
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049276
Total [baseline] (9.312 s) : 0, 9312287
Agent [candidate] (1.051 s) : 0, 1050810
Total [candidate] (9.334 s) : 0, 9334316
section appsec
Agent [baseline] (1.129 s) : 0, 1129137
Total [baseline] (9.443 s) : 0, 9442729
Agent [candidate] (1.138 s) : 0, 1138375
Total [candidate] (9.517 s) : 0, 9516650
section iast
Agent [baseline] (1.162 s) : 0, 1161722
Total [baseline] (9.588 s) : 0, 9587842
Agent [candidate] (1.17 s) : 0, 1169560
Total [candidate] (9.632 s) : 0, 9631882
section profiling
Agent [baseline] (1.237 s) : 0, 1237368
Total [baseline] (9.639 s) : 0, 9638649
Agent [candidate] (1.235 s) : 0, 1235153
Total [candidate] (9.726 s) : 0, 9726486

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.129 s	79.861 ms (7.6%)
Agent	iast	1.162 s	112.446 ms (10.7%)
Agent	profiling	1.237 s	188.092 ms (17.9%)
Total	tracing	9.312 s	-
Total	appsec	9.443 s	130.442 ms (1.4%)
Total	iast	9.588 s	275.555 ms (3.0%)
Total	profiling	9.639 s	326.362 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	appsec	1.138 s	87.565 ms (8.3%)
Agent	iast	1.17 s	118.75 ms (11.3%)
Agent	profiling	1.235 s	184.343 ms (17.5%)
Total	tracing	9.334 s	-
Total	appsec	9.517 s	182.334 ms (2.0%)
Total	iast	9.632 s	297.566 ms (3.2%)
Total	profiling	9.726 s	392.17 ms (4.2%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (649.29 ms) : 0, 649290
BytebuddyAgent [candidate] (650.803 ms) : 0, 650803
GlobalTracer [baseline] (308.573 ms) : 0, 308573
GlobalTracer [candidate] (309.224 ms) : 0, 309224
AppSec [baseline] (49.121 ms) : 0, 49121
AppSec [candidate] (48.611 ms) : 0, 48611
Remote Config [baseline] (684.399 µs) : 0, 684
Remote Config [candidate] (677.789 µs) : 0, 678
Telemetry [baseline] (7.138 ms) : 0, 7138
Telemetry [candidate] (7.118 ms) : 0, 7118
section appsec
BytebuddyAgent [baseline] (644.301 ms) : 0, 644301
BytebuddyAgent [candidate] (650.566 ms) : 0, 650566
GlobalTracer [baseline] (305.784 ms) : 0, 305784
GlobalTracer [candidate] (308.344 ms) : 0, 308344
AppSec [baseline] (136.849 ms) : 0, 136849
AppSec [candidate] (137.55 ms) : 0, 137550
Remote Config [baseline] (639.936 µs) : 0, 640
Remote Config [candidate] (650.163 µs) : 0, 650
Telemetry [baseline] (7.442 ms) : 0, 7442
Telemetry [candidate] (6.856 ms) : 0, 6856
section iast
BytebuddyAgent [baseline] (765.792 ms) : 0, 765792
BytebuddyAgent [candidate] (772.912 ms) : 0, 772912
GlobalTracer [baseline] (286.137 ms) : 0, 286137
GlobalTracer [candidate] (287.634 ms) : 0, 287634
AppSec [baseline] (46.505 ms) : 0, 46505
AppSec [candidate] (47.167 ms) : 0, 47167
Remote Config [baseline] (606.557 µs) : 0, 607
Remote Config [candidate] (605.63 µs) : 0, 606
Telemetry [baseline] (6.576 ms) : 0, 6576
Telemetry [candidate] (6.594 ms) : 0, 6594
IAST [baseline] (21.756 ms) : 0, 21756
IAST [candidate] (20.151 ms) : 0, 20151
section profiling
ProfilingAgent [baseline] (88.316 ms) : 0, 88316
ProfilingAgent [candidate] (88.586 ms) : 0, 88586
BytebuddyAgent [baseline] (659.209 ms) : 0, 659209
BytebuddyAgent [candidate] (657.362 ms) : 0, 657362
GlobalTracer [baseline] (378.754 ms) : 0, 378754
GlobalTracer [candidate] (378.393 ms) : 0, 378393
AppSec [baseline] (48.651 ms) : 0, 48651
AppSec [candidate] (48.67 ms) : 0, 48670
Remote Config [baseline] (676.012 µs) : 0, 676
Remote Config [candidate] (690.391 µs) : 0, 690
Telemetry [baseline] (7.421 ms) : 0, 7421
Telemetry [candidate] (7.499 ms) : 0, 7499
Profiling [baseline] (88.341 ms) : 0, 88341
Profiling [candidate] (88.611 ms) : 0, 88611

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048637
Total [baseline] (8.742 s) : 0, 8741779
Agent [candidate] (1.05 s) : 0, 1050028
Total [candidate] (8.726 s) : 0, 8726247
section iast
Agent [baseline] (1.157 s) : 0, 1156742
Total [baseline] (9.316 s) : 0, 9315899
Agent [candidate] (1.163 s) : 0, 1162709
Total [candidate] (9.263 s) : 0, 9262876
section iast_TELEMETRY_OFF
Agent [baseline] (1.148 s) : 0, 1147760
Total [baseline] (9.262 s) : 0, 9262039
Agent [candidate] (1.153 s) : 0, 1152927
Total [candidate] (9.296 s) : 0, 9296120

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.157 s	108.105 ms (10.3%)
Agent	iast_TELEMETRY_OFF	1.148 s	99.122 ms (9.5%)
Total	tracing	8.742 s	-
Total	iast	9.316 s	574.12 ms (6.6%)
Total	iast_TELEMETRY_OFF	9.262 s	520.261 ms (6.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.05 s	-
Agent	iast	1.163 s	112.68 ms (10.7%)
Agent	iast_TELEMETRY_OFF	1.153 s	102.899 ms (9.8%)
Total	tracing	8.726 s	-
Total	iast	9.263 s	536.63 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.296 s	569.874 ms (6.5%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (648.69 ms) : 0, 648690
BytebuddyAgent [candidate] (650.649 ms) : 0, 650649
GlobalTracer [baseline] (308.843 ms) : 0, 308843
GlobalTracer [candidate] (308.503 ms) : 0, 308503
AppSec [baseline] (48.793 ms) : 0, 48793
AppSec [candidate] (48.734 ms) : 0, 48734
Remote Config [baseline] (683.715 µs) : 0, 684
Remote Config [candidate] (681.321 µs) : 0, 681
Telemetry [baseline] (7.161 ms) : 0, 7161
Telemetry [candidate] (7.089 ms) : 0, 7089
section iast
BytebuddyAgent [baseline] (764.751 ms) : 0, 764751
BytebuddyAgent [candidate] (767.384 ms) : 0, 767384
GlobalTracer [baseline] (283.945 ms) : 0, 283945
GlobalTracer [candidate] (285.267 ms) : 0, 285267
AppSec [baseline] (46.203 ms) : 0, 46203
AppSec [candidate] (46.935 ms) : 0, 46935
IAST [baseline] (19.573 ms) : 0, 19573
IAST [candidate] (20.289 ms) : 0, 20289
Remote Config [baseline] (612.859 µs) : 0, 613
Remote Config [candidate] (597.25 µs) : 0, 597
Telemetry [baseline] (7.475 ms) : 0, 7475
Telemetry [candidate] (7.961 ms) : 0, 7961
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (757.064 ms) : 0, 757064
BytebuddyAgent [candidate] (760.028 ms) : 0, 760028
GlobalTracer [baseline] (284.213 ms) : 0, 284213
GlobalTracer [candidate] (285.754 ms) : 0, 285754
AppSec [baseline] (46.003 ms) : 0, 46003
AppSec [candidate] (46.032 ms) : 0, 46032
IAST [baseline] (16.217 ms) : 0, 16217
IAST [candidate] (19.027 ms) : 0, 19027
Remote Config [baseline] (597.571 µs) : 0, 598
Remote Config [candidate] (598.365 µs) : 0, 598
Telemetry [baseline] (9.526 ms) : 0, 9526
Telemetry [candidate] (7.292 ms) : 0, 7292

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-11T20:44:40	2023-12-11T21:01:16
git_branch	master	nikita-tkachenko/retry-flaky-tests-cucumber-junit-5
git_commit_date	1702310882	1702325677
git_commit_sha	`163ba8a`	`777177d`
release_version	1.26.0-SNAPSHOT~163ba8a996	1.26.0-SNAPSHOT~777177d886
start_time	2023-12-11T20:44:26	2023-12-11T21:01:03

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702328729	1702328729
ci_job_id	387087754	387087754
ci_pipeline_id	24836285	24836285
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.367 ms) : 1348, 1387
.   : milestone, 1367,
appsec (1.76 ms) : 1735, 1785
.   : milestone, 1760,
iast (1.53 ms) : 1505, 1555
.   : milestone, 1530,
profiling (1.53 ms) : 1504, 1556
.   : milestone, 1530,
tracing (1.484 ms) : 1460, 1509
.   : milestone, 1484,
section candidate
no_agent (1.346 ms) : 1327, 1365
.   : milestone, 1346,
appsec (1.756 ms) : 1731, 1781
.   : milestone, 1756,
iast (1.54 ms) : 1515, 1565
.   : milestone, 1540,
profiling (1.509 ms) : 1483, 1534
.   : milestone, 1509,
tracing (1.504 ms) : 1479, 1528
.   : milestone, 1504,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.367 ms [1.348 ms, 1.387 ms]	-
appsec	1.76 ms [1.735 ms, 1.785 ms]	392.738 µs (28.7%)
iast	1.53 ms [1.505 ms, 1.555 ms]	162.908 µs (11.9%)
profiling	1.53 ms [1.504 ms, 1.556 ms]	162.746 µs (11.9%)
tracing	1.484 ms [1.46 ms, 1.509 ms]	117.265 µs (8.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.346 ms [1.327 ms, 1.365 ms]	-
appsec	1.756 ms [1.731 ms, 1.781 ms]	410.17 µs (30.5%)
iast	1.54 ms [1.515 ms, 1.565 ms]	194.149 µs (14.4%)
profiling	1.509 ms [1.483 ms, 1.534 ms]	163.078 µs (12.1%)
tracing	1.504 ms [1.479 ms, 1.528 ms]	157.626 µs (11.7%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~777177d886, baseline=1.26.0-SNAPSHOT~163ba8a996
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.718 µs) : 348, 388
.   : milestone, 368,
iast (476.56 µs) : 456, 497
.   : milestone, 477,
iast_FULL (534.512 µs) : 514, 555
.   : milestone, 535,
iast_INACTIVE (448.638 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (467.473 µs) : 447, 488
.   : milestone, 467,
tracing (439.214 µs) : 419, 459
.   : milestone, 439,
section candidate
no_agent (364.549 µs) : 345, 385
.   : milestone, 365,
iast (477.61 µs) : 457, 498
.   : milestone, 478,
iast_FULL (536.126 µs) : 516, 557
.   : milestone, 536,
iast_INACTIVE (447.916 µs) : 427, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (466.013 µs) : 446, 486
.   : milestone, 466,
tracing (445.575 µs) : 424, 467
.   : milestone, 446,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.718 µs [347.557 µs, 387.88 µs]	-
iast	476.56 µs [456.287 µs, 496.833 µs]	108.842 µs (29.6%)
iast_FULL	534.512 µs [513.553 µs, 555.472 µs]	166.794 µs (45.4%)
iast_INACTIVE	448.638 µs [427.589 µs, 469.686 µs]	80.92 µs (22.0%)
iast_TELEMETRY_OFF	467.473 µs [446.839 µs, 488.108 µs]	99.755 µs (27.1%)
tracing	439.214 µs [419.204 µs, 459.225 µs]	71.496 µs (19.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.549 µs [344.521 µs, 384.578 µs]	-
iast	477.61 µs [456.781 µs, 498.439 µs]	113.06 µs (31.0%)
iast_FULL	536.126 µs [515.541 µs, 556.711 µs]	171.577 µs (47.1%)
iast_INACTIVE	447.916 µs [426.58 µs, 469.251 µs]	83.366 µs (22.9%)
iast_TELEMETRY_OFF	466.013 µs [445.718 µs, 486.309 µs]	101.464 µs (27.8%)
tracing	445.575 µs [424.151 µs, 466.998 µs]	81.025 µs (22.2%)

…r-junit-5

nikita-tkachenko-datadog added 4 commits December 9, 2023 17:16

Implement flaky test retries for JUnit 5.3

18c21ce

Fix CI failures

a5d9bda

Implement flaky tests retry for Spock

1988553

Implement flaky tests retry for Cucumber JUnit 5

6ccfd48

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 10, 2023

nikita-tkachenko-datadog added 2 commits December 10, 2023 21:43

Fix Muzzle

08d5bb3

Fix expected tests data

777177d

nikita-tkachenko-datadog marked this pull request as ready for review December 12, 2023 09:56

nikita-tkachenko-datadog requested a review from a team as a code owner December 12, 2023 09:56

nikita-tkachenko-datadog requested a review from romainkomorndatadog December 12, 2023 09:56

romainkomorndatadog approved these changes Dec 12, 2023

View reviewed changes

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests-spock branch from 1988553 to bdf2753 Compare December 13, 2023 15:00

nikita-tkachenko-datadog requested a review from a team as a code owner December 13, 2023 15:00

nikita-tkachenko-datadog requested review from bantonsson and am312 December 13, 2023 15:00

Base automatically changed from nikita-tkachenko/retry-flaky-tests-spock to master December 14, 2023 16:52

Merge branch 'master' into nikita-tkachenko/retry-flaky-tests-cucumbe…

284675d

…r-junit-5

nikita-tkachenko-datadog merged commit 61aca5c into master Dec 16, 2023
72 of 73 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/retry-flaky-tests-cucumber-junit-5 branch December 16, 2023 18:41

github-actions bot added this to the 1.26.0 milestone Dec 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement flaky tests retry for Cucumber JUnit 5 #6339

Implement flaky tests retry for Cucumber JUnit 5 #6339

nikita-tkachenko-datadog commented Dec 10, 2023

pr-commenter bot commented Dec 10, 2023 •

edited

Loading

Implement flaky tests retry for Cucumber JUnit 5 #6339

Implement flaky tests retry for Cucumber JUnit 5 #6339

Conversation

nikita-tkachenko-datadog commented Dec 10, 2023

What Does This Do

Motivation

pr-commenter bot commented Dec 10, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

pr-commenter bot commented Dec 10, 2023 •

edited

Loading