Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement flaky tests retry for Spock #6338

Merged
merged 2 commits into from
Dec 14, 2023
Merged

Implement flaky tests retry for Spock #6338

merged 2 commits into from
Dec 14, 2023

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Dec 10, 2023 •
edited
Loading

What Does This Do

Implements retrying flaky Spock Junit 5 tests.

Motivation

Adding flaky test retries feature to CI Visibility.

Jira ticket: CIVIS-8243

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 10, 2023
@pr-commenter
Copy link

pr-commenter bot commented Dec 10, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/retry-flaky-tests-spock
git_commit_date 1702479976 1702479621
git_commit_sha 4908b2c bdf2753
release_version 1.26.0-SNAPSHOT~4908b2cf3e 1.26.0-SNAPSHOT~bdf2753c05
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702482408 1702482408
ci_job_id 389046722 389046722
ci_pipeline_id 25032910 25032910
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 5 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.046 s) : 0, 1046072
Total [baseline] (8.692 s) : 0, 8692457
Agent [candidate] (1.047 s) : 0, 1047052
Total [candidate] (8.696 s) : 0, 8695533
section iast
Agent [baseline] (1.161 s) : 0, 1161049
Total [baseline] (9.255 s) : 0, 9255463
Agent [candidate] (1.166 s) : 0, 1166352
Total [candidate] (9.246 s) : 0, 9245824
section iast_TELEMETRY_OFF
Agent [baseline] (1.158 s) : 0, 1158044
Total [baseline] (9.237 s) : 0, 9236776
Agent [candidate] (1.161 s) : 0, 1160501
Total [candidate] (9.239 s) : 0, 9238689
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.046 s -
Agent iast 1.161 s 114.977 ms (11.0%)
Agent iast_TELEMETRY_OFF 1.158 s 111.972 ms (10.7%)
Total tracing 8.692 s -
Total iast 9.255 s 563.006 ms (6.5%)
Total iast_TELEMETRY_OFF 9.237 s 544.318 ms (6.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.047 s -
Agent iast 1.166 s 119.3 ms (11.4%)
Agent iast_TELEMETRY_OFF 1.161 s 113.449 ms (10.8%)
Total tracing 8.696 s -
Total iast 9.246 s 550.292 ms (6.3%)
Total iast_TELEMETRY_OFF 9.239 s 543.156 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.997 ms) : 0, 645997
BytebuddyAgent [candidate] (647.472 ms) : 0, 647472
GlobalTracer [baseline] (306.29 ms) : 0, 306290
GlobalTracer [candidate] (306.349 ms) : 0, 306349
AppSec [baseline] (51.561 ms) : 0, 51561
AppSec [candidate] (51.21 ms) : 0, 51210
Remote Config [baseline] (675.203 µs) : 0, 675
Remote Config [candidate] (681.191 µs) : 0, 681
Telemetry [baseline] (7.299 ms) : 0, 7299
Telemetry [candidate] (7.219 ms) : 0, 7219
section iast
BytebuddyAgent [baseline] (766.515 ms) : 0, 766515
BytebuddyAgent [candidate] (768.578 ms) : 0, 768578
GlobalTracer [baseline] (284.417 ms) : 0, 284417
GlobalTracer [candidate] (285.438 ms) : 0, 285438
AppSec [baseline] (49.084 ms) : 0, 49084
AppSec [candidate] (48.848 ms) : 0, 48848
IAST [baseline] (19.615 ms) : 0, 19615
IAST [candidate] (21.993 ms) : 0, 21993
Remote Config [baseline] (613.318 µs) : 0, 613
Remote Config [candidate] (605.475 µs) : 0, 605
Telemetry [baseline] (6.525 ms) : 0, 6525
Telemetry [candidate] (6.444 ms) : 0, 6444
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (761.081 ms) : 0, 761081
BytebuddyAgent [candidate] (761.821 ms) : 0, 761821
GlobalTracer [baseline] (285.782 ms) : 0, 285782
GlobalTracer [candidate] (286.165 ms) : 0, 286165
AppSec [baseline] (49.375 ms) : 0, 49375
AppSec [candidate] (49.286 ms) : 0, 49286
IAST [baseline] (19.403 ms) : 0, 19403
IAST [candidate] (19.337 ms) : 0, 19337
Remote Config [baseline] (615.681 µs) : 0, 616
Remote Config [candidate] (618.66 µs) : 0, 619
Telemetry [baseline] (7.414 ms) : 0, 7414
Telemetry [candidate] (8.825 ms) : 0, 8825
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050714
Total [baseline] (9.357 s) : 0, 9356873
Agent [candidate] (1.053 s) : 0, 1053384
Total [candidate] (9.284 s) : 0, 9284124
section appsec
Agent [baseline] (1.146 s) : 0, 1145819
Total [baseline] (9.483 s) : 0, 9482542
Agent [candidate] (1.153 s) : 0, 1152903
Total [candidate] (9.54 s) : 0, 9540053
section iast
Agent [baseline] (1.172 s) : 0, 1171567
Total [baseline] (9.52 s) : 0, 9519575
Agent [candidate] (1.168 s) : 0, 1167958
Total [candidate] (9.529 s) : 0, 9528547
section profiling
Agent [baseline] (1.237 s) : 0, 1237048
Total [baseline] (9.531 s) : 0, 9531277
Agent [candidate] (1.243 s) : 0, 1242565
Total [candidate] (9.526 s) : 0, 9525648
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.051 s -
Agent appsec 1.146 s 95.105 ms (9.1%)
Agent iast 1.172 s 120.852 ms (11.5%)
Agent profiling 1.237 s 186.334 ms (17.7%)
Total tracing 9.357 s -
Total appsec 9.483 s 125.669 ms (1.3%)
Total iast 9.52 s 162.702 ms (1.7%)
Total profiling 9.531 s 174.404 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent appsec 1.153 s 99.519 ms (9.4%)
Agent iast 1.168 s 114.574 ms (10.9%)
Agent profiling 1.243 s 189.181 ms (18.0%)
Total tracing 9.284 s -
Total appsec 9.54 s 255.929 ms (2.8%)
Total iast 9.529 s 244.422 ms (2.6%)
Total profiling 9.526 s 241.524 ms (2.6%) 
gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (649.407 ms) : 0, 649407
BytebuddyAgent [candidate] (651.324 ms) : 0, 651324
GlobalTracer [baseline] (307.408 ms) : 0, 307408
GlobalTracer [candidate] (307.973 ms) : 0, 307973
AppSec [baseline] (51.444 ms) : 0, 51444
AppSec [candidate] (51.78 ms) : 0, 51780
Remote Config [baseline] (677.159 µs) : 0, 677
Remote Config [candidate] (677.771 µs) : 0, 678
Telemetry [baseline] (7.238 ms) : 0, 7238
Telemetry [candidate] (7.266 ms) : 0, 7266
section appsec
BytebuddyAgent [baseline] (647.036 ms) : 0, 647036
BytebuddyAgent [candidate] (652.046 ms) : 0, 652046
GlobalTracer [baseline] (307.274 ms) : 0, 307274
GlobalTracer [candidate] (309.013 ms) : 0, 309013
AppSec [baseline] (149.66 ms) : 0, 149660
AppSec [candidate] (149.775 ms) : 0, 149775
Remote Config [baseline] (646.202 µs) : 0, 646
Remote Config [candidate] (654.866 µs) : 0, 655
Telemetry [baseline] (6.846 ms) : 0, 6846
Telemetry [candidate] (6.893 ms) : 0, 6893
section iast
BytebuddyAgent [baseline] (771.706 ms) : 0, 771706
BytebuddyAgent [candidate] (769.526 ms) : 0, 769526
GlobalTracer [baseline] (286.648 ms) : 0, 286648
GlobalTracer [candidate] (285.612 ms) : 0, 285612
AppSec [baseline] (49.406 ms) : 0, 49406
AppSec [candidate] (49.348 ms) : 0, 49348
IAST [baseline] (21.443 ms) : 0, 21443
IAST [candidate] (21.96 ms) : 0, 21960
Remote Config [baseline] (625.143 µs) : 0, 625
Remote Config [candidate] (632.828 µs) : 0, 633
Telemetry [baseline] (7.319 ms) : 0, 7319
Telemetry [candidate] (6.517 ms) : 0, 6517
section profiling
BytebuddyAgent [baseline] (655.132 ms) : 0, 655132
BytebuddyAgent [candidate] (657.863 ms) : 0, 657863
GlobalTracer [baseline] (376.069 ms) : 0, 376069
GlobalTracer [candidate] (376.673 ms) : 0, 376673
AppSec [baseline] (50.452 ms) : 0, 50452
AppSec [candidate] (51.368 ms) : 0, 51368
Remote Config [baseline] (662.987 µs) : 0, 663
Remote Config [candidate] (669.134 µs) : 0, 669
Telemetry [baseline] (7.452 ms) : 0, 7452
Telemetry [candidate] (7.401 ms) : 0, 7401
ProfilingAgent [baseline] (93.106 ms) : 0, 93106
ProfilingAgent [candidate] (94.48 ms) : 0, 94480
Profiling [baseline] (93.131 ms) : 0, 93131
Profiling [candidate] (94.505 ms) : 0, 94505
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-12-13T15:26:05 2023-12-13T15:42:36
git_branch master nikita-tkachenko/retry-flaky-tests-spock
git_commit_date 1702479976 1702479621
git_commit_sha 4908b2c bdf2753
release_version 1.26.0-SNAPSHOT~4908b2cf3e 1.26.0-SNAPSHOT~bdf2753c05
start_time 2023-12-13T15:25:52 2023-12-13T15:42:23
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702482408 1702482408
ci_job_id 389046722 389046722
ci_pipeline_id 25032910 25032910
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.181 µs) : 343, 385
.   : milestone, 364,
iast (466.61 µs) : 446, 487
.   : milestone, 467,
iast_FULL (530.887 µs) : 510, 552
.   : milestone, 531,
iast_INACTIVE (448.496 µs) : 428, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (459.474 µs) : 439, 480
.   : milestone, 459,
tracing (437.311 µs) : 417, 458
.   : milestone, 437,
section candidate
no_agent (361.322 µs) : 342, 381
.   : milestone, 361,
iast (470.331 µs) : 450, 491
.   : milestone, 470,
iast_FULL (534.371 µs) : 514, 555
.   : milestone, 534,
iast_INACTIVE (433.695 µs) : 413, 454
.   : milestone, 434,
iast_TELEMETRY_OFF (464.764 µs) : 443, 486
.   : milestone, 465,
tracing (438.822 µs) : 417, 460
.   : milestone, 439,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.181 µs [343.445 µs, 384.917 µs] -
iast 466.61 µs [446.074 µs, 487.147 µs] 102.429 µs (28.1%)
iast_FULL 530.887 µs [509.844 µs, 551.93 µs] 166.706 µs (45.8%)
iast_INACTIVE 448.496 µs [427.577 µs, 469.416 µs] 84.315 µs (23.2%)
iast_TELEMETRY_OFF 459.474 µs [439.013 µs, 479.934 µs] 95.293 µs (26.2%)
tracing 437.311 µs [416.959 µs, 457.664 µs] 73.13 µs (20.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 361.322 µs [341.699 µs, 380.945 µs] -
iast 470.331 µs [449.506 µs, 491.155 µs] 109.009 µs (30.2%)
iast_FULL 534.371 µs [513.979 µs, 554.763 µs] 173.049 µs (47.9%)
iast_INACTIVE 433.695 µs [413.267 µs, 454.123 µs] 72.373 µs (20.0%)
iast_TELEMETRY_OFF 464.764 µs [443.448 µs, 486.08 µs] 103.442 µs (28.6%)
tracing 438.822 µs [417.477 µs, 460.166 µs] 77.5 µs (21.4%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~bdf2753c05, baseline=1.26.0-SNAPSHOT~4908b2cf3e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.354 ms) : 1335, 1373
.   : milestone, 1354,
appsec (1.748 ms) : 1723, 1773
.   : milestone, 1748,
iast (1.524 ms) : 1500, 1548
.   : milestone, 1524,
profiling (1.562 ms) : 1536, 1588
.   : milestone, 1562,
tracing (1.477 ms) : 1453, 1502
.   : milestone, 1477,
section candidate
no_agent (1.341 ms) : 1322, 1360
.   : milestone, 1341,
appsec (1.755 ms) : 1730, 1781
.   : milestone, 1755,
iast (1.516 ms) : 1492, 1540
.   : milestone, 1516,
profiling (1.514 ms) : 1490, 1539
.   : milestone, 1514,
tracing (1.499 ms) : 1474, 1525
.   : milestone, 1499,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.354 ms [1.335 ms, 1.373 ms] -
appsec 1.748 ms [1.723 ms, 1.773 ms] 394.107 µs (29.1%)
iast 1.524 ms [1.5 ms, 1.548 ms] 170.301 µs (12.6%)
profiling 1.562 ms [1.536 ms, 1.588 ms] 208.036 µs (15.4%)
tracing 1.477 ms [1.453 ms, 1.502 ms] 123.594 µs (9.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.341 ms [1.322 ms, 1.36 ms] -
appsec 1.755 ms [1.73 ms, 1.781 ms] 413.828 µs (30.9%)
iast 1.516 ms [1.492 ms, 1.54 ms] 174.71 µs (13.0%)
profiling 1.514 ms [1.49 ms, 1.539 ms] 172.856 µs (12.9%)
tracing 1.499 ms [1.474 ms, 1.525 ms] 158.081 µs (11.8%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review December 10, 2023 17:40
Base automatically changed from nikita-tkachenko/retry-flaky-tests to master December 13, 2023 14:59
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests-spock branch from 1988553 to bdf2753 Compare December 13, 2023 15:00
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit a445d05 into master Dec 14, 2023
73 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/retry-flaky-tests-spock branch December 14, 2023 16:52
@github-actions github-actions bot added this to the 1.26.0 milestone Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes

@bantonsson bantonsson Awaiting requested review from bantonsson bantonsson was automatically assigned from DataDog/apm-java

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal was automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility
Projects
None yet
Milestone
1.26.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @anmarchenko