Skip to content

Disabled flaky tests for IBM JVM #7527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jpbempel merged 1 commit into from
Aug 30, 2024
Merged

Disabled flaky tests for IBM JVM #7527

jpbempel merged 1 commit into from
Aug 30, 2024

Conversation

@jpbempel
Copy link
Member

@jpbempel jpbempel commented Aug 28, 2024

What Does This Do

Motivation

Falky tests

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@jpbempel jpbempel marked this pull request as ready for review August 28, 2024 14:54
@jpbempel jpbempel requested a review from a team as a code owner August 28, 2024 14:54
@jpbempel jpbempel requested review from ojung and removed request for a team August 28, 2024 14:54
@jpbempel jpbempel added comp: debugger Dynamic Instrumentation comp: testing Testing labels Aug 28, 2024
@pr-commenter
Copy link

pr-commenter bot commented Aug 28, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-flaky-smoke-tests
git_commit_date 1724852994 1724856829
git_commit_sha 78086bf 882d821
release_version 1.39.0-SNAPSHOT~78086bff11 1.39.0-SNAPSHOT~882d82103d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1724859198 1724859198
ci_job_id 620156541 620156541
ci_pipeline_id 43044748 43044748
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 13 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050780
Total [baseline] (10.549 s) : 0, 10549099
Agent [candidate] (1.064 s) : 0, 1064312
Total [candidate] (10.636 s) : 0, 10635599
section appsec
Agent [baseline] (1.172 s) : 0, 1171917
Total [baseline] (10.467 s) : 0, 10466691
Agent [candidate] (1.173 s) : 0, 1173465
Total [candidate] (10.485 s) : 0, 10485340
section iast
Agent [baseline] (1.187 s) : 0, 1186639
Total [baseline] (11.053 s) : 0, 11053375
Agent [candidate] (1.175 s) : 0, 1175121
Total [candidate] (10.835 s) : 0, 10834919
section profiling
Agent [baseline] (1.257 s) : 0, 1256692
Total [baseline] (10.615 s) : 0, 10615420
Agent [candidate] (1.26 s) : 0, 1259710
Total [candidate] (10.668 s) : 0, 10667871
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.051 s -
Agent appsec 1.172 s 121.137 ms (11.5%)
Agent iast 1.187 s 135.859 ms (12.9%)
Agent profiling 1.257 s 205.912 ms (19.6%)
Total tracing 10.549 s -
Total appsec 10.467 s -82.408 ms (-0.8%)
Total iast 11.053 s 504.276 ms (4.8%)
Total profiling 10.615 s 66.321 ms (0.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent appsec 1.173 s 109.153 ms (10.3%)
Agent iast 1.175 s 110.81 ms (10.4%)
Agent profiling 1.26 s 195.398 ms (18.4%)
Total tracing 10.636 s -
Total appsec 10.485 s -150.259 ms (-1.4%)
Total iast 10.835 s 199.32 ms (1.9%)
Total profiling 10.668 s 32.272 ms (0.3%) 
gantt
    title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.595 ms) : 0, 670595
BytebuddyAgent [candidate] (679.378 ms) : 0, 679378
GlobalTracer [baseline] (307.256 ms) : 0, 307256
GlobalTracer [candidate] (310.802 ms) : 0, 310802
AppSec [baseline] (51.314 ms) : 0, 51314
AppSec [candidate] (52.144 ms) : 0, 52144
Remote Config [baseline] (658.013 µs) : 0, 658
Remote Config [candidate] (669.514 µs) : 0, 670
Telemetry [baseline] (7.388 ms) : 0, 7388
Telemetry [candidate] (7.57 ms) : 0, 7570
section appsec
BytebuddyAgent [baseline] (680.157 ms) : 0, 680157
BytebuddyAgent [candidate] (680.043 ms) : 0, 680043
GlobalTracer [baseline] (300.804 ms) : 0, 300804
GlobalTracer [candidate] (302.224 ms) : 0, 302224
AppSec [baseline] (160.391 ms) : 0, 160391
AppSec [candidate] (160.19 ms) : 0, 160190
Remote Config [baseline] (617.314 µs) : 0, 617
Remote Config [candidate] (630.43 µs) : 0, 630
Telemetry [baseline] (7.143 ms) : 0, 7143
Telemetry [candidate] (7.453 ms) : 0, 7453
IAST [baseline] (18.49 ms) : 0, 18490
IAST [candidate] (18.674 ms) : 0, 18674
section iast
BytebuddyAgent [baseline] (786.633 ms) : 0, 786633
BytebuddyAgent [candidate] (780.319 ms) : 0, 780319
GlobalTracer [baseline] (301.074 ms) : 0, 301074
GlobalTracer [candidate] (296.883 ms) : 0, 296883
AppSec [baseline] (50.296 ms) : 0, 50296
AppSec [candidate] (49.775 ms) : 0, 49775
Remote Config [baseline] (614.52 µs) : 0, 615
Remote Config [candidate] (592.191 µs) : 0, 592
Telemetry [baseline] (10.279 ms) : 0, 10279
Telemetry [candidate] (9.58 ms) : 0, 9580
IAST [baseline] (24.04 ms) : 0, 24040
IAST [candidate] (24.346 ms) : 0, 24346
section profiling
BytebuddyAgent [baseline] (669.916 ms) : 0, 669916
BytebuddyAgent [candidate] (670.925 ms) : 0, 670925
GlobalTracer [baseline] (392.058 ms) : 0, 392058
GlobalTracer [candidate] (392.419 ms) : 0, 392419
AppSec [baseline] (52.435 ms) : 0, 52435
AppSec [candidate] (52.739 ms) : 0, 52739
Remote Config [baseline] (686.397 µs) : 0, 686
Remote Config [candidate] (703.708 µs) : 0, 704
Telemetry [baseline] (7.382 ms) : 0, 7382
Telemetry [candidate] (7.465 ms) : 0, 7465
ProfilingAgent [baseline] (96.557 ms) : 0, 96557
ProfilingAgent [candidate] (97.705 ms) : 0, 97705
Profiling [baseline] (96.58 ms) : 0, 96580
Profiling [candidate] (97.729 ms) : 0, 97729
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.048 s) : 0, 1048446
Total [baseline] (8.513 s) : 0, 8512705
Agent [candidate] (1.056 s) : 0, 1055598
Total [candidate] (8.517 s) : 0, 8516570
section iast
Agent [baseline] (1.177 s) : 0, 1177179
Total [baseline] (8.987 s) : 0, 8987021
Agent [candidate] (1.183 s) : 0, 1182511
Total [candidate] (9.056 s) : 0, 9056402
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.174 s) : 0, 1174403
Total [baseline] (9.007 s) : 0, 9006555
Agent [candidate] (1.177 s) : 0, 1176577
Total [candidate] (8.993 s) : 0, 8992597
section iast_TELEMETRY_OFF
Agent [baseline] (1.171 s) : 0, 1171360
Total [baseline] (8.988 s) : 0, 8988190
Agent [candidate] (1.181 s) : 0, 1181232
Total [candidate] (8.959 s) : 0, 8959097
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.048 s -
Agent iast 1.177 s 128.733 ms (12.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.174 s 125.957 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.171 s 122.913 ms (11.7%)
Total tracing 8.513 s -
Total iast 8.987 s 474.316 ms (5.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.007 s 493.85 ms (5.8%)
Total iast_TELEMETRY_OFF 8.988 s 475.485 ms (5.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.183 s 126.913 ms (12.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.177 s 120.98 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.181 s 125.634 ms (11.9%)
Total tracing 8.517 s -
Total iast 9.056 s 539.831 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 8.993 s 476.027 ms (5.6%)
Total iast_TELEMETRY_OFF 8.959 s 442.527 ms (5.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (668.93 ms) : 0, 668930
BytebuddyAgent [candidate] (673.42 ms) : 0, 673420
GlobalTracer [baseline] (306.651 ms) : 0, 306651
GlobalTracer [candidate] (308.799 ms) : 0, 308799
AppSec [baseline] (51.207 ms) : 0, 51207
AppSec [candidate] (51.474 ms) : 0, 51474
Remote Config [baseline] (657.47 µs) : 0, 657
Remote Config [candidate] (671.111 µs) : 0, 671
Telemetry [baseline] (7.433 ms) : 0, 7433
Telemetry [candidate] (7.574 ms) : 0, 7574
section iast
BytebuddyAgent [baseline] (781.544 ms) : 0, 781544
BytebuddyAgent [candidate] (784.933 ms) : 0, 784933
GlobalTracer [baseline] (296.927 ms) : 0, 296927
GlobalTracer [candidate] (298.677 ms) : 0, 298677
AppSec [baseline] (48.963 ms) : 0, 48963
AppSec [candidate] (50.835 ms) : 0, 50835
Remote Config [baseline] (602.033 µs) : 0, 602
Remote Config [candidate] (614.212 µs) : 0, 614
Telemetry [baseline] (7.993 ms) : 0, 7993
Telemetry [candidate] (8.936 ms) : 0, 8936
IAST [baseline] (27.54 ms) : 0, 27540
IAST [candidate] (24.814 ms) : 0, 24814
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (778.684 ms) : 0, 778684
BytebuddyAgent [candidate] (780.58 ms) : 0, 780580
GlobalTracer [baseline] (297.418 ms) : 0, 297418
GlobalTracer [candidate] (297.718 ms) : 0, 297718
AppSec [baseline] (49.398 ms) : 0, 49398
AppSec [candidate] (48.364 ms) : 0, 48364
Remote Config [baseline] (601.539 µs) : 0, 602
Remote Config [candidate] (597.816 µs) : 0, 598
Telemetry [baseline] (10.33 ms) : 0, 10330
Telemetry [candidate] (8.862 ms) : 0, 8862
IAST [baseline] (24.369 ms) : 0, 24369
IAST [candidate] (26.836 ms) : 0, 26836
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (776.579 ms) : 0, 776579
BytebuddyAgent [candidate] (783.307 ms) : 0, 783307
GlobalTracer [baseline] (296.934 ms) : 0, 296934
GlobalTracer [candidate] (299.341 ms) : 0, 299341
AppSec [baseline] (50.22 ms) : 0, 50220
AppSec [candidate] (51.643 ms) : 0, 51643
Remote Config [baseline] (586.57 µs) : 0, 587
Remote Config [candidate] (577.672 µs) : 0, 578
Telemetry [baseline] (10.981 ms) : 0, 10981
Telemetry [candidate] (8.592 ms) : 0, 8592
IAST [baseline] (22.468 ms) : 0, 22468
IAST [candidate] (24.049 ms) : 0, 24049
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-08-28T15:03:51 2024-08-28T15:10:38
git_branch master jpbempel/fix-flaky-smoke-tests
git_commit_date 1724852994 1724856829
git_commit_sha 78086bf 882d821
release_version 1.39.0-SNAPSHOT~78086bff11 1.39.0-SNAPSHOT~882d82103d
start_time 2024-08-28T15:03:38 2024-08-28T15:10:24
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1724858182 1724858182
ci_job_id 620156543 620156543
ci_pipeline_id 43044748 43044748
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.621 µs) : 345, 386
.   : milestone, 366,
iast (479.832 µs) : 457, 502
.   : milestone, 480,
iast_FULL (541.622 µs) : 521, 563
.   : milestone, 542,
iast_GLOBAL (502.259 µs) : 481, 524
.   : milestone, 502,
iast_HARDCODED_SECRET_DISABLED (485.015 µs) : 463, 507
.   : milestone, 485,
iast_INACTIVE (438.553 µs) : 418, 459
.   : milestone, 439,
iast_TELEMETRY_OFF (464.582 µs) : 442, 487
.   : milestone, 465,
tracing (445.611 µs) : 425, 467
.   : milestone, 446,
section candidate
no_agent (369.715 µs) : 350, 389
.   : milestone, 370,
iast (474.32 µs) : 452, 497
.   : milestone, 474,
iast_FULL (545.776 µs) : 524, 567
.   : milestone, 546,
iast_GLOBAL (509.993 µs) : 488, 532
.   : milestone, 510,
iast_HARDCODED_SECRET_DISABLED (478.546 µs) : 456, 501
.   : milestone, 479,
iast_INACTIVE (440.533 µs) : 419, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (467.137 µs) : 445, 489
.   : milestone, 467,
tracing (435.774 µs) : 415, 457
.   : milestone, 436,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.621 µs [344.967 µs, 386.274 µs] -
iast 479.832 µs [457.452 µs, 502.213 µs] 114.212 µs (31.2%)
iast_FULL 541.622 µs [520.571 µs, 562.673 µs] 176.001 µs (48.1%)
iast_GLOBAL 502.259 µs [480.528 µs, 523.991 µs] 136.639 µs (37.4%)
iast_HARDCODED_SECRET_DISABLED 485.015 µs [462.745 µs, 507.285 µs] 119.394 µs (32.7%)
iast_INACTIVE 438.553 µs [417.92 µs, 459.186 µs] 72.932 µs (19.9%)
iast_TELEMETRY_OFF 464.582 µs [442.491 µs, 486.672 µs] 98.961 µs (27.1%)
tracing 445.611 µs [424.639 µs, 466.584 µs] 79.991 µs (21.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 369.715 µs [350.222 µs, 389.208 µs] -
iast 474.32 µs [452.113 µs, 496.527 µs] 104.605 µs (28.3%)
iast_FULL 545.776 µs [524.305 µs, 567.248 µs] 176.061 µs (47.6%)
iast_GLOBAL 509.993 µs [488.441 µs, 531.545 µs] 140.278 µs (37.9%)
iast_HARDCODED_SECRET_DISABLED 478.546 µs [456.092 µs, 501.0 µs] 108.831 µs (29.4%)
iast_INACTIVE 440.533 µs [419.209 µs, 461.857 µs] 70.818 µs (19.2%)
iast_TELEMETRY_OFF 467.137 µs [445.003 µs, 489.271 µs] 97.422 µs (26.4%)
tracing 435.774 µs [415.025 µs, 456.524 µs] 66.059 µs (17.9%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~78086bff11
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.333 ms) : 1314, 1352
.   : milestone, 1333,
appsec (1.72 ms) : 1696, 1743
.   : milestone, 1720,
appsec_no_iast (1.719 ms) : 1695, 1744
.   : milestone, 1719,
iast (1.455 ms) : 1432, 1478
.   : milestone, 1455,
profiling (1.483 ms) : 1459, 1506
.   : milestone, 1483,
tracing (1.451 ms) : 1427, 1476
.   : milestone, 1451,
section candidate
no_agent (1.345 ms) : 1326, 1364
.   : milestone, 1345,
appsec (1.702 ms) : 1677, 1726
.   : milestone, 1702,
appsec_no_iast (1.732 ms) : 1707, 1756
.   : milestone, 1732,
iast (1.466 ms) : 1443, 1488
.   : milestone, 1466,
profiling (1.507 ms) : 1483, 1531
.   : milestone, 1507,
tracing (1.48 ms) : 1455, 1504
.   : milestone, 1480,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.333 ms [1.314 ms, 1.352 ms] -
appsec 1.72 ms [1.696 ms, 1.743 ms] 386.83 µs (29.0%)
appsec_no_iast 1.719 ms [1.695 ms, 1.744 ms] 386.618 µs (29.0%)
iast 1.455 ms [1.432 ms, 1.478 ms] 121.892 µs (9.1%)
profiling 1.483 ms [1.459 ms, 1.506 ms] 149.835 µs (11.2%)
tracing 1.451 ms [1.427 ms, 1.476 ms] 118.58 µs (8.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.345 ms [1.326 ms, 1.364 ms] -
appsec 1.702 ms [1.677 ms, 1.726 ms] 356.593 µs (26.5%)
appsec_no_iast 1.732 ms [1.707 ms, 1.756 ms] 386.426 µs (28.7%)
iast 1.466 ms [1.443 ms, 1.488 ms] 120.565 µs (9.0%)
profiling 1.507 ms [1.483 ms, 1.531 ms] 162.096 µs (12.1%)
tracing 1.48 ms [1.455 ms, 1.504 ms] 134.432 µs (10.0%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-flaky-smoke-tests
git_commit_date 1724857102 1724856829
git_commit_sha 1283079 882d821
release_version 1.39.0-SNAPSHOT~128307945d 1.39.0-SNAPSHOT~882d82103d
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1724858922 1724858922
ci_job_id 620156545 620156545
ci_pipeline_id 43044748 43044748
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 1 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:dacapo:tomcat:tracing better
[-243.043µs; -172.347µs] or [-11.965%; -8.485%]		 1.824ms 2.031ms
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~128307945d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.454 ms) : 1442, 1465
.   : milestone, 1454,
appsec (2.206 ms) : 2172, 2241
.   : milestone, 2206,
iast (1.956 ms) : 1913, 1998
.   : milestone, 1956,
iast_GLOBAL (2.005 ms) : 1961, 2048
.   : milestone, 2005,
profiling (2.35 ms) : 2161, 2539
.   : milestone, 2350,
tracing (2.031 ms) : 1998, 2065
.   : milestone, 2031,
section candidate
no_agent (1.456 ms) : 1445, 1468
.   : milestone, 1456,
appsec (2.223 ms) : 2188, 2258
.   : milestone, 2223,
iast (1.969 ms) : 1926, 2011
.   : milestone, 1969,
iast_GLOBAL (2.014 ms) : 1971, 2058
.   : milestone, 2014,
profiling (1.848 ms) : 1815, 1882
.   : milestone, 1848,
tracing (1.824 ms) : 1791, 1856
.   : milestone, 1824,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.454 ms [1.442 ms, 1.465 ms] -
appsec 2.206 ms [2.172 ms, 2.241 ms] 752.561 µs (51.8%)
iast 1.956 ms [1.913 ms, 1.998 ms] 502.201 µs (34.5%)
iast_GLOBAL 2.005 ms [1.961 ms, 2.048 ms] 550.871 µs (37.9%)
profiling 2.35 ms [2.161 ms, 2.539 ms] 896.573 µs (61.7%)
tracing 2.031 ms [1.998 ms, 2.065 ms] 577.579 µs (39.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.456 ms [1.445 ms, 1.468 ms] -
appsec 2.223 ms [2.188 ms, 2.258 ms] 766.913 µs (52.7%)
iast 1.969 ms [1.926 ms, 2.011 ms] 512.431 µs (35.2%)
iast_GLOBAL 2.014 ms [1.971 ms, 2.058 ms] 558.002 µs (38.3%)
profiling 1.848 ms [1.815 ms, 1.882 ms] 391.944 µs (26.9%)
tracing 1.824 ms [1.791 ms, 1.856 ms] 367.19 µs (25.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~882d82103d, baseline=1.39.0-SNAPSHOT~128307945d
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.22 s) : 15220000, 15220000
.   : milestone, 15220000,
appsec (14.896 s) : 14896000, 14896000
.   : milestone, 14896000,
iast (18.678 s) : 18678000, 18678000
.   : milestone, 18678000,
iast_GLOBAL (17.827 s) : 17827000, 17827000
.   : milestone, 17827000,
profiling (15.215 s) : 15215000, 15215000
.   : milestone, 15215000,
tracing (15.358 s) : 15358000, 15358000
.   : milestone, 15358000,
section candidate
no_agent (14.87 s) : 14870000, 14870000
.   : milestone, 14870000,
appsec (15.285 s) : 15285000, 15285000
.   : milestone, 15285000,
iast (18.827 s) : 18827000, 18827000
.   : milestone, 18827000,
iast_GLOBAL (17.763 s) : 17763000, 17763000
.   : milestone, 17763000,
profiling (14.811 s) : 14811000, 14811000
.   : milestone, 14811000,
tracing (15.014 s) : 15014000, 15014000
.   : milestone, 15014000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.22 s [15.22 s, 15.22 s] -
appsec 14.896 s [14.896 s, 14.896 s] -324.0 ms (-2.1%)
iast 18.678 s [18.678 s, 18.678 s] 3.458 s (22.7%)
iast_GLOBAL 17.827 s [17.827 s, 17.827 s] 2.607 s (17.1%)
profiling 15.215 s [15.215 s, 15.215 s] -5.0 ms (-0.0%)
tracing 15.358 s [15.358 s, 15.358 s] 138.0 ms (0.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.87 s [14.87 s, 14.87 s] -
appsec 15.285 s [15.285 s, 15.285 s] 415.0 ms (2.8%)
iast 18.827 s [18.827 s, 18.827 s] 3.957 s (26.6%)
iast_GLOBAL 17.763 s [17.763 s, 17.763 s] 2.893 s (19.5%)
profiling 14.811 s [14.811 s, 14.811 s] -59.0 ms (-0.4%)
tracing 15.014 s [15.014 s, 15.014 s] 144.0 ms (1.0%)

@smola smola added tag: no release notes Changes to exclude from release notes tag: flaky test/disabled Disabled flaky tests labels Aug 28, 2024
@PerfectSlayer PerfectSlayer added tag: flaky test Flaky tests and removed comp: testing Testing labels Aug 30, 2024
@jpbempel jpbempel merged commit f31ddbb into master Aug 30, 2024
@jpbempel jpbempel deleted the jpbempel/fix-flaky-smoke-tests branch August 30, 2024 08:26
@github-actions github-actions bot added this to the 1.39.0 milestone Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@ojung ojung Awaiting requested review from ojung ojung is a code owner automatically assigned from DataDog/debugger-java

Assignees

No one assigned

Labels

comp: debugger Dynamic Instrumentation tag: flaky test/disabled Disabled flaky tests tag: flaky test Flaky tests tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.39.0

Development

Successfully merging this pull request may close these issues.

4 participants

@jpbempel @PerfectSlayer @smola