Implement early flakiness detection

[nikita-tkachenko-datadog]

What Does This Do

Implements early flakiness detection CI Visibility feature (RFC).

Jira ticket: CIVIS-8329

[pr-commenter]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/early-flakiness-detection
git_commit_date	1709036372	1709037393
git_commit_sha	ef26b91	4d273d4
release_version	1.31.0-SNAPSHOT~ef26b91da3	1.31.0-SNAPSHOT~4d273d473c

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1709040230	1709040230
ci_job_id	444375749	444375749
ci_pipeline_id	29011801	29011801
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 42 metrics, 12 unstable metrics.

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-02-27T13:00:16	2024-02-27T13:18:51
git_branch	master	nikita-tkachenko/early-flakiness-detection
git_commit_date	1709036372	1709037393
git_commit_sha	ef26b91	4d273d4
release_version	1.31.0-SNAPSHOT~ef26b91da3	1.31.0-SNAPSHOT~4d273d473c
start_time	2024-02-27T13:00:03	2024-02-27T13:18:39

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1709040230	1709040230
ci_job_id	444375749	444375749
ci_pipeline_id	29011801	29011801
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 16 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling	better [-100.100µs; -47.672µs] or [-6.296%; -2.998%]	unstable [-331.017op/s; +770.577op/s] or [-11.586%; +26.970%]	1.516ms	3076.923op/s	1.590ms	2857.143op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~4d273d473c, baseline=1.31.0-SNAPSHOT~ef26b91da3
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.578 µs) : 350, 395
.   : milestone, 373,
iast (474.758 µs) : 454, 495
.   : milestone, 475,
iast_FULL (541.095 µs) : 521, 562
.   : milestone, 541,
iast_GLOBAL (498.614 µs) : 477, 520
.   : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (478.317 µs) : 458, 499
.   : milestone, 478,
iast_INACTIVE (444.877 µs) : 424, 465
.   : milestone, 445,
iast_TELEMETRY_OFF (473.438 µs) : 453, 494
.   : milestone, 473,
tracing (436.894 µs) : 416, 457
.   : milestone, 437,
section candidate
no_agent (363.348 µs) : 343, 384
.   : milestone, 363,
iast (470.209 µs) : 450, 491
.   : milestone, 470,
iast_FULL (538.962 µs) : 519, 559
.   : milestone, 539,
iast_GLOBAL (489.187 µs) : 469, 510
.   : milestone, 489,
iast_HARDCODED_SECRET_DISABLED (472.981 µs) : 452, 494
.   : milestone, 473,
iast_INACTIVE (445.361 µs) : 424, 467
.   : milestone, 445,
iast_TELEMETRY_OFF (469.753 µs) : 449, 490
.   : milestone, 470,
tracing (439.042 µs) : 419, 459
.   : milestone, 439,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	372.578 µs [350.086 µs, 395.069 µs]	-
iast	474.758 µs [454.229 µs, 495.287 µs]	102.181 µs (27.4%)
iast_FULL	541.095 µs [520.518 µs, 561.673 µs]	168.518 µs (45.2%)
iast_GLOBAL	498.614 µs [477.22 µs, 520.009 µs]	126.037 µs (33.8%)
iast_HARDCODED_SECRET_DISABLED	478.317 µs [457.8 µs, 498.834 µs]	105.739 µs (28.4%)
iast_INACTIVE	444.877 µs [424.445 µs, 465.309 µs]	72.299 µs (19.4%)
iast_TELEMETRY_OFF	473.438 µs [452.704 µs, 494.173 µs]	100.861 µs (27.1%)
tracing	436.894 µs [416.335 µs, 457.453 µs]	64.317 µs (17.3%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	363.348 µs [343.026 µs, 383.669 µs]	-
iast	470.209 µs [449.669 µs, 490.75 µs]	106.862 µs (29.4%)
iast_FULL	538.962 µs [518.58 µs, 559.344 µs]	175.615 µs (48.3%)
iast_GLOBAL	489.187 µs [468.86 µs, 509.515 µs]	125.84 µs (34.6%)
iast_HARDCODED_SECRET_DISABLED	472.981 µs [452.014 µs, 493.949 µs]	109.633 µs (30.2%)
iast_INACTIVE	445.361 µs [424.121 µs, 466.601 µs]	82.014 µs (22.6%)
iast_TELEMETRY_OFF	469.753 µs [449.085 µs, 490.42 µs]	106.405 µs (29.3%)
tracing	439.042 µs [418.94 µs, 459.145 µs]	75.695 µs (20.8%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~4d273d473c, baseline=1.31.0-SNAPSHOT~ef26b91da3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.344 ms) : 1325, 1363
.   : milestone, 1344,
appsec (1.762 ms) : 1738, 1785
.   : milestone, 1762,
iast (1.509 ms) : 1486, 1532
.   : milestone, 1509,
profiling (1.59 ms) : 1565, 1615
.   : milestone, 1590,
tracing (1.495 ms) : 1472, 1518
.   : milestone, 1495,
section candidate
no_agent (1.347 ms) : 1328, 1366
.   : milestone, 1347,
appsec (1.787 ms) : 1763, 1810
.   : milestone, 1787,
iast (1.513 ms) : 1489, 1536
.   : milestone, 1513,
profiling (1.516 ms) : 1492, 1540
.   : milestone, 1516,
tracing (1.496 ms) : 1472, 1519
.   : milestone, 1496,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.344 ms [1.325 ms, 1.363 ms]	-
appsec	1.762 ms [1.738 ms, 1.785 ms]	417.682 µs (31.1%)
iast	1.509 ms [1.486 ms, 1.532 ms]	164.486 µs (12.2%)
profiling	1.59 ms [1.565 ms, 1.615 ms]	245.778 µs (18.3%)
tracing	1.495 ms [1.472 ms, 1.518 ms]	150.53 µs (11.2%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.347 ms [1.328 ms, 1.366 ms]	-
appsec	1.787 ms [1.763 ms, 1.81 ms]	439.599 µs (32.6%)
iast	1.513 ms [1.489 ms, 1.536 ms]	165.827 µs (12.3%)
profiling	1.516 ms [1.492 ms, 1.54 ms]	169.107 µs (12.6%)
tracing	1.496 ms [1.472 ms, 1.519 ms]	148.775 µs (11.0%)