Force test failure on unhandled instrumented exception

[amarziali]

What Does This Do

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

[pr-commenter]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/exit-on-error
git_commit_date	1706188997	1706190898
git_commit_sha	1557ad1	678f896
release_version	1.29.0-SNAPSHOT~1557ad19cf	1.29.0-SNAPSHOT~678f89692b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1706193796	1706193796
ci_job_id	418505180	418505180
ci_pipeline_id	27177040	27177040
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-01-25T14:19:10	2024-01-25T14:38:13
git_branch	master	andrea.marziali/exit-on-error
git_commit_date	1706188997	1706190898
git_commit_sha	1557ad1	678f896
release_version	1.29.0-SNAPSHOT~1557ad19cf	1.29.0-SNAPSHOT~678f89692b
start_time	2024-01-25T14:18:57	2024-01-25T14:38:00

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1706193796	1706193796
ci_job_id	418505180	418505180
ci_pipeline_id	27177040	27177040
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 15 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling	better [-96.374µs; -40.076µs] or [-6.039%; -2.511%]	unstable [-406.086op/s; +634.006op/s] or [-13.705%; +21.398%]	1.528ms	3076.923op/s	1.596ms	2962.963op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~678f89692b, baseline=1.29.0-SNAPSHOT~1557ad19cf
    dateFormat X
    axisFormat %s
section baseline
no_agent (369.181 µs) : 349, 390
.   : milestone, 369,
iast (477.247 µs) : 457, 498
.   : milestone, 477,
iast_FULL (538.229 µs) : 518, 559
.   : milestone, 538,
iast_GLOBAL (512.219 µs) : 491, 534
.   : milestone, 512,
iast_HARDCODED_SECRET_DISABLED (478.911 µs) : 458, 500
.   : milestone, 479,
iast_INACTIVE (447.579 µs) : 427, 468
.   : milestone, 448,
iast_TELEMETRY_OFF (471.307 µs) : 450, 492
.   : milestone, 471,
tracing (433.351 µs) : 413, 453
.   : milestone, 433,
section candidate
no_agent (364.505 µs) : 345, 384
.   : milestone, 365,
iast (471.102 µs) : 450, 492
.   : milestone, 471,
iast_FULL (540.612 µs) : 520, 561
.   : milestone, 541,
iast_GLOBAL (504.375 µs) : 483, 525
.   : milestone, 504,
iast_HARDCODED_SECRET_DISABLED (476.382 µs) : 455, 498
.   : milestone, 476,
iast_INACTIVE (442.783 µs) : 422, 463
.   : milestone, 443,
iast_TELEMETRY_OFF (472.201 µs) : 451, 493
.   : milestone, 472,
tracing (453.434 µs) : 431, 476
.   : milestone, 453,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	369.181 µs [348.776 µs, 389.585 µs]	-
iast	477.247 µs [456.513 µs, 497.981 µs]	108.067 µs (29.3%)
iast_FULL	538.229 µs [517.509 µs, 558.949 µs]	169.048 µs (45.8%)
iast_GLOBAL	512.219 µs [490.579 µs, 533.858 µs]	143.038 µs (38.7%)
iast_HARDCODED_SECRET_DISABLED	478.911 µs [457.624 µs, 500.198 µs]	109.731 µs (29.7%)
iast_INACTIVE	447.579 µs [426.931 µs, 468.227 µs]	78.399 µs (21.2%)
iast_TELEMETRY_OFF	471.307 µs [450.453 µs, 492.161 µs]	102.126 µs (27.7%)
tracing	433.351 µs [413.375 µs, 453.326 µs]	64.17 µs (17.4%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.505 µs [344.702 µs, 384.308 µs]	-
iast	471.102 µs [450.428 µs, 491.777 µs]	106.598 µs (29.2%)
iast_FULL	540.612 µs [520.108 µs, 561.115 µs]	176.107 µs (48.3%)
iast_GLOBAL	504.375 µs [483.411 µs, 525.338 µs]	139.87 µs (38.4%)
iast_HARDCODED_SECRET_DISABLED	476.382 µs [454.935 µs, 497.829 µs]	111.877 µs (30.7%)
iast_INACTIVE	442.783 µs [422.392 µs, 463.175 µs]	78.279 µs (21.5%)
iast_TELEMETRY_OFF	472.201 µs [451.179 µs, 493.224 µs]	107.696 µs (29.5%)
tracing	453.434 µs [430.845 µs, 476.023 µs]	88.929 µs (24.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.29.0-SNAPSHOT~678f89692b, baseline=1.29.0-SNAPSHOT~1557ad19cf
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.353 ms) : 1334, 1372
.   : milestone, 1353,
appsec (1.772 ms) : 1747, 1797
.   : milestone, 1772,
iast (1.521 ms) : 1497, 1546
.   : milestone, 1521,
profiling (1.596 ms) : 1570, 1622
.   : milestone, 1596,
tracing (1.521 ms) : 1496, 1546
.   : milestone, 1521,
section candidate
no_agent (1.359 ms) : 1340, 1378
.   : milestone, 1359,
appsec (1.795 ms) : 1770, 1820
.   : milestone, 1795,
iast (1.517 ms) : 1493, 1542
.   : milestone, 1517,
profiling (1.528 ms) : 1502, 1554
.   : milestone, 1528,
tracing (1.504 ms) : 1479, 1529
.   : milestone, 1504,

Loading

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.334 ms, 1.372 ms]	-
appsec	1.772 ms [1.747 ms, 1.797 ms]	418.392 µs (30.9%)
iast	1.521 ms [1.497 ms, 1.546 ms]	168.066 µs (12.4%)
profiling	1.596 ms [1.57 ms, 1.622 ms]	242.527 µs (17.9%)
tracing	1.521 ms [1.496 ms, 1.546 ms]	167.429 µs (12.4%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.359 ms [1.34 ms, 1.378 ms]	-
appsec	1.795 ms [1.77 ms, 1.82 ms]	435.484 µs (32.0%)
iast	1.517 ms [1.493 ms, 1.542 ms]	158.124 µs (11.6%)
profiling	1.528 ms [1.502 ms, 1.554 ms]	168.455 µs (12.4%)
tracing	1.504 ms [1.479 ms, 1.529 ms]	144.65 µs (10.6%)

[mcculls]

Nice! Especially like the idea to count these errors, as we can then expose that as a metric :)

[amarziali]

@smola as discussed there are ASM tests failing because of "Interaction with TraceSegment after root span has already finished"

Till today this error was triggered but the exception handler did not record the failure. Today is happening and the tests are failing. This check makes sense when strict trace writing is enabled. Now, I aligned it and disabled strict trace writing for grizzly tests that are showing those failures. A fixme is reminding that at long term we should probably fix that instrumentation