Keep track of public classes in compact filter #6860

mcculls · 2024-03-31T23:50:18Z

Motivation

Reduces the number of class-file lookups needed when byte-buddy checks which types are visible from other types.

(cuts the number of additional class-file lookups by almost 50% - most of the remaining lookups relate to muzzle)

pr-commenter · 2024-04-01T08:27:24Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/add-visibility-filter
git_commit_date	1711742224	1711928809
git_commit_sha	`e49f1a1`	`d2b8f33`
release_version	1.32.0-SNAPSHOT~e49f1a1efb	1.32.0-SNAPSHOT~d2b8f3306b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1711959995	1711959995
ci_job_id	474146562	474146562
ci_pipeline_id	31158721	31158721
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 16 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.07 s) : 0, 1069902
Total [baseline] (8.552 s) : 0, 8551810
Agent [candidate] (1.083 s) : 0, 1082736
Total [candidate] (8.562 s) : 0, 8561765
section iast
Agent [baseline] (1.199 s) : 0, 1199327
Total [baseline] (9.075 s) : 0, 9074676
Agent [candidate] (1.201 s) : 0, 1201172
Total [candidate] (9.016 s) : 0, 9015741
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.205 s) : 0, 1204770
Total [baseline] (9.007 s) : 0, 9007080
Agent [candidate] (1.216 s) : 0, 1215949
Total [candidate] (9.039 s) : 0, 9039356
section iast_TELEMETRY_OFF
Agent [baseline] (1.195 s) : 0, 1194528
Total [baseline] (9.103 s) : 0, 9102650
Agent [candidate] (1.198 s) : 0, 1198226
Total [candidate] (9.048 s) : 0, 9048444

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.07 s	-
Agent	iast	1.199 s	129.425 ms (12.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.205 s	134.868 ms (12.6%)
Agent	iast_TELEMETRY_OFF	1.195 s	124.626 ms (11.6%)
Total	tracing	8.552 s	-
Total	iast	9.075 s	522.865 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.007 s	455.27 ms (5.3%)
Total	iast_TELEMETRY_OFF	9.103 s	550.84 ms (6.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.083 s	-
Agent	iast	1.201 s	118.435 ms (10.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.216 s	133.213 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.198 s	115.489 ms (10.7%)
Total	tracing	8.562 s	-
Total	iast	9.016 s	453.976 ms (5.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.039 s	477.591 ms (5.6%)
Total	iast_TELEMETRY_OFF	9.048 s	486.678 ms (5.7%)

gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (669.707 ms) : 0, 669707
BytebuddyAgent [candidate] (677.594 ms) : 0, 677594
GlobalTracer [baseline] (308.289 ms) : 0, 308289
GlobalTracer [candidate] (312.256 ms) : 0, 312256
AppSec [baseline] (49.386 ms) : 0, 49386
AppSec [candidate] (49.882 ms) : 0, 49882
Remote Config [baseline] (665.26 µs) : 0, 665
Remote Config [candidate] (673.576 µs) : 0, 674
Telemetry [baseline] (7.61 ms) : 0, 7610
Telemetry [candidate] (7.735 ms) : 0, 7735
section iast
BytebuddyAgent [baseline] (794.729 ms) : 0, 794729
BytebuddyAgent [candidate] (796.088 ms) : 0, 796088
GlobalTracer [baseline] (288.311 ms) : 0, 288311
GlobalTracer [candidate] (288.865 ms) : 0, 288865
AppSec [baseline] (50.138 ms) : 0, 50138
AppSec [candidate] (50.75 ms) : 0, 50750
Remote Config [baseline] (583.162 µs) : 0, 583
Remote Config [candidate] (588.342 µs) : 0, 588
Telemetry [baseline] (8.116 ms) : 0, 8116
Telemetry [candidate] (7.431 ms) : 0, 7431
IAST [baseline] (23.09 ms) : 0, 23090
IAST [candidate] (23.115 ms) : 0, 23115
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (798.689 ms) : 0, 798689
BytebuddyAgent [candidate] (806.08 ms) : 0, 806080
GlobalTracer [baseline] (289.535 ms) : 0, 289535
GlobalTracer [candidate] (292.631 ms) : 0, 292631
AppSec [baseline] (51.247 ms) : 0, 51247
AppSec [candidate] (51.396 ms) : 0, 51396
Remote Config [baseline] (578.047 µs) : 0, 578
Remote Config [candidate] (595.564 µs) : 0, 596
Telemetry [baseline] (7.35 ms) : 0, 7350
Telemetry [candidate] (8.236 ms) : 0, 8236
IAST [baseline] (22.757 ms) : 0, 22757
IAST [candidate] (22.254 ms) : 0, 22254
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (790.109 ms) : 0, 790109
BytebuddyAgent [candidate] (792.93 ms) : 0, 792930
GlobalTracer [baseline] (289.485 ms) : 0, 289485
GlobalTracer [candidate] (289.527 ms) : 0, 289527
AppSec [baseline] (49.633 ms) : 0, 49633
AppSec [candidate] (48.158 ms) : 0, 48158
Remote Config [baseline] (582.063 µs) : 0, 582
Remote Config [candidate] (597.175 µs) : 0, 597
Telemetry [baseline] (6.589 ms) : 0, 6589
Telemetry [candidate] (8.157 ms) : 0, 8157
IAST [baseline] (23.849 ms) : 0, 23849
IAST [candidate] (24.347 ms) : 0, 24347

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1078786
Total [baseline] (10.5 s) : 0, 10499677
Agent [candidate] (1.076 s) : 0, 1075659
Total [candidate] (10.346 s) : 0, 10345758
section appsec
Agent [baseline] (1.194 s) : 0, 1194416
Total [baseline] (10.486 s) : 0, 10485537
Agent [candidate] (1.196 s) : 0, 1196459
Total [candidate] (10.615 s) : 0, 10614970
section iast
Agent [baseline] (1.197 s) : 0, 1196541
Total [baseline] (10.805 s) : 0, 10804675
Agent [candidate] (1.215 s) : 0, 1214922
Total [candidate] (10.837 s) : 0, 10837180
section profiling
Agent [baseline] (1.267 s) : 0, 1267078
Total [baseline] (10.685 s) : 0, 10685276
Agent [candidate] (1.271 s) : 0, 1270829
Total [candidate] (10.602 s) : 0, 10602157

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.079 s	-
Agent	appsec	1.194 s	115.63 ms (10.7%)
Agent	iast	1.197 s	117.755 ms (10.9%)
Agent	profiling	1.267 s	188.293 ms (17.5%)
Total	tracing	10.5 s	-
Total	appsec	10.486 s	-14.14 ms (-0.1%)
Total	iast	10.805 s	304.998 ms (2.9%)
Total	profiling	10.685 s	185.599 ms (1.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.076 s	-
Agent	appsec	1.196 s	120.801 ms (11.2%)
Agent	iast	1.215 s	139.264 ms (12.9%)
Agent	profiling	1.271 s	195.17 ms (18.1%)
Total	tracing	10.346 s	-
Total	appsec	10.615 s	269.213 ms (2.6%)
Total	iast	10.837 s	491.423 ms (4.7%)
Total	profiling	10.602 s	256.399 ms (2.5%)

gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.744 ms) : 0, 675744
BytebuddyAgent [candidate] (673.203 ms) : 0, 673203
GlobalTracer [baseline] (310.715 ms) : 0, 310715
GlobalTracer [candidate] (310.288 ms) : 0, 310288
AppSec [baseline] (49.483 ms) : 0, 49483
AppSec [candidate] (49.435 ms) : 0, 49435
Remote Config [baseline] (764.899 µs) : 0, 765
Remote Config [candidate] (664.23 µs) : 0, 664
Telemetry [baseline] (7.559 ms) : 0, 7559
Telemetry [candidate] (7.628 ms) : 0, 7628
section appsec
BytebuddyAgent [baseline] (693.802 ms) : 0, 693802
BytebuddyAgent [candidate] (694.495 ms) : 0, 694495
GlobalTracer [baseline] (290.562 ms) : 0, 290562
GlobalTracer [candidate] (291.849 ms) : 0, 291849
AppSec [baseline] (149.654 ms) : 0, 149654
AppSec [candidate] (149.331 ms) : 0, 149331
IAST [baseline] (18.732 ms) : 0, 18732
IAST [candidate] (18.969 ms) : 0, 18969
Remote Config [baseline] (602.077 µs) : 0, 602
Remote Config [candidate] (613.078 µs) : 0, 613
Telemetry [baseline] (6.788 ms) : 0, 6788
Telemetry [candidate] (6.875 ms) : 0, 6875
section iast
BytebuddyAgent [baseline] (792.476 ms) : 0, 792476
BytebuddyAgent [candidate] (806.139 ms) : 0, 806139
GlobalTracer [baseline] (287.728 ms) : 0, 287728
GlobalTracer [candidate] (292.48 ms) : 0, 292480
AppSec [baseline] (48.943 ms) : 0, 48943
AppSec [candidate] (50.975 ms) : 0, 50975
IAST [baseline] (24.46 ms) : 0, 24460
IAST [candidate] (23.123 ms) : 0, 23123
Remote Config [baseline] (566.311 µs) : 0, 566
Remote Config [candidate] (587.142 µs) : 0, 587
Telemetry [baseline] (8.09 ms) : 0, 8090
Telemetry [candidate] (6.695 ms) : 0, 6695
section profiling
BytebuddyAgent [baseline] (677.036 ms) : 0, 677036
BytebuddyAgent [candidate] (678.632 ms) : 0, 678632
GlobalTracer [baseline] (379.876 ms) : 0, 379876
GlobalTracer [candidate] (381.919 ms) : 0, 381919
AppSec [baseline] (50.247 ms) : 0, 50247
AppSec [candidate] (50.104 ms) : 0, 50104
Remote Config [baseline] (719.054 µs) : 0, 719
Remote Config [candidate] (709.549 µs) : 0, 710
Telemetry [baseline] (7.454 ms) : 0, 7454
Telemetry [candidate] (7.466 ms) : 0, 7466
ProfilingAgent [baseline] (95.591 ms) : 0, 95591
ProfilingAgent [candidate] (95.775 ms) : 0, 95775
Profiling [baseline] (95.615 ms) : 0, 95615
Profiling [candidate] (95.799 ms) : 0, 95799

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-04-01T07:58:49	2024-04-01T08:20:52
git_branch	master	mcculls/add-visibility-filter
git_commit_date	1711742224	1711928809
git_commit_sha	`e49f1a1`	`d2b8f33`
release_version	1.32.0-SNAPSHOT~e49f1a1efb	1.32.0-SNAPSHOT~d2b8f3306b
start_time	2024-04-01T07:58:36	2024-04-01T08:20:38

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1711959995	1711959995
ci_job_id	474146562	474146562
ci_pipeline_id	31158721	31158721
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 16 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling	better [-90.041µs; -36.628µs] or [-5.810%; -2.363%]	unstable [-552.336op/s; +552.336op/s] or [-17.951%; +17.951%]	1.486ms	3076.923op/s	1.550ms	3076.923op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.588 µs) : 347, 386
.   : milestone, 367,
iast (472.282 µs) : 452, 493
.   : milestone, 472,
iast_FULL (539.566 µs) : 519, 560
.   : milestone, 540,
iast_GLOBAL (492.876 µs) : 472, 513
.   : milestone, 493,
iast_HARDCODED_SECRET_DISABLED (477.876 µs) : 457, 498
.   : milestone, 478,
iast_INACTIVE (448.431 µs) : 427, 470
.   : milestone, 448,
iast_TELEMETRY_OFF (480.662 µs) : 459, 502
.   : milestone, 481,
tracing (442.036 µs) : 422, 462
.   : milestone, 442,
section candidate
no_agent (369.193 µs) : 350, 389
.   : milestone, 369,
iast (480.257 µs) : 459, 501
.   : milestone, 480,
iast_FULL (543.924 µs) : 523, 564
.   : milestone, 544,
iast_GLOBAL (496.389 µs) : 475, 518
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (471.941 µs) : 451, 493
.   : milestone, 472,
iast_INACTIVE (448.358 µs) : 428, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (469.341 µs) : 449, 490
.   : milestone, 469,
tracing (438.531 µs) : 419, 459
.   : milestone, 439,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.588 µs [347.025 µs, 386.152 µs]	-
iast	472.282 µs [451.732 µs, 492.832 µs]	105.693 µs (28.8%)
iast_FULL	539.566 µs [518.832 µs, 560.3 µs]	172.977 µs (47.2%)
iast_GLOBAL	492.876 µs [472.338 µs, 513.413 µs]	126.287 µs (34.4%)
iast_HARDCODED_SECRET_DISABLED	477.876 µs [457.311 µs, 498.441 µs]	111.288 µs (30.4%)
iast_INACTIVE	448.431 µs [426.976 µs, 469.887 µs]	81.843 µs (22.3%)
iast_TELEMETRY_OFF	480.662 µs [459.218 µs, 502.106 µs]	114.074 µs (31.1%)
tracing	442.036 µs [421.712 µs, 462.361 µs]	75.448 µs (20.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	369.193 µs [349.647 µs, 388.739 µs]	-
iast	480.257 µs [459.29 µs, 501.224 µs]	111.064 µs (30.1%)
iast_FULL	543.924 µs [523.377 µs, 564.471 µs]	174.731 µs (47.3%)
iast_GLOBAL	496.389 µs [474.795 µs, 517.982 µs]	127.195 µs (34.5%)
iast_HARDCODED_SECRET_DISABLED	471.941 µs [450.986 µs, 492.896 µs]	102.748 µs (27.8%)
iast_INACTIVE	448.358 µs [427.808 µs, 468.908 µs]	79.165 µs (21.4%)
iast_TELEMETRY_OFF	469.341 µs [448.541 µs, 490.14 µs]	100.147 µs (27.1%)
tracing	438.531 µs [418.511 µs, 458.551 µs]	69.338 µs (18.8%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~d2b8f3306b, baseline=1.32.0-SNAPSHOT~e49f1a1efb
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1323, 1363
.   : milestone, 1343,
appsec (1.726 ms) : 1702, 1751
.   : milestone, 1726,
appsec_no_iast (1.721 ms) : 1696, 1746
.   : milestone, 1721,
iast (1.485 ms) : 1462, 1508
.   : milestone, 1485,
profiling (1.55 ms) : 1525, 1575
.   : milestone, 1550,
tracing (1.489 ms) : 1464, 1513
.   : milestone, 1489,
section candidate
no_agent (1.342 ms) : 1322, 1361
.   : milestone, 1342,
appsec (1.722 ms) : 1697, 1746
.   : milestone, 1722,
appsec_no_iast (1.708 ms) : 1683, 1732
.   : milestone, 1708,
iast (1.486 ms) : 1463, 1508
.   : milestone, 1486,
profiling (1.486 ms) : 1462, 1511
.   : milestone, 1486,
tracing (1.479 ms) : 1454, 1503
.   : milestone, 1479,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.343 ms [1.323 ms, 1.363 ms]	-
appsec	1.726 ms [1.702 ms, 1.751 ms]	383.422 µs (28.5%)
appsec_no_iast	1.721 ms [1.696 ms, 1.746 ms]	378.07 µs (28.1%)
iast	1.485 ms [1.462 ms, 1.508 ms]	142.13 µs (10.6%)
profiling	1.55 ms [1.525 ms, 1.575 ms]	206.725 µs (15.4%)
tracing	1.489 ms [1.464 ms, 1.513 ms]	145.452 µs (10.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.342 ms [1.322 ms, 1.361 ms]	-
appsec	1.722 ms [1.697 ms, 1.746 ms]	380.069 µs (28.3%)
appsec_no_iast	1.708 ms [1.683 ms, 1.732 ms]	366.072 µs (27.3%)
iast	1.486 ms [1.463 ms, 1.508 ms]	144.098 µs (10.7%)
profiling	1.486 ms [1.462 ms, 1.511 ms]	144.865 µs (10.8%)
tracing	1.479 ms [1.454 ms, 1.503 ms]	137.221 µs (10.2%)

Record public classes in compact filter

d2b8f33

mcculls added comp: core Tracer core tag: performance Performance related changes labels Mar 31, 2024

mcculls marked this pull request as ready for review April 1, 2024 08:16

mcculls requested a review from a team as a code owner April 1, 2024 08:16

mcculls requested review from dougqh, nayeem-kamal, PerfectSlayer and am312 April 1, 2024 08:16

dougqh approved these changes Apr 2, 2024

View reviewed changes

mcculls merged commit ca680ba into master Apr 2, 2024
77 checks passed

mcculls deleted the mcculls/add-visibility-filter branch April 2, 2024 23:42

github-actions bot added this to the 1.32.0 milestone Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep track of public classes in compact filter #6860

Keep track of public classes in compact filter #6860

mcculls commented Mar 31, 2024 •

edited

pr-commenter bot commented Apr 1, 2024

Keep track of public classes in compact filter #6860

Keep track of public classes in compact filter #6860

Conversation

mcculls commented Mar 31, 2024 • edited

Motivation

pr-commenter bot commented Apr 1, 2024

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

mcculls commented Mar 31, 2024 •

edited