Add BP trigger for DSM Kafka Benchmark #6173

ddyurchenko · 2023-11-08T15:56:51Z

What Does This Do

This PR adds automatic kafka benchmark runs (benchmarks developed by DSM team, @hokitam) on changes in kafka-clients instrumentation.

Also the benchmark can be run manually.

Regardless if run was manually or automatically triggered, PR comment will be posted in corresponding PR (if PR exists).

More detailed results of benchmark runs (with visualizations) are available in Gitlab artifacts and in BP UI.

Motivation

Add performance regression testing for Kafka instrumentation to prevent uncontrollable overhead increase.

Additional Notes

Jira ticket: [PROJ-IDENT]

pr-commenter · 2023-11-08T16:38:46Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/dsm_kafka_cluster-in-bp
git_commit_date	1702298139	1702305876
git_commit_sha	`00358aa`	`e7150ff`
release_version	1.26.0-SNAPSHOT~00358aaa1a	1.26.0-SNAPSHOT~e7150ff5e1

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702309306	1702309306
ci_job_id	386597977	386597977
ci_pipeline_id	24791797	24791797
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 44 metrics, 8 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:ProfilingAgent	worse [+13.089ms; +22.894ms] or [+9.727%; +17.015%]	152.549ms	134.557ms
scenario:startup:petclinic:profiling:Profiling	worse [+13.089ms; +22.904ms] or [+9.724%; +17.015%]	152.603ms	134.607ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.57 s) : 0, 1570153
Total [baseline] (12.909 s) : 0, 12909202
Agent [candidate] (1.567 s) : 0, 1567152
Total [candidate] (12.875 s) : 0, 12874945
section appsec
Agent [baseline] (1.701 s) : 0, 1700811
Total [baseline] (13.003 s) : 0, 13003280
Agent [candidate] (1.702 s) : 0, 1701652
Total [candidate] (12.98 s) : 0, 12979724
section iast
Agent [baseline] (1.718 s) : 0, 1718245
Total [baseline] (13.283 s) : 0, 13282532
Agent [candidate] (1.72 s) : 0, 1719854
Total [candidate] (13.282 s) : 0, 13281836
section profiling
Agent [baseline] (1.861 s) : 0, 1860787
Total [baseline] (13.226 s) : 0, 13225972
Agent [candidate] (1.897 s) : 0, 1896625
Total [candidate] (13.279 s) : 0, 13279145

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.57 s	-
Agent	appsec	1.701 s	130.658 ms (8.3%)
Agent	iast	1.718 s	148.092 ms (9.4%)
Agent	profiling	1.861 s	290.634 ms (18.5%)
Total	tracing	12.909 s	-
Total	appsec	13.003 s	94.078 ms (0.7%)
Total	iast	13.283 s	373.33 ms (2.9%)
Total	profiling	13.226 s	316.77 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.567 s	-
Agent	appsec	1.702 s	134.5 ms (8.6%)
Agent	iast	1.72 s	152.701 ms (9.7%)
Agent	profiling	1.897 s	329.473 ms (21.0%)
Total	tracing	12.875 s	-
Total	appsec	12.98 s	104.779 ms (0.8%)
Total	iast	13.282 s	406.891 ms (3.2%)
Total	profiling	13.279 s	404.199 ms (3.1%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (971.369 ms) : 0, 971369
BytebuddyAgent [candidate] (971.342 ms) : 0, 971342
GlobalTracer [baseline] (458.954 ms) : 0, 458954
GlobalTracer [candidate] (455.922 ms) : 0, 455922
AppSec [baseline] (69.325 ms) : 0, 69325
AppSec [candidate] (69.199 ms) : 0, 69199
Remote Config [baseline] (856.311 µs) : 0, 856
Remote Config [candidate] (851.735 µs) : 0, 852
Telemetry [baseline] (9.419 ms) : 0, 9419
Telemetry [candidate] (9.329 ms) : 0, 9329
section appsec
BytebuddyAgent [baseline] (967.212 ms) : 0, 967212
BytebuddyAgent [candidate] (971.302 ms) : 0, 971302
GlobalTracer [baseline] (458.16 ms) : 0, 458160
GlobalTracer [candidate] (455.202 ms) : 0, 455202
AppSec [baseline] (203.799 ms) : 0, 203799
AppSec [candidate] (204.18 ms) : 0, 204180
Remote Config [baseline] (797.913 µs) : 0, 798
Remote Config [candidate] (794.415 µs) : 0, 794
Telemetry [baseline] (10.779 ms) : 0, 10779
Telemetry [candidate] (9.882 ms) : 0, 9882
section iast
BytebuddyAgent [baseline] (1.136 s) : 0, 1136089
BytebuddyAgent [candidate] (1.137 s) : 0, 1137342
GlobalTracer [baseline] (420.545 ms) : 0, 420545
GlobalTracer [candidate] (420.872 ms) : 0, 420872
AppSec [baseline] (66.407 ms) : 0, 66407
AppSec [candidate] (66.806 ms) : 0, 66806
Remote Config [baseline] (764.901 µs) : 0, 765
Remote Config [candidate] (779.095 µs) : 0, 779
Telemetry [baseline] (9.95 ms) : 0, 9950
Telemetry [candidate] (11.497 ms) : 0, 11497
IAST [baseline] (24.476 ms) : 0, 24476
IAST [candidate] (22.185 ms) : 0, 22185
section profiling
BytebuddyAgent [baseline] (986.583 ms) : 0, 986583
BytebuddyAgent [candidate] (999.869 ms) : 0, 999869
GlobalTracer [baseline] (569.574 ms) : 0, 569574
GlobalTracer [candidate] (572.392 ms) : 0, 572392
AppSec [baseline] (69.801 ms) : 0, 69801
AppSec [candidate] (70.341 ms) : 0, 70341
Remote Config [baseline] (873.247 µs) : 0, 873
Remote Config [candidate] (871.003 µs) : 0, 871
Telemetry [baseline] (9.679 ms) : 0, 9679
Telemetry [candidate] (9.833 ms) : 0, 9833
ProfilingAgent [baseline] (134.557 ms) : 0, 134557
ProfilingAgent [candidate] (152.549 ms) : 0, 152549
Profiling [baseline] (134.607 ms) : 0, 134607
Profiling [candidate] (152.603 ms) : 0, 152603

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.565 s) : 0, 1565150
Total [baseline] (12.227 s) : 0, 12226978
Agent [candidate] (1.568 s) : 0, 1567684
Total [candidate] (12.226 s) : 0, 12226424
section iast
Agent [baseline] (1.724 s) : 0, 1724049
Total [baseline] (12.949 s) : 0, 12949446
Agent [candidate] (1.718 s) : 0, 1718398
Total [candidate] (12.967 s) : 0, 12967054
section iast_TELEMETRY_OFF
Agent [baseline] (1.705 s) : 0, 1704976
Total [baseline] (12.929 s) : 0, 12929492
Agent [candidate] (1.708 s) : 0, 1708397
Total [candidate] (12.885 s) : 0, 12885095

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.565 s	-
Agent	iast	1.724 s	158.9 ms (10.2%)
Agent	iast_TELEMETRY_OFF	1.705 s	139.827 ms (8.9%)
Total	tracing	12.227 s	-
Total	iast	12.949 s	722.468 ms (5.9%)
Total	iast_TELEMETRY_OFF	12.929 s	702.514 ms (5.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.568 s	-
Agent	iast	1.718 s	150.714 ms (9.6%)
Agent	iast_TELEMETRY_OFF	1.708 s	140.713 ms (9.0%)
Total	tracing	12.226 s	-
Total	iast	12.967 s	740.629 ms (6.1%)
Total	iast_TELEMETRY_OFF	12.885 s	658.671 ms (5.4%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (967.365 ms) : 0, 967365
BytebuddyAgent [candidate] (970.346 ms) : 0, 970346
GlobalTracer [baseline] (458.431 ms) : 0, 458431
GlobalTracer [candidate] (457.103 ms) : 0, 457103
AppSec [baseline] (68.772 ms) : 0, 68772
AppSec [candidate] (69.695 ms) : 0, 69695
Remote Config [baseline] (846.399 µs) : 0, 846
Remote Config [candidate] (855.929 µs) : 0, 856
Telemetry [baseline] (9.409 ms) : 0, 9409
Telemetry [candidate] (9.324 ms) : 0, 9324
section iast
BytebuddyAgent [baseline] (1.141 s) : 0, 1141249
BytebuddyAgent [candidate] (1.138 s) : 0, 1138122
GlobalTracer [baseline] (421.759 ms) : 0, 421759
GlobalTracer [candidate] (420.33 ms) : 0, 420330
AppSec [baseline] (66.654 ms) : 0, 66654
AppSec [candidate] (66.598 ms) : 0, 66598
Remote Config [baseline] (773.036 µs) : 0, 773
Remote Config [candidate] (751.861 µs) : 0, 752
Telemetry [baseline] (8.52 ms) : 0, 8520
Telemetry [candidate] (9.234 ms) : 0, 9234
IAST [baseline] (24.516 ms) : 0, 24516
IAST [candidate] (22.881 ms) : 0, 22881
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (1.123 s) : 0, 1123135
BytebuddyAgent [candidate] (1.127 s) : 0, 1126502
GlobalTracer [baseline] (420.709 ms) : 0, 420709
GlobalTracer [candidate] (420.688 ms) : 0, 420688
AppSec [baseline] (66.871 ms) : 0, 66871
AppSec [candidate] (67.401 ms) : 0, 67401
Remote Config [baseline] (772.371 µs) : 0, 772
Remote Config [candidate] (766.423 µs) : 0, 766
Telemetry [baseline] (10.663 ms) : 0, 10663
Telemetry [candidate] (9.902 ms) : 0, 9902
IAST [baseline] (22.543 ms) : 0, 22543
IAST [candidate] (22.653 ms) : 0, 22653

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-11T15:13:35	2023-12-11T15:36:12
git_branch	master	ddyurchenko/dsm_kafka_cluster-in-bp
git_commit_date	1702298139	1702305876
git_commit_sha	`00358aa`	`e7150ff`
release_version	1.26.0-SNAPSHOT~00358aaa1a	1.26.0-SNAPSHOT~e7150ff5e1
start_time	2023-12-11T15:13:19	2023-12-11T15:35:57

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702309306	1702309306
ci_job_id	386597977	386597977
ci_pipeline_id	24791797	24791797
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 6 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.72 ms) : 1695, 1745
.   : milestone, 1720,
appsec (2.226 ms) : 2192, 2259
.   : milestone, 2226,
iast (1.952 ms) : 1921, 1984
.   : milestone, 1952,
profiling (1.932 ms) : 1898, 1965
.   : milestone, 1932,
tracing (1.915 ms) : 1882, 1948
.   : milestone, 1915,
section candidate
no_agent (1.727 ms) : 1702, 1751
.   : milestone, 1727,
appsec (2.201 ms) : 2168, 2234
.   : milestone, 2201,
iast (1.94 ms) : 1907, 1972
.   : milestone, 1940,
profiling (1.896 ms) : 1862, 1931
.   : milestone, 1896,
tracing (1.917 ms) : 1883, 1951
.   : milestone, 1917,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.72 ms [1.695 ms, 1.745 ms]	-
appsec	2.226 ms [2.192 ms, 2.259 ms]	505.917 µs (29.4%)
iast	1.952 ms [1.921 ms, 1.984 ms]	232.609 µs (13.5%)
profiling	1.932 ms [1.898 ms, 1.965 ms]	211.908 µs (12.3%)
tracing	1.915 ms [1.882 ms, 1.948 ms]	194.992 µs (11.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.727 ms [1.702 ms, 1.751 ms]	-
appsec	2.201 ms [2.168 ms, 2.234 ms]	474.254 µs (27.5%)
iast	1.94 ms [1.907 ms, 1.972 ms]	212.877 µs (12.3%)
profiling	1.896 ms [1.862 ms, 1.931 ms]	169.595 µs (9.8%)
tracing	1.917 ms [1.883 ms, 1.951 ms]	190.307 µs (11.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~e7150ff5e1, baseline=1.26.0-SNAPSHOT~00358aaa1a
    dateFormat X
    axisFormat %s
section baseline
no_agent (445.096 µs) : 416, 474
.   : milestone, 445,
iast (582.739 µs) : 551, 614
.   : milestone, 583,
iast_FULL (671.188 µs) : 641, 701
.   : milestone, 671,
iast_INACTIVE (544.211 µs) : 513, 576
.   : milestone, 544,
iast_TELEMETRY_OFF (574.764 µs) : 543, 606
.   : milestone, 575,
tracing (533.737 µs) : 504, 564
.   : milestone, 534,
section candidate
no_agent (446.517 µs) : 418, 475
.   : milestone, 447,
iast (569.762 µs) : 538, 601
.   : milestone, 570,
iast_FULL (665.655 µs) : 635, 696
.   : milestone, 666,
iast_INACTIVE (545.725 µs) : 514, 577
.   : milestone, 546,
iast_TELEMETRY_OFF (575.831 µs) : 544, 607
.   : milestone, 576,
tracing (536.202 µs) : 507, 566
.   : milestone, 536,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	445.096 µs [416.337 µs, 473.854 µs]	-
iast	582.739 µs [551.35 µs, 614.128 µs]	137.643 µs (30.9%)
iast_FULL	671.188 µs [640.948 µs, 701.428 µs]	226.092 µs (50.8%)
iast_INACTIVE	544.211 µs [512.879 µs, 575.544 µs]	99.116 µs (22.3%)
iast_TELEMETRY_OFF	574.764 µs [543.031 µs, 606.496 µs]	129.668 µs (29.1%)
tracing	533.737 µs [503.808 µs, 563.665 µs]	88.641 µs (19.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	446.517 µs [417.776 µs, 475.259 µs]	-
iast	569.762 µs [538.488 µs, 601.035 µs]	123.244 µs (27.6%)
iast_FULL	665.655 µs [635.115 µs, 696.195 µs]	219.138 µs (49.1%)
iast_INACTIVE	545.725 µs [514.341 µs, 577.11 µs]	99.208 µs (22.2%)
iast_TELEMETRY_OFF	575.831 µs [544.384 µs, 607.278 µs]	129.313 µs (29.0%)
tracing	536.202 µs [506.55 µs, 565.854 µs]	89.685 µs (20.1%)

pr-commenter · 2023-12-11T14:01:45Z

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/dsm_kafka_cluster-in-bp
git_commit_date	1702298139	1702305876
git_commit_sha	`00358aa`	`e7150ff`

See matching parameters

	Baseline	Candidate
ci_job_date	1702307071	1702307071
ci_job_id	386597978	386597978
ci_pipeline_id	24791797	24791797
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.21	11.0.21
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.21+9-post-Ubuntu-0ubuntu122.04	11.0.21+9-post-Ubuntu-0ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

pr-commenter · 2023-12-11T14:55:08Z

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	ddyurchenko/dsm_kafka_cluster-in-bp
git_commit_date	1702298139	1702305876
git_commit_sha	`00358aa`	`e7150ff`

See matching parameters

	Baseline	Candidate
ci_job_date	1702307115	1702307115
ci_job_id	386597979	386597979
ci_pipeline_id	24791797	24791797
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.21	11.0.21
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.21+9-post-Ubuntu-0ubuntu122.04	11.0.21+9-post-Ubuntu-0ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

bm1549 added the comp: data streams Data Streams Monitoring label Nov 13, 2023

ddyurchenko added 6 commits December 11, 2023 15:20

Add DSM Kafka benchmark runs

5fe5438

Add CI jobs to run Kafka scenarios in parallel

3866069

Add CI jobs to run Kafka scenarios in parallel

eed5edf

Fix gitlab url

29aca20

Add space

d377ebc

Run kafka benchmarks automatically on kafka clients change

9187c1f

ddyurchenko force-pushed the ddyurchenko/dsm_kafka_cluster-in-bp branch from 780f1e8 to 9187c1f Compare December 11, 2023 14:20

Use tweaked-metal runner instead of same-cpu

e7150ff

ddyurchenko marked this pull request as ready for review December 11, 2023 15:35

ddyurchenko requested a review from a team as a code owner December 11, 2023 15:35

ddyurchenko requested review from dougqh, PerfectSlayer and hokitam December 11, 2023 15:35

hokitam approved these changes Dec 11, 2023

View reviewed changes

ddyurchenko merged commit 32608fb into master Dec 12, 2023
74 checks passed

ddyurchenko deleted the ddyurchenko/dsm_kafka_cluster-in-bp branch December 12, 2023 09:35

github-actions bot added this to the 1.26.0 milestone Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BP trigger for DSM Kafka Benchmark #6173

Add BP trigger for DSM Kafka Benchmark #6173

ddyurchenko commented Nov 8, 2023 •

edited

Loading

pr-commenter bot commented Nov 8, 2023 •

edited

Loading

pr-commenter bot commented Dec 11, 2023 •

edited

Loading

pr-commenter bot commented Dec 11, 2023 •

edited

Loading

Add BP trigger for DSM Kafka Benchmark #6173

Add BP trigger for DSM Kafka Benchmark #6173

Conversation

ddyurchenko commented Nov 8, 2023 • edited Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Nov 8, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

pr-commenter bot commented Dec 11, 2023 • edited Loading

Kafka / producer-benchmark

Parameters

Summary

pr-commenter bot commented Dec 11, 2023 • edited Loading

Kafka / consumer-benchmark

Parameters

Summary

ddyurchenko commented Nov 8, 2023 •

edited

Loading

pr-commenter bot commented Nov 8, 2023 •

edited

Loading

pr-commenter bot commented Dec 11, 2023 •

edited

Loading

pr-commenter bot commented Dec 11, 2023 •

edited

Loading