Update system-tests to 68da674b83d7a1a1730387a3b27a1d4328392f3c by smola · Pull Request #5703 · DataDog/dd-trace-java

smola · 2023-08-09T10:17:56Z

What Does This Do

Comparison:
https://github.com/datadog/system-tests/compare/18afda441d6e92fe3befeddd4dd20ff558ca4f0c..68da674b83d7a1a1730387a3b27a1d4328392f3c

Motivation

Additional Notes

Relevant changes:

pr-commenter · 2023-08-09T10:52:40Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.19.0~b023c07015	1.20.0-SNAPSHOT~da63d036f8
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (994.178 ms) : 0, 994178
Total [baseline] (8.657 s) : 0, 8656844
Agent [candidate] (987.259 ms) : 0, 987259
Total [candidate] (8.654 s) : 0, 8654473
section appsec
Agent [baseline] (1.077 s) : 0, 1076954
Total [baseline] (8.766 s) : 0, 8766242
Agent [candidate] (1.072 s) : 0, 1072422
Total [candidate] (8.727 s) : 0, 8727096
section iast
Agent [baseline] (1.086 s) : 0, 1086392
Total [baseline] (9.168 s) : 0, 9167559
Agent [candidate] (1.086 s) : 0, 1086118
Total [candidate] (9.206 s) : 0, 9206000
section profiling
Agent [baseline] (1.163 s) : 0, 1163432
Total [baseline] (8.966 s) : 0, 8965880
Agent [candidate] (1.159 s) : 0, 1158607
Total [candidate] (8.89 s) : 0, 8890239

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	994.178 ms	-
Agent	appsec	1.077 s	82.777 ms (8.3%)
Agent	iast	1.086 s	92.214 ms (9.3%)
Agent	profiling	1.163 s	169.254 ms (17.0%)
Total	tracing	8.657 s	-
Total	appsec	8.766 s	109.398 ms (1.3%)
Total	iast	9.168 s	510.715 ms (5.9%)
Total	profiling	8.966 s	309.036 ms (3.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	987.259 ms	-
Agent	appsec	1.072 s	85.163 ms (8.6%)
Agent	iast	1.086 s	98.859 ms (10.0%)
Agent	profiling	1.159 s	171.348 ms (17.4%)
Total	tracing	8.654 s	-
Total	appsec	8.727 s	72.623 ms (0.8%)
Total	iast	9.206 s	551.527 ms (6.4%)
Total	profiling	8.89 s	235.766 ms (2.7%)

gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (615.947 ms) : 0, 615947
BytebuddyAgent [candidate] (612.122 ms) : 0, 612122
GlobalTracer [baseline] (286.992 ms) : 0, 286992
GlobalTracer [candidate] (284.778 ms) : 0, 284778
AppSec [baseline] (50.355 ms) : 0, 50355
AppSec [candidate] (49.971 ms) : 0, 49971
Remote Config [baseline] (674.925 µs) : 0, 675
Remote Config [candidate] (667.049 µs) : 0, 667
Telemetry [baseline] (5.448 ms) : 0, 5448
Telemetry [candidate] (5.35 ms) : 0, 5350
section appsec
BytebuddyAgent [baseline] (615.028 ms) : 0, 615028
BytebuddyAgent [candidate] (612.054 ms) : 0, 612054
GlobalTracer [baseline] (285.55 ms) : 0, 285550
GlobalTracer [candidate] (284.751 ms) : 0, 284751
AppSec [baseline] (136.04 ms) : 0, 136040
AppSec [candidate] (135.499 ms) : 0, 135499
Remote Config [baseline] (745.948 µs) : 0, 746
Remote Config [candidate] (728.688 µs) : 0, 729
Telemetry [baseline] (5.181 ms) : 0, 5181
Telemetry [candidate] (5.148 ms) : 0, 5148
section iast
BytebuddyAgent [baseline] (713.856 ms) : 0, 713856
BytebuddyAgent [candidate] (713.417 ms) : 0, 713417
GlobalTracer [baseline] (271.156 ms) : 0, 271156
GlobalTracer [candidate] (271.754 ms) : 0, 271754
AppSec [baseline] (47.37 ms) : 0, 47370
AppSec [candidate] (47.199 ms) : 0, 47199
Remote Config [baseline] (651.549 µs) : 0, 652
Remote Config [candidate] (639.085 µs) : 0, 639
Telemetry [baseline] (5.628 ms) : 0, 5628
Telemetry [candidate] (5.518 ms) : 0, 5518
IAST [baseline] (13.313 ms) : 0, 13313
IAST [candidate] (13.296 ms) : 0, 13296
section profiling
BytebuddyAgent [baseline] (627.877 ms) : 0, 627877
BytebuddyAgent [candidate] (624.234 ms) : 0, 624234
GlobalTracer [baseline] (351.195 ms) : 0, 351195
GlobalTracer [candidate] (349.656 ms) : 0, 349656
AppSec [baseline] (49.574 ms) : 0, 49574
AppSec [candidate] (49.322 ms) : 0, 49322
Remote Config [baseline] (660.496 µs) : 0, 660
Remote Config [candidate] (654.75 µs) : 0, 655
Telemetry [baseline] (5.404 ms) : 0, 5404
Telemetry [candidate] (5.354 ms) : 0, 5354
ProfilingAgent [baseline] (75.398 ms) : 0, 75398
ProfilingAgent [candidate] (76.591 ms) : 0, 76591
Profiling [baseline] (75.423 ms) : 0, 75423
Profiling [candidate] (76.617 ms) : 0, 76617

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (986.466 ms) : 0, 986466
Total [baseline] (9.193 s) : 0, 9192626
Agent [candidate] (995.038 ms) : 0, 995038
Total [candidate] (9.241 s) : 0, 9241223
section appsec
Agent [baseline] (1.08 s) : 0, 1080430
Total [baseline] (9.286 s) : 0, 9286019
Agent [candidate] (1.074 s) : 0, 1073942
Total [candidate] (9.287 s) : 0, 9286545
section iast
Agent [baseline] (1.088 s) : 0, 1087717
Total [baseline] (9.468 s) : 0, 9468436
Agent [candidate] (1.086 s) : 0, 1085827
Total [candidate] (9.446 s) : 0, 9445719
section profiling
Agent [baseline] (1.169 s) : 0, 1168677
Total [baseline] (9.578 s) : 0, 9577553
Agent [candidate] (1.165 s) : 0, 1165185
Total [candidate] (9.533 s) : 0, 9532673

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	986.466 ms	-
Agent	appsec	1.08 s	93.964 ms (9.5%)
Agent	iast	1.088 s	101.252 ms (10.3%)
Agent	profiling	1.169 s	182.211 ms (18.5%)
Total	tracing	9.193 s	-
Total	appsec	9.286 s	93.393 ms (1.0%)
Total	iast	9.468 s	275.81 ms (3.0%)
Total	profiling	9.578 s	384.927 ms (4.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	995.038 ms	-
Agent	appsec	1.074 s	78.904 ms (7.9%)
Agent	iast	1.086 s	90.789 ms (9.1%)
Agent	profiling	1.165 s	170.147 ms (17.1%)
Total	tracing	9.241 s	-
Total	appsec	9.287 s	45.322 ms (0.5%)
Total	iast	9.446 s	204.496 ms (2.2%)
Total	profiling	9.533 s	291.45 ms (3.2%)

gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (610.974 ms) : 0, 610974
BytebuddyAgent [candidate] (616.535 ms) : 0, 616535
GlobalTracer [baseline] (284.641 ms) : 0, 284641
GlobalTracer [candidate] (287.744 ms) : 0, 287744
AppSec [baseline] (50.384 ms) : 0, 50384
AppSec [candidate] (50.163 ms) : 0, 50163
Remote Config [baseline] (663.523 µs) : 0, 664
Remote Config [candidate] (674.273 µs) : 0, 674
Telemetry [baseline] (5.376 ms) : 0, 5376
Telemetry [candidate] (5.457 ms) : 0, 5457
section appsec
BytebuddyAgent [baseline] (615.927 ms) : 0, 615927
BytebuddyAgent [candidate] (612.858 ms) : 0, 612858
GlobalTracer [baseline] (287.554 ms) : 0, 287554
GlobalTracer [candidate] (286.01 ms) : 0, 286010
AppSec [baseline] (136.39 ms) : 0, 136390
AppSec [candidate] (134.798 ms) : 0, 134798
Remote Config [baseline] (740.415 µs) : 0, 740
Remote Config [candidate] (722.008 µs) : 0, 722
Telemetry [baseline] (5.179 ms) : 0, 5179
Telemetry [candidate] (5.221 ms) : 0, 5221
section iast
BytebuddyAgent [baseline] (714.437 ms) : 0, 714437
BytebuddyAgent [candidate] (712.751 ms) : 0, 712751
GlobalTracer [baseline] (271.847 ms) : 0, 271847
GlobalTracer [candidate] (271.999 ms) : 0, 271999
AppSec [baseline] (47.365 ms) : 0, 47365
AppSec [candidate] (47.201 ms) : 0, 47201
Remote Config [baseline] (649.144 µs) : 0, 649
Remote Config [candidate] (638.065 µs) : 0, 638
Telemetry [baseline] (5.629 ms) : 0, 5629
Telemetry [candidate] (5.587 ms) : 0, 5587
IAST [baseline] (13.374 ms) : 0, 13374
IAST [candidate] (13.375 ms) : 0, 13375
section profiling
BytebuddyAgent [baseline] (630.901 ms) : 0, 630901
BytebuddyAgent [candidate] (628.013 ms) : 0, 628013
GlobalTracer [baseline] (353.043 ms) : 0, 353043
GlobalTracer [candidate] (352.56 ms) : 0, 352560
AppSec [baseline] (49.798 ms) : 0, 49798
AppSec [candidate] (49.683 ms) : 0, 49683
Remote Config [baseline] (663.76 µs) : 0, 664
Remote Config [candidate] (658.125 µs) : 0, 658
Telemetry [baseline] (5.441 ms) : 0, 5441
Telemetry [candidate] (5.395 ms) : 0, 5395
ProfilingAgent [baseline] (75.226 ms) : 0, 75226
ProfilingAgent [candidate] (75.569 ms) : 0, 75569
Profiling [baseline] (75.251 ms) : 0, 75251
Profiling [candidate] (75.595 ms) : 0, 75595

Load

Parameters

	Baseline	Candidate
commit	1.19.0~b023c07015	1.20.0-SNAPSHOT~da63d036f8
config	baseline	candidate
end_time	2023-08-09T10:29:16	2023-08-09T10:47:24
start_time	2023-08-09T10:28:59	2023-08-09T10:47:07

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015
    dateFormat X
    axisFormat %s
section baseline
no_agent (359.82 µs) : 340, 380
.   : milestone, 360,
appsec (668.605 µs) : 648, 689
.   : milestone, 669,
iast (466.767 µs) : 446, 488
.   : milestone, 467,
iast_FULL (534.315 µs) : 513, 555
.   : milestone, 534,
iast_INACTIVE (439.909 µs) : 419, 461
.   : milestone, 440,
profiling (434.977 µs) : 415, 455
.   : milestone, 435,
tracing (436.919 µs) : 416, 458
.   : milestone, 437,
section candidate
no_agent (363.867 µs) : 344, 384
.   : milestone, 364,
appsec (660.563 µs) : 640, 681
.   : milestone, 661,
iast (461.138 µs) : 440, 482
.   : milestone, 461,
iast_FULL (527.104 µs) : 506, 548
.   : milestone, 527,
iast_INACTIVE (444.427 µs) : 422, 467
.   : milestone, 444,
profiling (438.258 µs) : 417, 460
.   : milestone, 438,
tracing (434.279 µs) : 414, 455
.   : milestone, 434,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	359.82 µs [339.881 µs, 379.758 µs]	-
appsec	668.605 µs [647.881 µs, 689.329 µs]	308.786 µs (85.8%)
iast	466.767 µs [445.739 µs, 487.795 µs]	106.948 µs (29.7%)
iast_FULL	534.315 µs [513.427 µs, 555.203 µs]	174.495 µs (48.5%)
iast_INACTIVE	439.909 µs [419.019 µs, 460.8 µs]	80.09 µs (22.3%)
profiling	434.977 µs [414.577 µs, 455.378 µs]	75.158 µs (20.9%)
tracing	436.919 µs [415.839 µs, 457.998 µs]	77.099 µs (21.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	363.867 µs [343.61 µs, 384.124 µs]	-
appsec	660.563 µs [640.404 µs, 680.721 µs]	296.696 µs (81.5%)
iast	461.138 µs [440.176 µs, 482.101 µs]	97.271 µs (26.7%)
iast_FULL	527.104 µs [506.397 µs, 547.812 µs]	163.237 µs (44.9%)
iast_INACTIVE	444.427 µs [422.08 µs, 466.774 µs]	80.56 µs (22.1%)
profiling	438.258 µs [416.6 µs, 459.916 µs]	74.391 µs (20.4%)
tracing	434.279 µs [413.777 µs, 454.781 µs]	70.412 µs (19.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~da63d036f8, baseline=1.19.0~b023c07015
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.348 ms) : 1329, 1367
.   : milestone, 1348,
appsec (1.683 ms) : 1659, 1707
.   : milestone, 1683,
iast (1.471 ms) : 1447, 1496
.   : milestone, 1471,
profiling (1.438 ms) : 1412, 1464
.   : milestone, 1438,
tracing (1.468 ms) : 1443, 1492
.   : milestone, 1468,
section candidate
no_agent (1.348 ms) : 1329, 1368
.   : milestone, 1348,
appsec (1.681 ms) : 1657, 1705
.   : milestone, 1681,
iast (1.465 ms) : 1440, 1490
.   : milestone, 1465,
profiling (1.49 ms) : 1465, 1515
.   : milestone, 1490,
tracing (1.466 ms) : 1442, 1490
.   : milestone, 1466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.348 ms [1.329 ms, 1.367 ms]	-
appsec	1.683 ms [1.659 ms, 1.707 ms]	334.913 µs (24.8%)
iast	1.471 ms [1.447 ms, 1.496 ms]	123.332 µs (9.1%)
profiling	1.438 ms [1.412 ms, 1.464 ms]	90.001 µs (6.7%)
tracing	1.468 ms [1.443 ms, 1.492 ms]	119.456 µs (8.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.348 ms [1.329 ms, 1.368 ms]	-
appsec	1.681 ms [1.657 ms, 1.705 ms]	332.792 µs (24.7%)
iast	1.465 ms [1.44 ms, 1.49 ms]	116.523 µs (8.6%)
profiling	1.49 ms [1.465 ms, 1.515 ms]	141.535 µs (10.5%)
tracing	1.466 ms [1.442 ms, 1.49 ms]	117.852 µs (8.7%)

Update system-tests to 68da674b83d7a1a1730387a3b27a1d4328392f3c

da63d03

smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Aug 9, 2023

smola requested a review from a team as a code owner August 9, 2023 10:17

smola enabled auto-merge (squash) August 9, 2023 14:47

manuel-alvarez-alvarez approved these changes Aug 10, 2023

View reviewed changes

smola merged commit 8c080b4 into master Aug 10, 2023

smola deleted the smola/system-tests-20230809 branch August 10, 2023 09:03

github-actions Bot added this to the 1.20.0 milestone Aug 10, 2023

smola mentioned this pull request Aug 18, 2023

Update system-tests to 9659a5e409cf8bacabd739b6af07c5f03c9bcbd6 #5738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update system-tests to 68da674b83d7a1a1730387a3b27a1d4328392f3c#5703

Update system-tests to 68da674b83d7a1a1730387a3b27a1d4328392f3c#5703
smola merged 1 commit intomasterfrom
smola/system-tests-20230809

smola commented Aug 9, 2023

Uh oh!

pr-commenter Bot commented Aug 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

smola commented Aug 9, 2023

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Aug 9, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants