Ensure test.framework and test.framework_version with multi-values have fixed order #6271

nikita-tkachenko-datadog · 2023-11-23T19:22:38Z

What Does This Do

Adds sorting to test.framework and test.framework_version tags that contain multiple values.

Motivation

In projects where tests are executed in parallel and multiple test frameworks are used the contents of the test framework name and version tags are non-deterministic: they depend on the order in which corresponding modules or test suites finished execution, which changes from run to run.
These tags are used for some internal analytics, and having non-deterministic content in them complicates analysis.

Additional Notes

In scope of this change Gradle Daemon instrumentation smoke test was rewritten to use the new JSON-based spans comparison mechanism.

Jira ticket: CIVIS-8119

pr-commenter · 2023-11-23T20:01:33Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/test-framework-tags-fixed-order
git_commit_date	1701249870	1701250079
git_commit_sha	`241f6c6`	`bcaa312`
release_version	1.25.0-SNAPSHOT~241f6c6a04	1.25.0-SNAPSHOT~bcaa312d0d

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701252794	1701252794
ci_job_id	377562963	377562963
ci_pipeline_id	23915981	23915981
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1029794
Total [baseline] (9.36 s) : 0, 9359817
Agent [candidate] (1.036 s) : 0, 1035765
Total [candidate] (9.382 s) : 0, 9382380
section appsec
Agent [baseline] (1.121 s) : 0, 1121424
Total [baseline] (9.419 s) : 0, 9419272
Agent [candidate] (1.125 s) : 0, 1125273
Total [candidate] (9.434 s) : 0, 9433871
section iast
Agent [baseline] (1.151 s) : 0, 1151145
Total [baseline] (9.6 s) : 0, 9599905
Agent [candidate] (1.15 s) : 0, 1150466
Total [candidate] (9.583 s) : 0, 9582958
section profiling
Agent [baseline] (1.222 s) : 0, 1222033
Total [baseline] (9.607 s) : 0, 9607252
Agent [candidate] (1.222 s) : 0, 1222409
Total [candidate] (9.615 s) : 0, 9615352

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.03 s	-
Agent	appsec	1.121 s	91.63 ms (8.9%)
Agent	iast	1.151 s	121.352 ms (11.8%)
Agent	profiling	1.222 s	192.239 ms (18.7%)
Total	tracing	9.36 s	-
Total	appsec	9.419 s	59.456 ms (0.6%)
Total	iast	9.6 s	240.089 ms (2.6%)
Total	profiling	9.607 s	247.435 ms (2.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	appsec	1.125 s	89.508 ms (8.6%)
Agent	iast	1.15 s	114.701 ms (11.1%)
Agent	profiling	1.222 s	186.644 ms (18.0%)
Total	tracing	9.382 s	-
Total	appsec	9.434 s	51.491 ms (0.5%)
Total	iast	9.583 s	200.578 ms (2.1%)
Total	profiling	9.615 s	232.972 ms (2.5%)

gantt
    title petclinic - break down per module: candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (641.997 ms) : 0, 641997
BytebuddyAgent [candidate] (644.444 ms) : 0, 644444
GlobalTracer [baseline] (297.382 ms) : 0, 297382
GlobalTracer [candidate] (300.199 ms) : 0, 300199
AppSec [baseline] (48.349 ms) : 0, 48349
AppSec [candidate] (48.841 ms) : 0, 48841
Remote Config [baseline] (683.296 µs) : 0, 683
Remote Config [candidate] (695.829 µs) : 0, 696
Telemetry [baseline] (7.213 ms) : 0, 7213
Telemetry [candidate] (7.348 ms) : 0, 7348
section appsec
BytebuddyAgent [baseline] (644.485 ms) : 0, 644485
BytebuddyAgent [candidate] (647.286 ms) : 0, 647286
GlobalTracer [baseline] (298.194 ms) : 0, 298194
GlobalTracer [candidate] (299.322 ms) : 0, 299322
AppSec [baseline] (136.869 ms) : 0, 136869
AppSec [candidate] (136.842 ms) : 0, 136842
Remote Config [baseline] (649.412 µs) : 0, 649
Remote Config [candidate] (644.262 µs) : 0, 644
Telemetry [baseline] (6.893 ms) : 0, 6893
Telemetry [candidate] (6.816 ms) : 0, 6816
section iast
BytebuddyAgent [baseline] (764.208 ms) : 0, 764208
BytebuddyAgent [candidate] (763.785 ms) : 0, 763785
GlobalTracer [baseline] (277.803 ms) : 0, 277803
GlobalTracer [candidate] (277.794 ms) : 0, 277794
AppSec [baseline] (47.181 ms) : 0, 47181
AppSec [candidate] (49.383 ms) : 0, 49383
IAST [baseline] (17.651 ms) : 0, 17651
IAST [candidate] (16.119 ms) : 0, 16119
Remote Config [baseline] (1.254 ms) : 0, 1254
Remote Config [candidate] (566.786 µs) : 0, 567
Telemetry [baseline] (8.773 ms) : 0, 8773
Telemetry [candidate] (8.646 ms) : 0, 8646
section profiling
BytebuddyAgent [baseline] (654.323 ms) : 0, 654323
BytebuddyAgent [candidate] (654.585 ms) : 0, 654585
GlobalTracer [baseline] (368.799 ms) : 0, 368799
GlobalTracer [candidate] (369.198 ms) : 0, 369198
AppSec [baseline] (48.574 ms) : 0, 48574
AppSec [candidate] (48.41 ms) : 0, 48410
Remote Config [baseline] (720.183 µs) : 0, 720
Remote Config [candidate] (709.794 µs) : 0, 710
Telemetry [baseline] (7.39 ms) : 0, 7390
Telemetry [candidate] (7.4 ms) : 0, 7400
ProfilingAgent [baseline] (88.12 ms) : 0, 88120
ProfilingAgent [candidate] (88.204 ms) : 0, 88204
Profiling [baseline] (88.143 ms) : 0, 88143
Profiling [candidate] (88.227 ms) : 0, 88227

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.027 s) : 0, 1027354
Total [baseline] (8.702 s) : 0, 8702404
Agent [candidate] (1.041 s) : 0, 1041434
Total [candidate] (8.707 s) : 0, 8707021
section iast
Agent [baseline] (1.159 s) : 0, 1159002
Total [baseline] (9.285 s) : 0, 9284759
Agent [candidate] (1.152 s) : 0, 1152499
Total [candidate] (9.215 s) : 0, 9214689
section iast_TELEMETRY_OFF
Agent [baseline] (1.143 s) : 0, 1143496
Total [baseline] (9.214 s) : 0, 9214019
Agent [candidate] (1.144 s) : 0, 1144186
Total [candidate] (9.195 s) : 0, 9194535

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.027 s	-
Agent	iast	1.159 s	131.648 ms (12.8%)
Agent	iast_TELEMETRY_OFF	1.143 s	116.143 ms (11.3%)
Total	tracing	8.702 s	-
Total	iast	9.285 s	582.355 ms (6.7%)
Total	iast_TELEMETRY_OFF	9.214 s	511.615 ms (5.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	iast	1.152 s	111.064 ms (10.7%)
Agent	iast_TELEMETRY_OFF	1.144 s	102.751 ms (9.9%)
Total	tracing	8.707 s	-
Total	iast	9.215 s	507.668 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.195 s	487.514 ms (5.6%)

gantt
    title insecure-bank - break down per module: candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (640.821 ms) : 0, 640821
BytebuddyAgent [candidate] (649.093 ms) : 0, 649093
GlobalTracer [baseline] (296.6 ms) : 0, 296600
GlobalTracer [candidate] (300.658 ms) : 0, 300658
AppSec [baseline] (48.007 ms) : 0, 48007
AppSec [candidate] (49.145 ms) : 0, 49145
Remote Config [baseline] (673.929 µs) : 0, 674
Remote Config [candidate] (691.325 µs) : 0, 691
Telemetry [baseline] (7.185 ms) : 0, 7185
Telemetry [candidate] (7.324 ms) : 0, 7324
section iast
BytebuddyAgent [baseline] (769.609 ms) : 0, 769609
BytebuddyAgent [candidate] (764.808 ms) : 0, 764808
GlobalTracer [baseline] (279.382 ms) : 0, 279382
GlobalTracer [candidate] (279.032 ms) : 0, 279032
AppSec [baseline] (49.241 ms) : 0, 49241
AppSec [candidate] (48.244 ms) : 0, 48244
IAST [baseline] (17.798 ms) : 0, 17798
IAST [candidate] (16.951 ms) : 0, 16951
Remote Config [baseline] (588.749 µs) : 0, 589
Remote Config [candidate] (579.685 µs) : 0, 580
Telemetry [baseline] (7.986 ms) : 0, 7986
Telemetry [candidate] (8.643 ms) : 0, 8643
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (757.924 ms) : 0, 757924
BytebuddyAgent [candidate] (758.149 ms) : 0, 758149
GlobalTracer [baseline] (278.085 ms) : 0, 278085
GlobalTracer [candidate] (279.106 ms) : 0, 279106
AppSec [baseline] (46.568 ms) : 0, 46568
AppSec [candidate] (46.916 ms) : 0, 46916
IAST [baseline] (18.845 ms) : 0, 18845
IAST [candidate] (15.839 ms) : 0, 15839
Remote Config [baseline] (563.562 µs) : 0, 564
Remote Config [candidate] (560.549 µs) : 0, 561
Telemetry [baseline] (7.165 ms) : 0, 7165
Telemetry [candidate] (9.333 ms) : 0, 9333

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-11-29T09:52:28	2023-11-29T10:09:00
git_branch	master	nikita-tkachenko/test-framework-tags-fixed-order
git_commit_date	1701249870	1701250079
git_commit_sha	`241f6c6`	`bcaa312`
release_version	1.25.0-SNAPSHOT~241f6c6a04	1.25.0-SNAPSHOT~bcaa312d0d
start_time	2023-11-29T09:52:15	2023-11-29T10:08:47

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701252794	1701252794
ci_job_id	377562963	377562963
ci_pipeline_id	23915981	23915981
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.352 ms) : 1333, 1371
.   : milestone, 1352,
appsec (1.768 ms) : 1743, 1792
.   : milestone, 1768,
iast (1.514 ms) : 1490, 1539
.   : milestone, 1514,
profiling (1.51 ms) : 1485, 1536
.   : milestone, 1510,
tracing (1.498 ms) : 1473, 1523
.   : milestone, 1498,
section candidate
no_agent (1.35 ms) : 1332, 1369
.   : milestone, 1350,
appsec (1.756 ms) : 1731, 1781
.   : milestone, 1756,
iast (1.521 ms) : 1497, 1545
.   : milestone, 1521,
profiling (1.541 ms) : 1514, 1567
.   : milestone, 1541,
tracing (1.526 ms) : 1501, 1552
.   : milestone, 1526,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.352 ms [1.333 ms, 1.371 ms]	-
appsec	1.768 ms [1.743 ms, 1.792 ms]	415.776 µs (30.8%)
iast	1.514 ms [1.49 ms, 1.539 ms]	162.696 µs (12.0%)
profiling	1.51 ms [1.485 ms, 1.536 ms]	158.684 µs (11.7%)
tracing	1.498 ms [1.473 ms, 1.523 ms]	146.233 µs (10.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.35 ms [1.332 ms, 1.369 ms]	-
appsec	1.756 ms [1.731 ms, 1.781 ms]	405.725 µs (30.0%)
iast	1.521 ms [1.497 ms, 1.545 ms]	170.63 µs (12.6%)
profiling	1.541 ms [1.514 ms, 1.567 ms]	190.331 µs (14.1%)
tracing	1.526 ms [1.501 ms, 1.552 ms]	175.898 µs (13.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~bcaa312d0d, baseline=1.25.0-SNAPSHOT~241f6c6a04
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.318 µs) : 345, 386
.   : milestone, 365,
iast (468.72 µs) : 448, 489
.   : milestone, 469,
iast_FULL (531.986 µs) : 512, 552
.   : milestone, 532,
iast_INACTIVE (451.745 µs) : 431, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (468.81 µs) : 448, 490
.   : milestone, 469,
tracing (446.174 µs) : 425, 468
.   : milestone, 446,
section candidate
no_agent (367.954 µs) : 347, 389
.   : milestone, 368,
iast (472.551 µs) : 452, 493
.   : milestone, 473,
iast_FULL (539.464 µs) : 519, 560
.   : milestone, 539,
iast_INACTIVE (440.382 µs) : 420, 461
.   : milestone, 440,
iast_TELEMETRY_OFF (470.169 µs) : 450, 491
.   : milestone, 470,
tracing (441.514 µs) : 421, 462
.   : milestone, 442,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.318 µs [344.735 µs, 385.9 µs]	-
iast	468.72 µs [448.499 µs, 488.942 µs]	103.403 µs (28.3%)
iast_FULL	531.986 µs [511.631 µs, 552.341 µs]	166.668 µs (45.6%)
iast_INACTIVE	451.745 µs [430.605 µs, 472.884 µs]	86.427 µs (23.7%)
iast_TELEMETRY_OFF	468.81 µs [447.587 µs, 490.034 µs]	103.493 µs (28.3%)
tracing	446.174 µs [424.687 µs, 467.661 µs]	80.857 µs (22.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.954 µs [346.577 µs, 389.331 µs]	-
iast	472.551 µs [451.715 µs, 493.387 µs]	104.598 µs (28.4%)
iast_FULL	539.464 µs [518.928 µs, 559.999 µs]	171.51 µs (46.6%)
iast_INACTIVE	440.382 µs [419.8 µs, 460.963 µs]	72.428 µs (19.7%)
iast_TELEMETRY_OFF	470.169 µs [449.59 µs, 490.748 µs]	102.215 µs (27.8%)
tracing	441.514 µs [420.987 µs, 462.041 µs]	73.561 µs (20.0%)

…ve fixed order

…alues have fixed order (#6271)" This reverts commit 672726b.

nikita-tkachenko-datadog added comp: ci visibility Continuous Integration Visibility run-tests: ibm8 Run tests using IBM 8 JDK labels Nov 23, 2023

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/propagate-itr-tags-to-session branch from 72859ed to e43ed9e Compare November 28, 2023 11:04

Base automatically changed from nikita-tkachenko/propagate-itr-tags-to-session to master November 28, 2023 16:21

nikita-tkachenko-datadog added 2 commits November 28, 2023 17:52

Ensure test.framework and test.framework_version with multi-values ha…

7848427

…ve fixed order

Fix repo index builder to use repo root instead of project root

5db5fd4

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/test-framework-tags-fixed-order branch from 657ce4d to 5db5fd4 Compare November 28, 2023 16:53

Fix compilation errors post merge conflicts resolution

6edd1ad

nikita-tkachenko-datadog marked this pull request as ready for review November 28, 2023 19:00

nikita-tkachenko-datadog requested review from a team as code owners November 28, 2023 19:00

nikita-tkachenko-datadog requested review from anmarchenko, mcculls and PerfectSlayer November 28, 2023 19:00

anmarchenko approved these changes Nov 29, 2023

View reviewed changes

nikita-tkachenko-datadog removed the run-tests: ibm8 Run tests using IBM 8 JDK label Nov 29, 2023

Commit to re-run tests

bcaa312

nikita-tkachenko-datadog merged commit 672726b into master Nov 29, 2023
71 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/test-framework-tags-fixed-order branch November 29, 2023 10:56

github-actions bot added this to the 1.25.0 milestone Nov 29, 2023

nikita-tkachenko-datadog added a commit that referenced this pull request Dec 4, 2023

Revert "Ensure test.framework and test.framework_version with multi-v…

b19b9a1

…alues have fixed order (#6271)" This reverts commit 672726b.

nikita-tkachenko-datadog added a commit that referenced this pull request Dec 4, 2023

Revert "Ensure test.framework and test.framework_version with multi-v…

f17d6fc

…alues have fixed order (#6271)" This reverts commit 672726b.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure test.framework and test.framework_version with multi-values have fixed order #6271

Ensure test.framework and test.framework_version with multi-values have fixed order #6271

nikita-tkachenko-datadog commented Nov 23, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Nov 23, 2023 •

edited

Loading

Ensure test.framework and test.framework_version with multi-values have fixed order #6271

Ensure test.framework and test.framework_version with multi-values have fixed order #6271

Conversation

nikita-tkachenko-datadog commented Nov 23, 2023 • edited by jira bot Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Nov 23, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

nikita-tkachenko-datadog commented Nov 23, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Nov 23, 2023 •

edited

Loading