Added test suite level visibility for JUnit 3.8 test cases #6320

nikita-tkachenko-datadog · 2023-12-05T17:14:34Z

What Does This Do

Add CI Visibility support for JUnit 3-style test cases: the ones that extend junit.framework.TestCase rather than using the standard @Test JUnit 4 annotation.
The missing piece for those test cases were test suite started/finished events, which JUnit does not emit.
Custom events were added by instrumenting JUnit 3.8 runner.

Motivation

Although this is not the recommended way of creating test cases, JUnit 4 supports it.
There are projects that contain such tests (so far one Datadog customer and one internal dogfooding repo)

Additional Notes

In scope of this PR Junit4Utils class was refactored to use MethodHandles helper instead of the boilerplate reflection code.

Jira ticket: CIVIS-8237

pr-commenter · 2023-12-05T17:55:14Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/fix-junit-38-test-cases
git_commit_date	1701791196	1701796195
git_commit_sha	`d3754fe`	`5b2777a`
release_version	1.26.0-SNAPSHOT~d3754fe8b0	1.26.0-SNAPSHOT~5b2777a951

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701798872	1701798872
ci_job_id	382350505	382350505
ci_pipeline_id	24430604	24430604
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049279
Total [baseline] (8.702 s) : 0, 8701842
Agent [candidate] (1.051 s) : 0, 1051134
Total [candidate] (8.686 s) : 0, 8685890
section iast
Agent [baseline] (1.159 s) : 0, 1159249
Total [baseline] (9.258 s) : 0, 9258365
Agent [candidate] (1.161 s) : 0, 1160975
Total [candidate] (9.267 s) : 0, 9267147
section iast_TELEMETRY_OFF
Agent [baseline] (1.155 s) : 0, 1154957
Total [baseline] (9.237 s) : 0, 9237253
Agent [candidate] (1.155 s) : 0, 1154951
Total [candidate] (9.244 s) : 0, 9243632

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.159 s	109.97 ms (10.5%)
Agent	iast_TELEMETRY_OFF	1.155 s	105.678 ms (10.1%)
Total	tracing	8.702 s	-
Total	iast	9.258 s	556.523 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.237 s	535.412 ms (6.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.161 s	109.841 ms (10.4%)
Agent	iast_TELEMETRY_OFF	1.155 s	103.817 ms (9.9%)
Total	tracing	8.686 s	-
Total	iast	9.267 s	581.256 ms (6.7%)
Total	iast_TELEMETRY_OFF	9.244 s	557.742 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (649.773 ms) : 0, 649773
BytebuddyAgent [candidate] (650.923 ms) : 0, 650923
GlobalTracer [baseline] (308.363 ms) : 0, 308363
GlobalTracer [candidate] (308.844 ms) : 0, 308844
AppSec [baseline] (48.633 ms) : 0, 48633
AppSec [candidate] (49.041 ms) : 0, 49041
Remote Config [baseline] (682.744 µs) : 0, 683
Remote Config [candidate] (681.657 µs) : 0, 682
Telemetry [baseline] (7.277 ms) : 0, 7277
Telemetry [candidate] (7.195 ms) : 0, 7195
section iast
BytebuddyAgent [baseline] (765.993 ms) : 0, 765993
BytebuddyAgent [candidate] (766.306 ms) : 0, 766306
GlobalTracer [baseline] (284.272 ms) : 0, 284272
GlobalTracer [candidate] (285.409 ms) : 0, 285409
AppSec [baseline] (46.304 ms) : 0, 46304
AppSec [candidate] (46.678 ms) : 0, 46678
IAST [baseline] (21.092 ms) : 0, 21092
IAST [candidate] (20.959 ms) : 0, 20959
Remote Config [baseline] (614.184 µs) : 0, 614
Remote Config [candidate] (629.081 µs) : 0, 629
Telemetry [baseline] (6.619 ms) : 0, 6619
Telemetry [candidate] (6.625 ms) : 0, 6625
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (760.421 ms) : 0, 760421
BytebuddyAgent [candidate] (759.709 ms) : 0, 759709
GlobalTracer [baseline] (285.617 ms) : 0, 285617
GlobalTracer [candidate] (286.125 ms) : 0, 286125
AppSec [baseline] (47.097 ms) : 0, 47097
AppSec [candidate] (46.355 ms) : 0, 46355
IAST [baseline] (18.181 ms) : 0, 18181
IAST [candidate] (18.34 ms) : 0, 18340
Remote Config [baseline] (636.145 µs) : 0, 636
Remote Config [candidate] (598.017 µs) : 0, 598
Telemetry [baseline] (8.619 ms) : 0, 8619
Telemetry [candidate] (9.516 ms) : 0, 9516

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1045370
Total [baseline] (9.335 s) : 0, 9335290
Agent [candidate] (1.056 s) : 0, 1055644
Total [candidate] (9.359 s) : 0, 9359067
section appsec
Agent [baseline] (1.137 s) : 0, 1137144
Total [baseline] (9.439 s) : 0, 9438627
Agent [candidate] (1.132 s) : 0, 1132389
Total [candidate] (9.395 s) : 0, 9394713
section iast
Agent [baseline] (1.167 s) : 0, 1166794
Total [baseline] (9.6 s) : 0, 9600075
Agent [candidate] (1.163 s) : 0, 1163199
Total [candidate] (9.584 s) : 0, 9583876
section profiling
Agent [baseline] (1.232 s) : 0, 1232228
Total [baseline] (9.623 s) : 0, 9623138
Agent [candidate] (1.238 s) : 0, 1237644
Total [candidate] (9.636 s) : 0, 9635753

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	appsec	1.137 s	91.774 ms (8.8%)
Agent	iast	1.167 s	121.424 ms (11.6%)
Agent	profiling	1.232 s	186.858 ms (17.9%)
Total	tracing	9.335 s	-
Total	appsec	9.439 s	103.337 ms (1.1%)
Total	iast	9.6 s	264.785 ms (2.8%)
Total	profiling	9.623 s	287.848 ms (3.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.132 s	76.744 ms (7.3%)
Agent	iast	1.163 s	107.555 ms (10.2%)
Agent	profiling	1.238 s	181.999 ms (17.2%)
Total	tracing	9.359 s	-
Total	appsec	9.395 s	35.646 ms (0.4%)
Total	iast	9.584 s	224.808 ms (2.4%)
Total	profiling	9.636 s	276.685 ms (3.0%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (648.618 ms) : 0, 648618
BytebuddyAgent [candidate] (654.481 ms) : 0, 654481
GlobalTracer [baseline] (305.826 ms) : 0, 305826
GlobalTracer [candidate] (309.442 ms) : 0, 309442
AppSec [baseline] (48.609 ms) : 0, 48609
AppSec [candidate] (49.1 ms) : 0, 49100
Remote Config [baseline] (670.655 µs) : 0, 671
Remote Config [candidate] (690.315 µs) : 0, 690
Telemetry [baseline] (7.104 ms) : 0, 7104
Telemetry [candidate] (7.193 ms) : 0, 7193
section appsec
BytebuddyAgent [baseline] (649.428 ms) : 0, 649428
BytebuddyAgent [candidate] (646.567 ms) : 0, 646567
GlobalTracer [baseline] (308.152 ms) : 0, 308152
GlobalTracer [candidate] (307.161 ms) : 0, 307161
AppSec [baseline] (137.396 ms) : 0, 137396
AppSec [candidate] (136.303 ms) : 0, 136303
Remote Config [baseline] (644.14 µs) : 0, 644
Remote Config [candidate] (640.549 µs) : 0, 641
Telemetry [baseline] (6.859 ms) : 0, 6859
Telemetry [candidate] (7.415 ms) : 0, 7415
section iast
BytebuddyAgent [baseline] (770.456 ms) : 0, 770456
BytebuddyAgent [candidate] (768.967 ms) : 0, 768967
GlobalTracer [baseline] (286.636 ms) : 0, 286636
GlobalTracer [candidate] (286.095 ms) : 0, 286095
AppSec [baseline] (46.209 ms) : 0, 46209
AppSec [candidate] (46.325 ms) : 0, 46325
Remote Config [baseline] (603.211 µs) : 0, 603
Remote Config [candidate] (595.925 µs) : 0, 596
Telemetry [baseline] (6.506 ms) : 0, 6506
Telemetry [candidate] (6.588 ms) : 0, 6588
IAST [baseline] (21.764 ms) : 0, 21764
IAST [candidate] (20.214 ms) : 0, 20214
section profiling
BytebuddyAgent [baseline] (656.193 ms) : 0, 656193
BytebuddyAgent [candidate] (661.317 ms) : 0, 661317
GlobalTracer [baseline] (377.178 ms) : 0, 377178
GlobalTracer [candidate] (376.894 ms) : 0, 376894
AppSec [baseline] (48.317 ms) : 0, 48317
AppSec [candidate] (48.442 ms) : 0, 48442
Remote Config [baseline] (658.496 µs) : 0, 658
Remote Config [candidate] (670.811 µs) : 0, 671
Telemetry [baseline] (7.305 ms) : 0, 7305
Telemetry [candidate] (7.366 ms) : 0, 7366
ProfilingAgent [baseline] (88.534 ms) : 0, 88534
ProfilingAgent [candidate] (88.237 ms) : 0, 88237
Profiling [baseline] (88.559 ms) : 0, 88559
Profiling [candidate] (88.264 ms) : 0, 88264

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-05T17:33:46	2023-12-05T17:50:19
git_branch	master	nikita-tkachenko/fix-junit-38-test-cases
git_commit_date	1701791196	1701796195
git_commit_sha	`d3754fe`	`5b2777a`
release_version	1.26.0-SNAPSHOT~d3754fe8b0	1.26.0-SNAPSHOT~5b2777a951
start_time	2023-12-05T17:33:33	2023-12-05T17:50:06

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701798872	1701798872
ci_job_id	382350505	382350505
ci_pipeline_id	24430604	24430604
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.552 µs) : 347, 386
.   : milestone, 367,
iast (466.816 µs) : 447, 487
.   : milestone, 467,
iast_FULL (537.436 µs) : 517, 558
.   : milestone, 537,
iast_INACTIVE (439.99 µs) : 419, 461
.   : milestone, 440,
iast_TELEMETRY_OFF (476.887 µs) : 456, 497
.   : milestone, 477,
tracing (435.582 µs) : 415, 456
.   : milestone, 436,
section candidate
no_agent (365.362 µs) : 345, 386
.   : milestone, 365,
iast (473.28 µs) : 453, 494
.   : milestone, 473,
iast_FULL (530.423 µs) : 510, 551
.   : milestone, 530,
iast_INACTIVE (454.117 µs) : 433, 475
.   : milestone, 454,
iast_TELEMETRY_OFF (461.311 µs) : 441, 482
.   : milestone, 461,
tracing (434.748 µs) : 414, 455
.   : milestone, 435,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.552 µs [346.661 µs, 386.442 µs]	-
iast	466.816 µs [446.69 µs, 486.942 µs]	100.264 µs (27.4%)
iast_FULL	537.436 µs [517.132 µs, 557.74 µs]	170.884 µs (46.6%)
iast_INACTIVE	439.99 µs [419.337 µs, 460.642 µs]	73.438 µs (20.0%)
iast_TELEMETRY_OFF	476.887 µs [456.304 µs, 497.471 µs]	110.336 µs (30.1%)
tracing	435.582 µs [414.85 µs, 456.313 µs]	69.03 µs (18.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.362 µs [345.121 µs, 385.603 µs]	-
iast	473.28 µs [452.989 µs, 493.571 µs]	107.918 µs (29.5%)
iast_FULL	530.423 µs [510.022 µs, 550.824 µs]	165.061 µs (45.2%)
iast_INACTIVE	454.117 µs [432.975 µs, 475.258 µs]	88.755 µs (24.3%)
iast_TELEMETRY_OFF	461.311 µs [440.882 µs, 481.739 µs]	95.949 µs (26.3%)
tracing	434.748 µs [414.297 µs, 455.199 µs]	69.386 µs (19.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~5b2777a951, baseline=1.26.0-SNAPSHOT~d3754fe8b0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.349 ms) : 1330, 1368
.   : milestone, 1349,
appsec (1.754 ms) : 1729, 1779
.   : milestone, 1754,
iast (1.525 ms) : 1500, 1549
.   : milestone, 1525,
profiling (1.503 ms) : 1478, 1528
.   : milestone, 1503,
tracing (1.501 ms) : 1476, 1526
.   : milestone, 1501,
section candidate
no_agent (1.346 ms) : 1327, 1366
.   : milestone, 1346,
appsec (1.741 ms) : 1715, 1766
.   : milestone, 1741,
iast (1.512 ms) : 1488, 1537
.   : milestone, 1512,
profiling (1.518 ms) : 1493, 1543
.   : milestone, 1518,
tracing (1.503 ms) : 1478, 1527
.   : milestone, 1503,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.349 ms [1.33 ms, 1.368 ms]	-
appsec	1.754 ms [1.729 ms, 1.779 ms]	404.365 µs (30.0%)
iast	1.525 ms [1.5 ms, 1.549 ms]	175.492 µs (13.0%)
profiling	1.503 ms [1.478 ms, 1.528 ms]	153.764 µs (11.4%)
tracing	1.501 ms [1.476 ms, 1.526 ms]	151.676 µs (11.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.346 ms [1.327 ms, 1.366 ms]	-
appsec	1.741 ms [1.715 ms, 1.766 ms]	394.095 µs (29.3%)
iast	1.512 ms [1.488 ms, 1.537 ms]	165.749 µs (12.3%)
profiling	1.518 ms [1.493 ms, 1.543 ms]	171.58 µs (12.7%)
tracing	1.503 ms [1.478 ms, 1.527 ms]	156.244 µs (11.6%)

Added test suite level visibility for JUnit 3.8 test cases

5b2777a

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 5, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 5, 2023 18:08

nikita-tkachenko-datadog requested review from a team as code owners December 5, 2023 18:08

nikita-tkachenko-datadog requested review from ericlaz, mcculls and ygree December 5, 2023 18:08

romainkomorndatadog approved these changes Dec 7, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 3fe1b2d into master Dec 7, 2023
74 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/fix-junit-38-test-cases branch December 7, 2023 18:06

github-actions bot added this to the 1.26.0 milestone Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added test suite level visibility for JUnit 3.8 test cases #6320

Added test suite level visibility for JUnit 3.8 test cases #6320

nikita-tkachenko-datadog commented Dec 5, 2023 •

edited by jira bot

Loading

pr-commenter bot commented Dec 5, 2023

Added test suite level visibility for JUnit 3.8 test cases #6320

Added test suite level visibility for JUnit 3.8 test cases #6320

Conversation

nikita-tkachenko-datadog commented Dec 5, 2023 • edited by jira bot Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Dec 5, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

nikita-tkachenko-datadog commented Dec 5, 2023 •

edited by jira bot

Loading