Optimise per-test code coverage for multi-threaded tests #6412

nikita-tkachenko-datadog · 2023-12-21T18:45:30Z

What Does This Do

Optimises code coverage for multi-threaded tests.

Motivation

Reducing code coverage overhead.

Additional Notes

Tests that are multi-threaded are different from tests that are executed concurrently: the latter imply running multiple tests in parallel in different threads. A multi-threaded test is a single test that runs in multiple threads (e.g. a test that creates a thread pool or uses a fork-join pool).

Previous version of the code used a single ConcurrentHashMap instance per test to store covered classes, but given how often coverage probes are activated, some tests showed heavy contention and a lot of overhead.
The new version maintains a separate map of covered classes for each execution thread of a test, thus reducing contention.

com.dd.logs.shardingbalancer.ShardingWFZBalancerTest#it_should_be_better test in logs-backend was used to evaluate performance. The test took 30s with the old version of the code, and 10s with the new one.

Jira ticket: CIVIS-8426

pr-commenter · 2023-12-21T19:13:01Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/optimize-coverage-for-multi-threaded-tests
git_commit_date	1703179736	1703183276
git_commit_sha	`ee75e9f`	`a451219`
release_version	1.27.0-SNAPSHOT~ee75e9f5da	1.27.0-SNAPSHOT~a45121976e

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1703185940	1703185940
ci_job_id	395423621	395423621
ci_pipeline_id	25528292	25528292
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 6 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1049839
Total [baseline] (8.7 s) : 0, 8700029
Agent [candidate] (1.058 s) : 0, 1057654
Total [candidate] (8.769 s) : 0, 8769164
section iast
Agent [baseline] (1.17 s) : 0, 1170441
Total [baseline] (9.262 s) : 0, 9262156
Agent [candidate] (1.18 s) : 0, 1180090
Total [candidate] (9.303 s) : 0, 9303473
section iast_TELEMETRY_OFF
Agent [baseline] (1.17 s) : 0, 1170251
Total [baseline] (9.257 s) : 0, 9256985
Agent [candidate] (1.187 s) : 0, 1187497
Total [candidate] (9.334 s) : 0, 9334346

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.05 s	-
Agent	iast	1.17 s	120.601 ms (11.5%)
Agent	iast_TELEMETRY_OFF	1.17 s	120.412 ms (11.5%)
Total	tracing	8.7 s	-
Total	iast	9.262 s	562.128 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.257 s	556.957 ms (6.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.18 s	122.436 ms (11.6%)
Agent	iast_TELEMETRY_OFF	1.187 s	129.843 ms (12.3%)
Total	tracing	8.769 s	-
Total	iast	9.303 s	534.308 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.334 s	565.182 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.011 ms) : 0, 650011
BytebuddyAgent [candidate] (655.65 ms) : 0, 655650
GlobalTracer [baseline] (306.771 ms) : 0, 306771
GlobalTracer [candidate] (308.743 ms) : 0, 308743
AppSec [baseline] (50.923 ms) : 0, 50923
AppSec [candidate] (50.809 ms) : 0, 50809
Remote Config [baseline] (669.062 µs) : 0, 669
Remote Config [candidate] (675.909 µs) : 0, 676
Telemetry [baseline] (7.196 ms) : 0, 7196
Telemetry [candidate] (7.266 ms) : 0, 7266
section iast
BytebuddyAgent [baseline] (771.614 ms) : 0, 771614
BytebuddyAgent [candidate] (778.023 ms) : 0, 778023
GlobalTracer [baseline] (285.419 ms) : 0, 285419
GlobalTracer [candidate] (287.43 ms) : 0, 287430
AppSec [baseline] (54.644 ms) : 0, 54644
AppSec [candidate] (53.804 ms) : 0, 53804
IAST [baseline] (17.377 ms) : 0, 17377
IAST [candidate] (19.217 ms) : 0, 19217
Remote Config [baseline] (599.757 µs) : 0, 600
Remote Config [candidate] (574.598 µs) : 0, 575
Telemetry [baseline] (6.407 ms) : 0, 6407
Telemetry [candidate] (6.457 ms) : 0, 6457
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (769.291 ms) : 0, 769291
BytebuddyAgent [candidate] (780.582 ms) : 0, 780582
GlobalTracer [baseline] (287.5 ms) : 0, 287500
GlobalTracer [candidate] (291.207 ms) : 0, 291207
AppSec [baseline] (49.516 ms) : 0, 49516
AppSec [candidate] (49.939 ms) : 0, 49939
IAST [baseline] (22.265 ms) : 0, 22265
IAST [candidate] (23.717 ms) : 0, 23717
Remote Config [baseline] (590.815 µs) : 0, 591
Remote Config [candidate] (585.946 µs) : 0, 586
Telemetry [baseline] (6.473 ms) : 0, 6473
Telemetry [candidate] (6.485 ms) : 0, 6485

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1055275
Total [baseline] (9.373 s) : 0, 9372735
Agent [candidate] (1.053 s) : 0, 1052645
Total [candidate] (9.354 s) : 0, 9354283
section appsec
Agent [baseline] (1.145 s) : 0, 1145116
Total [baseline] (9.395 s) : 0, 9394704
Agent [candidate] (1.151 s) : 0, 1150723
Total [candidate] (9.448 s) : 0, 9447907
section iast
Agent [baseline] (1.174 s) : 0, 1174013
Total [baseline] (9.655 s) : 0, 9654987
Agent [candidate] (1.178 s) : 0, 1177794
Total [candidate] (9.698 s) : 0, 9697990
section profiling
Agent [baseline] (1.254 s) : 0, 1253733
Total [baseline] (9.632 s) : 0, 9631584
Agent [candidate] (1.262 s) : 0, 1262485
Total [candidate] (9.614 s) : 0, 9613534

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.145 s	89.841 ms (8.5%)
Agent	iast	1.174 s	118.739 ms (11.3%)
Agent	profiling	1.254 s	198.458 ms (18.8%)
Total	tracing	9.373 s	-
Total	appsec	9.395 s	21.969 ms (0.2%)
Total	iast	9.655 s	282.252 ms (3.0%)
Total	profiling	9.632 s	258.85 ms (2.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.151 s	98.077 ms (9.3%)
Agent	iast	1.178 s	125.148 ms (11.9%)
Agent	profiling	1.262 s	209.84 ms (19.9%)
Total	tracing	9.354 s	-
Total	appsec	9.448 s	93.624 ms (1.0%)
Total	iast	9.698 s	343.707 ms (3.7%)
Total	profiling	9.614 s	259.251 ms (2.8%)

gantt
    title petclinic - break down per module: candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (654.411 ms) : 0, 654411
BytebuddyAgent [candidate] (651.405 ms) : 0, 651405
GlobalTracer [baseline] (307.934 ms) : 0, 307934
GlobalTracer [candidate] (308.362 ms) : 0, 308362
AppSec [baseline] (50.526 ms) : 0, 50526
AppSec [candidate] (50.707 ms) : 0, 50707
Remote Config [baseline] (659.962 µs) : 0, 660
Remote Config [candidate] (671.076 µs) : 0, 671
Telemetry [baseline] (7.178 ms) : 0, 7178
Telemetry [candidate] (7.208 ms) : 0, 7208
section appsec
BytebuddyAgent [baseline] (649.23 ms) : 0, 649230
BytebuddyAgent [candidate] (651.944 ms) : 0, 651944
GlobalTracer [baseline] (305.853 ms) : 0, 305853
GlobalTracer [candidate] (308.141 ms) : 0, 308141
AppSec [baseline] (148.397 ms) : 0, 148397
AppSec [candidate] (148.698 ms) : 0, 148698
Remote Config [baseline] (642.31 µs) : 0, 642
Remote Config [candidate] (644.583 µs) : 0, 645
Telemetry [baseline] (6.887 ms) : 0, 6887
Telemetry [candidate] (6.928 ms) : 0, 6928
section iast
BytebuddyAgent [baseline] (774.316 ms) : 0, 774316
BytebuddyAgent [candidate] (775.437 ms) : 0, 775437
GlobalTracer [baseline] (284.865 ms) : 0, 284865
GlobalTracer [candidate] (287.824 ms) : 0, 287824
AppSec [baseline] (52.743 ms) : 0, 52743
AppSec [candidate] (53.722 ms) : 0, 53722
IAST [baseline] (20.389 ms) : 0, 20389
IAST [candidate] (19.35 ms) : 0, 19350
Remote Config [baseline] (593.956 µs) : 0, 594
Remote Config [candidate] (583.121 µs) : 0, 583
Telemetry [baseline] (6.506 ms) : 0, 6506
Telemetry [candidate] (6.515 ms) : 0, 6515
section profiling
BytebuddyAgent [baseline] (664.366 ms) : 0, 664366
BytebuddyAgent [candidate] (670.047 ms) : 0, 670047
GlobalTracer [baseline] (379.498 ms) : 0, 379498
GlobalTracer [candidate] (381.639 ms) : 0, 381639
AppSec [baseline] (51.338 ms) : 0, 51338
AppSec [candidate] (51.666 ms) : 0, 51666
Remote Config [baseline] (1.005 ms) : 0, 1005
Remote Config [candidate] (1.012 ms) : 0, 1012
Telemetry [baseline] (7.248 ms) : 0, 7248
Telemetry [candidate] (7.359 ms) : 0, 7359
ProfilingAgent [baseline] (95.503 ms) : 0, 95503
ProfilingAgent [candidate] (95.743 ms) : 0, 95743
Profiling [baseline] (95.529 ms) : 0, 95529
Profiling [candidate] (95.769 ms) : 0, 95769

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-21T18:51:31	2023-12-21T19:08:07
git_branch	master	nikita-tkachenko/optimize-coverage-for-multi-threaded-tests
git_commit_date	1703179736	1703183276
git_commit_sha	`ee75e9f`	`a451219`
release_version	1.27.0-SNAPSHOT~ee75e9f5da	1.27.0-SNAPSHOT~a45121976e
start_time	2023-12-21T18:51:18	2023-12-21T19:07:54

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1703185940	1703185940
ci_job_id	395423621	395423621
ci_pipeline_id	25528292	25528292
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.011 µs) : 348, 392
.   : milestone, 370,
iast (478.408 µs) : 458, 499
.   : milestone, 478,
iast_FULL (537.642 µs) : 517, 558
.   : milestone, 538,
iast_INACTIVE (453.121 µs) : 432, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (476.718 µs) : 455, 498
.   : milestone, 477,
tracing (440.491 µs) : 420, 461
.   : milestone, 440,
section candidate
no_agent (369.972 µs) : 350, 390
.   : milestone, 370,
iast (471.007 µs) : 451, 491
.   : milestone, 471,
iast_FULL (536.839 µs) : 516, 557
.   : milestone, 537,
iast_INACTIVE (449.232 µs) : 428, 471
.   : milestone, 449,
iast_TELEMETRY_OFF (465.905 µs) : 445, 486
.   : milestone, 466,
tracing (442.689 µs) : 422, 464
.   : milestone, 443,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.011 µs [347.829 µs, 392.193 µs]	-
iast	478.408 µs [457.676 µs, 499.141 µs]	108.398 µs (29.3%)
iast_FULL	537.642 µs [517.378 µs, 557.907 µs]	167.632 µs (45.3%)
iast_INACTIVE	453.121 µs [432.148 µs, 474.094 µs]	83.11 µs (22.5%)
iast_TELEMETRY_OFF	476.718 µs [455.041 µs, 498.395 µs]	106.707 µs (28.8%)
tracing	440.491 µs [419.797 µs, 461.184 µs]	70.48 µs (19.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	369.972 µs [350.329 µs, 389.615 µs]	-
iast	471.007 µs [450.702 µs, 491.311 µs]	101.035 µs (27.3%)
iast_FULL	536.839 µs [516.466 µs, 557.211 µs]	166.867 µs (45.1%)
iast_INACTIVE	449.232 µs [427.56 µs, 470.904 µs]	79.26 µs (21.4%)
iast_TELEMETRY_OFF	465.905 µs [445.375 µs, 486.435 µs]	95.933 µs (25.9%)
tracing	442.689 µs [421.827 µs, 463.55 µs]	72.717 µs (19.7%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.27.0-SNAPSHOT~a45121976e, baseline=1.27.0-SNAPSHOT~ee75e9f5da
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.349 ms) : 1330, 1368
.   : milestone, 1349,
appsec (1.785 ms) : 1760, 1810
.   : milestone, 1785,
iast (1.526 ms) : 1502, 1549
.   : milestone, 1526,
profiling (1.523 ms) : 1497, 1548
.   : milestone, 1523,
tracing (1.464 ms) : 1440, 1489
.   : milestone, 1464,
section candidate
no_agent (1.368 ms) : 1349, 1387
.   : milestone, 1368,
appsec (1.762 ms) : 1737, 1786
.   : milestone, 1762,
iast (1.527 ms) : 1503, 1551
.   : milestone, 1527,
profiling (1.524 ms) : 1499, 1548
.   : milestone, 1524,
tracing (1.481 ms) : 1456, 1505
.   : milestone, 1481,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.349 ms [1.33 ms, 1.368 ms]	-
appsec	1.785 ms [1.76 ms, 1.81 ms]	436.141 µs (32.3%)
iast	1.526 ms [1.502 ms, 1.549 ms]	177.129 µs (13.1%)
profiling	1.523 ms [1.497 ms, 1.548 ms]	174.022 µs (12.9%)
tracing	1.464 ms [1.44 ms, 1.489 ms]	115.764 µs (8.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.368 ms [1.349 ms, 1.387 ms]	-
appsec	1.762 ms [1.737 ms, 1.786 ms]	393.669 µs (28.8%)
iast	1.527 ms [1.503 ms, 1.551 ms]	159.377 µs (11.7%)
profiling	1.524 ms [1.499 ms, 1.548 ms]	155.855 µs (11.4%)
tracing	1.481 ms [1.456 ms, 1.505 ms]	112.898 µs (8.3%)

Optimize per-test code coverage for multi-threaded tests

a451219

nikita-tkachenko-datadog added tag: performance Performance related changes comp: ci visibility Continuous Integration Visibility labels Dec 21, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 21, 2023 19:46

nikita-tkachenko-datadog requested a review from a team as a code owner December 21, 2023 19:46

nikita-tkachenko-datadog requested a review from ericlaz December 21, 2023 19:46

romainkomorndatadog approved these changes Dec 22, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 1b27e34 into master Dec 22, 2023
75 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/optimize-coverage-for-multi-threaded-tests branch December 22, 2023 11:50

github-actions bot added this to the 1.27.0 milestone Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimise per-test code coverage for multi-threaded tests #6412

Optimise per-test code coverage for multi-threaded tests #6412

nikita-tkachenko-datadog commented Dec 21, 2023 •

edited

Loading

pr-commenter bot commented Dec 21, 2023

Optimise per-test code coverage for multi-threaded tests #6412

Optimise per-test code coverage for multi-threaded tests #6412

Conversation

nikita-tkachenko-datadog commented Dec 21, 2023 • edited Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Dec 21, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

nikita-tkachenko-datadog commented Dec 21, 2023 •

edited

Loading