Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement custom code coverage mechanism for Intelligent Test Runner #6390

Merged
merged 3 commits into from
Dec 19, 2023
Merged

Implement custom code coverage mechanism for Intelligent Test Runner #6390

merged 3 commits into from
Dec 19, 2023

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Dec 18, 2023 •
edited
Loading

What Does This Do

Adds custom per-test code coverage mechanism for Intelligent Test Runner.
The new mechanism is not based on Jacoco, and only supports file-level coverage (specific lines within files are not calculated).

Motivation

Reduce code coverage overhead.
Simplify configuration by getting rid of Jacoco dependency.

Jira ticket: CIVIS-8376

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 18, 2023
@pr-commenter
Copy link

pr-commenter bot commented Dec 18, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/custom-itr-code-coverage
git_commit_date 1702921792 1702929519
git_commit_sha df9ada3 fefee03
release_version 1.26.0-SNAPSHOT~df9ada3ba1 1.26.0-SNAPSHOT~fefee03f72
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702932290 1702932290
ci_job_id 392403365 392403365
ci_pipeline_id 25286834 25286834
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044815
Total [baseline] (8.741 s) : 0, 8741342
Agent [candidate] (1.053 s) : 0, 1053064
Total [candidate] (8.73 s) : 0, 8730351
section iast
Agent [baseline] (1.164 s) : 0, 1163674
Total [baseline] (9.262 s) : 0, 9261828
Agent [candidate] (1.174 s) : 0, 1173872
Total [candidate] (9.315 s) : 0, 9315369
section iast_TELEMETRY_OFF
Agent [baseline] (1.158 s) : 0, 1157731
Total [baseline] (9.229 s) : 0, 9229406
Agent [candidate] (1.16 s) : 0, 1160426
Total [candidate] (9.303 s) : 0, 9302731
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.045 s -
Agent iast 1.164 s 118.859 ms (11.4%)
Agent iast_TELEMETRY_OFF 1.158 s 112.916 ms (10.8%)
Total tracing 8.741 s -
Total iast 9.262 s 520.486 ms (6.0%)
Total iast_TELEMETRY_OFF 9.229 s 488.064 ms (5.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent iast 1.174 s 120.808 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.16 s 107.362 ms (10.2%)
Total tracing 8.73 s -
Total iast 9.315 s 585.018 ms (6.7%)
Total iast_TELEMETRY_OFF 9.303 s 572.38 ms (6.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.391 ms) : 0, 646391
BytebuddyAgent [candidate] (651.255 ms) : 0, 651255
GlobalTracer [baseline] (305.492 ms) : 0, 305492
GlobalTracer [candidate] (307.894 ms) : 0, 307894
AppSec [baseline] (50.821 ms) : 0, 50821
AppSec [candidate] (51.684 ms) : 0, 51684
Remote Config [baseline] (672.218 µs) : 0, 672
Remote Config [candidate] (677.645 µs) : 0, 678
Telemetry [baseline] (7.255 ms) : 0, 7255
Telemetry [candidate] (7.211 ms) : 0, 7211
section iast
BytebuddyAgent [baseline] (767.854 ms) : 0, 767854
BytebuddyAgent [candidate] (774.057 ms) : 0, 774057
GlobalTracer [baseline] (284.528 ms) : 0, 284528
GlobalTracer [candidate] (286.857 ms) : 0, 286857
AppSec [baseline] (48.801 ms) : 0, 48801
AppSec [candidate] (49.337 ms) : 0, 49337
IAST [baseline] (20.331 ms) : 0, 20331
IAST [candidate] (21.045 ms) : 0, 21045
Remote Config [baseline] (617.036 µs) : 0, 617
Remote Config [candidate] (612.089 µs) : 0, 612
Telemetry [baseline] (7.228 ms) : 0, 7228
Telemetry [candidate] (7.238 ms) : 0, 7238
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (761.247 ms) : 0, 761247
BytebuddyAgent [candidate] (761.659 ms) : 0, 761659
GlobalTracer [baseline] (284.958 ms) : 0, 284958
GlobalTracer [candidate] (285.897 ms) : 0, 285897
AppSec [baseline] (49.356 ms) : 0, 49356
AppSec [candidate] (49.638 ms) : 0, 49638
IAST [baseline] (20.742 ms) : 0, 20742
IAST [candidate] (20.203 ms) : 0, 20203
Remote Config [baseline] (610.996 µs) : 0, 611
Remote Config [candidate] (615.083 µs) : 0, 615
Telemetry [baseline] (6.649 ms) : 0, 6649
Telemetry [candidate] (8.146 ms) : 0, 8146
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1051876
Total [baseline] (9.351 s) : 0, 9350941
Agent [candidate] (1.068 s) : 0, 1068446
Total [candidate] (9.465 s) : 0, 9464624
section appsec
Agent [baseline] (1.139 s) : 0, 1138884
Total [baseline] (9.446 s) : 0, 9446307
Agent [candidate] (1.147 s) : 0, 1146696
Total [candidate] (9.517 s) : 0, 9517304
section iast
Agent [baseline] (1.169 s) : 0, 1168619
Total [baseline] (9.631 s) : 0, 9631050
Agent [candidate] (1.168 s) : 0, 1168129
Total [candidate] (9.665 s) : 0, 9664520
section profiling
Agent [baseline] (1.25 s) : 0, 1249922
Total [baseline] (9.622 s) : 0, 9622079
Agent [candidate] (1.241 s) : 0, 1241173
Total [candidate] (9.619 s) : 0, 9618865
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.052 s -
Agent appsec 1.139 s 87.009 ms (8.3%)
Agent iast 1.169 s 116.743 ms (11.1%)
Agent profiling 1.25 s 198.047 ms (18.8%)
Total tracing 9.351 s -
Total appsec 9.446 s 95.366 ms (1.0%)
Total iast 9.631 s 280.11 ms (3.0%)
Total profiling 9.622 s 271.138 ms (2.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.068 s -
Agent appsec 1.147 s 78.251 ms (7.3%)
Agent iast 1.168 s 99.683 ms (9.3%)
Agent profiling 1.241 s 172.727 ms (16.2%)
Total tracing 9.465 s -
Total appsec 9.517 s 52.68 ms (0.6%)
Total iast 9.665 s 199.896 ms (2.1%)
Total profiling 9.619 s 154.241 ms (1.6%) 
gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (651.521 ms) : 0, 651521
BytebuddyAgent [candidate] (660.843 ms) : 0, 660843
GlobalTracer [baseline] (306.919 ms) : 0, 306919
GlobalTracer [candidate] (312.687 ms) : 0, 312687
AppSec [baseline] (51.062 ms) : 0, 51062
AppSec [candidate] (51.99 ms) : 0, 51990
Remote Config [baseline] (684.517 µs) : 0, 685
Remote Config [candidate] (688.906 µs) : 0, 689
Telemetry [baseline] (7.292 ms) : 0, 7292
Telemetry [candidate] (7.305 ms) : 0, 7305
section appsec
BytebuddyAgent [baseline] (645.12 ms) : 0, 645120
BytebuddyAgent [candidate] (648.858 ms) : 0, 648858
GlobalTracer [baseline] (303.905 ms) : 0, 303905
GlobalTracer [candidate] (306.368 ms) : 0, 306368
AppSec [baseline] (148.491 ms) : 0, 148491
AppSec [candidate] (148.971 ms) : 0, 148971
Remote Config [baseline] (637.366 µs) : 0, 637
Remote Config [candidate] (638.464 µs) : 0, 638
Telemetry [baseline] (6.822 ms) : 0, 6822
Telemetry [candidate] (7.423 ms) : 0, 7423
section iast
BytebuddyAgent [baseline] (770.298 ms) : 0, 770298
BytebuddyAgent [candidate] (769.887 ms) : 0, 769887
GlobalTracer [baseline] (285.726 ms) : 0, 285726
GlobalTracer [candidate] (285.554 ms) : 0, 285554
AppSec [baseline] (48.878 ms) : 0, 48878
AppSec [candidate] (49.216 ms) : 0, 49216
IAST [baseline] (21.403 ms) : 0, 21403
IAST [candidate] (22.067 ms) : 0, 22067
Remote Config [baseline] (610.19 µs) : 0, 610
Remote Config [candidate] (619.171 µs) : 0, 619
Telemetry [baseline] (7.241 ms) : 0, 7241
Telemetry [candidate] (6.523 ms) : 0, 6523
section profiling
BytebuddyAgent [baseline] (662.009 ms) : 0, 662009
BytebuddyAgent [candidate] (657.292 ms) : 0, 657292
GlobalTracer [baseline] (378.744 ms) : 0, 378744
GlobalTracer [candidate] (376.157 ms) : 0, 376157
AppSec [baseline] (51.457 ms) : 0, 51457
AppSec [candidate] (51.332 ms) : 0, 51332
Remote Config [baseline] (677.979 µs) : 0, 678
Remote Config [candidate] (679.592 µs) : 0, 680
Telemetry [baseline] (7.461 ms) : 0, 7461
Telemetry [candidate] (7.423 ms) : 0, 7423
ProfilingAgent [baseline] (94.988 ms) : 0, 94988
ProfilingAgent [candidate] (94.018 ms) : 0, 94018
Profiling [baseline] (95.013 ms) : 0, 95013
Profiling [candidate] (94.044 ms) : 0, 94044
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-12-18T20:23:57 2023-12-18T20:40:36
git_branch master nikita-tkachenko/custom-itr-code-coverage
git_commit_date 1702921792 1702929519
git_commit_sha df9ada3 fefee03
release_version 1.26.0-SNAPSHOT~df9ada3ba1 1.26.0-SNAPSHOT~fefee03f72
start_time 2023-12-18T20:23:43 2023-12-18T20:40:22
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1702932290 1702932290
ci_job_id 392403365 392403365
ci_pipeline_id 25286834 25286834
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1
    dateFormat X
    axisFormat %s
section baseline
no_agent (374.907 µs) : 355, 395
.   : milestone, 375,
iast (485.776 µs) : 465, 507
.   : milestone, 486,
iast_FULL (538.647 µs) : 518, 559
.   : milestone, 539,
iast_INACTIVE (449.109 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (474.514 µs) : 453, 496
.   : milestone, 475,
tracing (443.511 µs) : 423, 464
.   : milestone, 444,
section candidate
no_agent (368.914 µs) : 348, 390
.   : milestone, 369,
iast (478.251 µs) : 458, 498
.   : milestone, 478,
iast_FULL (538.961 µs) : 518, 559
.   : milestone, 539,
iast_INACTIVE (459.89 µs) : 439, 481
.   : milestone, 460,
iast_TELEMETRY_OFF (474.363 µs) : 454, 495
.   : milestone, 474,
tracing (460.616 µs) : 440, 481
.   : milestone, 461,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 374.907 µs [354.705 µs, 395.109 µs] -
iast 485.776 µs [464.647 µs, 506.904 µs] 110.869 µs (29.6%)
iast_FULL 538.647 µs [518.186 µs, 559.107 µs] 163.74 µs (43.7%)
iast_INACTIVE 449.109 µs [428.481 µs, 469.737 µs] 74.202 µs (19.8%)
iast_TELEMETRY_OFF 474.514 µs [453.408 µs, 495.619 µs] 99.607 µs (26.6%)
tracing 443.511 µs [422.678 µs, 464.344 µs] 68.604 µs (18.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.914 µs [348.326 µs, 389.502 µs] -
iast 478.251 µs [458.015 µs, 498.488 µs] 109.337 µs (29.6%)
iast_FULL 538.961 µs [518.486 µs, 559.435 µs] 170.047 µs (46.1%)
iast_INACTIVE 459.89 µs [438.711 µs, 481.069 µs] 90.976 µs (24.7%)
iast_TELEMETRY_OFF 474.363 µs [453.616 µs, 495.11 µs] 105.449 µs (28.6%)
tracing 460.616 µs [439.866 µs, 481.367 µs] 91.702 µs (24.9%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~fefee03f72, baseline=1.26.0-SNAPSHOT~df9ada3ba1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.341 ms) : 1322, 1360
.   : milestone, 1341,
appsec (1.781 ms) : 1756, 1806
.   : milestone, 1781,
iast (1.535 ms) : 1510, 1559
.   : milestone, 1535,
profiling (1.538 ms) : 1513, 1564
.   : milestone, 1538,
tracing (1.479 ms) : 1454, 1504
.   : milestone, 1479,
section candidate
no_agent (1.356 ms) : 1337, 1375
.   : milestone, 1356,
appsec (1.774 ms) : 1748, 1799
.   : milestone, 1774,
iast (1.517 ms) : 1493, 1542
.   : milestone, 1517,
profiling (1.523 ms) : 1498, 1548
.   : milestone, 1523,
tracing (1.489 ms) : 1464, 1514
.   : milestone, 1489,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.341 ms [1.322 ms, 1.36 ms] -
appsec 1.781 ms [1.756 ms, 1.806 ms] 439.772 µs (32.8%)
iast 1.535 ms [1.51 ms, 1.559 ms] 193.466 µs (14.4%)
profiling 1.538 ms [1.513 ms, 1.564 ms] 197.22 µs (14.7%)
tracing 1.479 ms [1.454 ms, 1.504 ms] 138.035 µs (10.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.356 ms [1.337 ms, 1.375 ms] -
appsec 1.774 ms [1.748 ms, 1.799 ms] 417.653 µs (30.8%)
iast 1.517 ms [1.493 ms, 1.542 ms] 161.15 µs (11.9%)
profiling 1.523 ms [1.498 ms, 1.548 ms] 167.072 µs (12.3%)
tracing 1.489 ms [1.464 ms, 1.514 ms] 132.957 µs (9.8%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review December 18, 2023 21:51
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit ad2dbbf into master Dec 19, 2023
73 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/custom-itr-code-coverage branch December 19, 2023 08:52
@github-actions github-actions bot added this to the 1.26.0 milestone Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romainkomorndatadog romainkomorndatadog romainkomorndatadog approved these changes

@tonyredondo tonyredondo Awaiting requested review from tonyredondo tonyredondo is a code owner automatically assigned from DataDog/ci-app-libraries-java

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility
Projects
None yet
Milestone
1.26.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @romainkomorndatadog