Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement telemetry for CI Visibility #6664

Merged
merged 1 commit into from
Feb 13, 2024
Merged

Implement telemetry for CI Visibility #6664

merged 1 commit into from
Feb 13, 2024

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Feb 12, 2024 •
edited by jira bot
Loading

What Does This Do

Implements telemetry mechanism for CI Visibility logic.

Motivation

To have better observability of CI Visibility behaviour.

Additional Notes

This PR only contains the generic logic for submitting and aggregating metrics. Actual metric invocations will be added in subsequent PRs.

Jira ticket: CIVIS-2427

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Feb 12, 2024
@pr-commenter
Copy link

pr-commenter bot commented Feb 12, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/ci-vis-telemetry
git_commit_date 1707732199 1707733875
git_commit_sha d2d8296 8936a3f
release_version 1.31.0-SNAPSHOT~d2d8296831 1.31.0-SNAPSHOT~8936a3f0a8
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1707736701 1707736701
ci_job_id 431443068 431443068
ci_pipeline_id 28109163 28109163
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.31.0-SNAPSHOT~8936a3f0a8, baseline=1.31.0-SNAPSHOT~d2d8296831

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1057546
Total [baseline] (9.335 s) : 0, 9335140
Agent [candidate] (1.062 s) : 0, 1061509
Total [candidate] (9.329 s) : 0, 9328741
section appsec
Agent [baseline] (1.157 s) : 0, 1157276
Total [baseline] (9.464 s) : 0, 9463897
Agent [candidate] (1.166 s) : 0, 1165906
Total [candidate] (9.495 s) : 0, 9495436
section iast
Agent [baseline] (1.182 s) : 0, 1182482
Total [baseline] (9.688 s) : 0, 9687863
Agent [candidate] (1.188 s) : 0, 1188014
Total [candidate] (9.716 s) : 0, 9715861
section profiling
Agent [baseline] (1.282 s) : 0, 1281969
Total [baseline] (9.596 s) : 0, 9596023
Agent [candidate] (1.278 s) : 0, 1277908
Total [candidate] (9.568 s) : 0, 9567586
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent appsec 1.157 s 99.73 ms (9.4%)
Agent iast 1.182 s 124.936 ms (11.8%)
Agent profiling 1.282 s 224.423 ms (21.2%)
Total tracing 9.335 s -
Total appsec 9.464 s 128.757 ms (1.4%)
Total iast 9.688 s 352.723 ms (3.8%)
Total profiling 9.596 s 260.882 ms (2.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.166 s 104.397 ms (9.8%)
Agent iast 1.188 s 126.506 ms (11.9%)
Agent profiling 1.278 s 216.4 ms (20.4%)
Total tracing 9.329 s -
Total appsec 9.495 s 166.696 ms (1.8%)
Total iast 9.716 s 387.12 ms (4.1%)
Total profiling 9.568 s 238.845 ms (2.6%) 
gantt
    title petclinic - break down per module: candidate=1.31.0-SNAPSHOT~8936a3f0a8, baseline=1.31.0-SNAPSHOT~d2d8296831

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (666.656 ms) : 0, 666656
BytebuddyAgent [candidate] (668.65 ms) : 0, 668650
GlobalTracer [baseline] (296.87 ms) : 0, 296870
GlobalTracer [candidate] (298.466 ms) : 0, 298466
AppSec [baseline] (51.449 ms) : 0, 51449
AppSec [candidate] (51.936 ms) : 0, 51936
Remote Config [baseline] (689.962 µs) : 0, 690
Remote Config [candidate] (680.404 µs) : 0, 680
Telemetry [baseline] (7.62 ms) : 0, 7620
Telemetry [candidate] (7.588 ms) : 0, 7588
section appsec
BytebuddyAgent [baseline] (666.835 ms) : 0, 666835
BytebuddyAgent [candidate] (672.231 ms) : 0, 672231
GlobalTracer [baseline] (297.48 ms) : 0, 297480
GlobalTracer [candidate] (299.918 ms) : 0, 299918
AppSec [baseline] (151.267 ms) : 0, 151267
AppSec [candidate] (151.792 ms) : 0, 151792
Remote Config [baseline] (653.162 µs) : 0, 653
Remote Config [candidate] (651.416 µs) : 0, 651
Telemetry [baseline] (6.842 ms) : 0, 6842
Telemetry [candidate] (6.875 ms) : 0, 6875
section iast
BytebuddyAgent [baseline] (777.571 ms) : 0, 777571
BytebuddyAgent [candidate] (781.34 ms) : 0, 781340
GlobalTracer [baseline] (287.948 ms) : 0, 287948
GlobalTracer [candidate] (289.822 ms) : 0, 289822
AppSec [baseline] (52.458 ms) : 0, 52458
AppSec [candidate] (55.269 ms) : 0, 55269
IAST [baseline] (23.028 ms) : 0, 23028
IAST [candidate] (19.951 ms) : 0, 19951
Remote Config [baseline] (619.7 µs) : 0, 620
Remote Config [candidate] (617.667 µs) : 0, 618
Telemetry [baseline] (6.544 ms) : 0, 6544
Telemetry [candidate] (6.644 ms) : 0, 6644
section profiling
BytebuddyAgent [baseline] (668.968 ms) : 0, 668968
BytebuddyAgent [candidate] (665.157 ms) : 0, 665157
GlobalTracer [baseline] (382.016 ms) : 0, 382016
GlobalTracer [candidate] (382.472 ms) : 0, 382472
AppSec [baseline] (52.371 ms) : 0, 52371
AppSec [candidate] (52.056 ms) : 0, 52056
Remote Config [baseline] (662.45 µs) : 0, 662
Remote Config [candidate] (667.676 µs) : 0, 668
Telemetry [baseline] (10.172 ms) : 0, 10172
Telemetry [candidate] (11.476 ms) : 0, 11476
ProfilingAgent [baseline] (112.997 ms) : 0, 112997
ProfilingAgent [candidate] (111.495 ms) : 0, 111495
Profiling [baseline] (113.021 ms) : 0, 113021
Profiling [candidate] (111.519 ms) : 0, 111519
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-02-12T10:54:21 2024-02-12T11:13:18
git_branch master nikita-tkachenko/ci-vis-telemetry
git_commit_date 1707732199 1707733875
git_commit_sha d2d8296 8936a3f
release_version 1.31.0-SNAPSHOT~d2d8296831 1.31.0-SNAPSHOT~8936a3f0a8
start_time 2024-02-12T10:54:07 2024-02-12T11:13:05
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1707736701 1707736701
ci_job_id 431443068 431443068
ci_pipeline_id 28109163 28109163
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 15 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~8936a3f0a8, baseline=1.31.0-SNAPSHOT~d2d8296831
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.365 ms) : 1346, 1384
.   : milestone, 1365,
appsec (1.775 ms) : 1750, 1801
.   : milestone, 1775,
iast (1.536 ms) : 1512, 1561
.   : milestone, 1536,
profiling (1.53 ms) : 1505, 1556
.   : milestone, 1530,
tracing (1.498 ms) : 1473, 1523
.   : milestone, 1498,
section candidate
no_agent (1.355 ms) : 1336, 1374
.   : milestone, 1355,
appsec (1.791 ms) : 1766, 1817
.   : milestone, 1791,
iast (1.531 ms) : 1506, 1555
.   : milestone, 1531,
profiling (1.516 ms) : 1491, 1542
.   : milestone, 1516,
tracing (1.498 ms) : 1473, 1523
.   : milestone, 1498,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.365 ms [1.346 ms, 1.384 ms] -
appsec 1.775 ms [1.75 ms, 1.801 ms] 409.994 µs (30.0%)
iast 1.536 ms [1.512 ms, 1.561 ms] 171.282 µs (12.5%)
profiling 1.53 ms [1.505 ms, 1.556 ms] 165.105 µs (12.1%)
tracing 1.498 ms [1.473 ms, 1.523 ms] 132.738 µs (9.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.355 ms [1.336 ms, 1.374 ms] -
appsec 1.791 ms [1.766 ms, 1.817 ms] 436.224 µs (32.2%)
iast 1.531 ms [1.506 ms, 1.555 ms] 175.789 µs (13.0%)
profiling 1.516 ms [1.491 ms, 1.542 ms] 161.333 µs (11.9%)
tracing 1.498 ms [1.473 ms, 1.523 ms] 142.803 µs (10.5%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~8936a3f0a8, baseline=1.31.0-SNAPSHOT~d2d8296831
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.687 µs) : 352, 393
.   : milestone, 373,
iast (484.853 µs) : 463, 506
.   : milestone, 485,
iast_FULL (549.793 µs) : 529, 571
.   : milestone, 550,
iast_GLOBAL (509.959 µs) : 488, 532
.   : milestone, 510,
iast_HARDCODED_SECRET_DISABLED (477.66 µs) : 457, 499
.   : milestone, 478,
iast_INACTIVE (451.119 µs) : 431, 472
.   : milestone, 451,
iast_TELEMETRY_OFF (478.722 µs) : 457, 500
.   : milestone, 479,
tracing (446.613 µs) : 425, 468
.   : milestone, 447,
section candidate
no_agent (372.146 µs) : 352, 393
.   : milestone, 372,
iast (476.322 µs) : 455, 497
.   : milestone, 476,
iast_FULL (544.595 µs) : 524, 565
.   : milestone, 545,
iast_GLOBAL (507.738 µs) : 486, 530
.   : milestone, 508,
iast_HARDCODED_SECRET_DISABLED (475.337 µs) : 455, 496
.   : milestone, 475,
iast_INACTIVE (453.509 µs) : 432, 475
.   : milestone, 454,
iast_TELEMETRY_OFF (483.717 µs) : 463, 505
.   : milestone, 484,
tracing (447.265 µs) : 426, 469
.   : milestone, 447,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.687 µs [352.278 µs, 393.097 µs] -
iast 484.853 µs [463.48 µs, 506.227 µs] 112.166 µs (30.1%)
iast_FULL 549.793 µs [528.974 µs, 570.611 µs] 177.105 µs (47.5%)
iast_GLOBAL 509.959 µs [488.205 µs, 531.713 µs] 137.272 µs (36.8%)
iast_HARDCODED_SECRET_DISABLED 477.66 µs [456.736 µs, 498.585 µs] 104.973 µs (28.2%)
iast_INACTIVE 451.119 µs [430.549 µs, 471.69 µs] 78.432 µs (21.0%)
iast_TELEMETRY_OFF 478.722 µs [457.228 µs, 500.216 µs] 106.034 µs (28.5%)
tracing 446.613 µs [425.471 µs, 467.755 µs] 73.926 µs (19.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.146 µs [351.572 µs, 392.72 µs] -
iast 476.322 µs [455.404 µs, 497.241 µs] 104.176 µs (28.0%)
iast_FULL 544.595 µs [523.788 µs, 565.401 µs] 172.448 µs (46.3%)
iast_GLOBAL 507.738 µs [485.794 µs, 529.681 µs] 135.592 µs (36.4%)
iast_HARDCODED_SECRET_DISABLED 475.337 µs [454.703 µs, 495.972 µs] 103.191 µs (27.7%)
iast_INACTIVE 453.509 µs [432.236 µs, 474.781 µs] 81.363 µs (21.9%)
iast_TELEMETRY_OFF 483.717 µs [462.619 µs, 504.816 µs] 111.571 µs (30.0%)
tracing 447.265 µs [425.839 µs, 468.691 µs] 75.119 µs (20.2%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review February 12, 2024 11:28
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit c471269 into master Feb 13, 2024
79 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/ci-vis-telemetry branch February 13, 2024 17:32
@github-actions github-actions bot added this to the 1.30.0 milestone Feb 13, 2024
jandro996 pushed a commit that referenced this pull request Feb 16, 2024
@nikita-tkachenko-datadog @jandro996
Implement telemetry for CI Visibility (#6664)
23b997c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonyredondo tonyredondo tonyredondo approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility
Projects
None yet
Milestone
1.30.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@nikita-tkachenko-datadog @tonyredondo