Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for disablement of tracing via remote config #6827

Merged
merged 7 commits into from Mar 27, 2024
Merged

Allow for disablement of tracing via remote config #6827

merged 7 commits into from Mar 27, 2024

Conversation

nayeem-kamal
Copy link
Contributor

@nayeem-kamal nayeem-kamal commented Mar 20, 2024 •
edited

What Does This Do

This PR implements tracing disablement by stopping trace publishing but keeping span creation.
Changes from #6731 have been implemented here.

Motivation

RFC for disablement of tracing via remote config

Additional Notes

Jira ticket: APMJAVA-1248

@pr-commenter
Copy link

pr-commenter bot commented Mar 20, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/RC-experimental-tracing-enabled-tmp
git_commit_date 1711366976 1711376404
git_commit_sha bfd3a15 c535080
release_version 1.32.0-SNAPSHOT~bfd3a15e17 1.32.0-SNAPSHOT~c53508057e
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1711379914 1711379914
ci_job_id 468440497 468440497
ci_pipeline_id 30736153 30736153
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1075592
Total [baseline] (8.56 s) : 0, 8559994
Agent [candidate] (1.075 s) : 0, 1075465
Total [candidate] (8.552 s) : 0, 8551508
section iast
Agent [baseline] (1.209 s) : 0, 1208819
Total [baseline] (9.096 s) : 0, 9096394
Agent [candidate] (1.203 s) : 0, 1202999
Total [candidate] (9.095 s) : 0, 9094761
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.196 s) : 0, 1196098
Total [baseline] (9.083 s) : 0, 9082894
Agent [candidate] (1.208 s) : 0, 1208335
Total [candidate] (9.068 s) : 0, 9067913
section iast_TELEMETRY_OFF
Agent [baseline] (1.195 s) : 0, 1194611
Total [baseline] (9.046 s) : 0, 9046458
Agent [candidate] (1.197 s) : 0, 1197489
Total [candidate] (9.072 s) : 0, 9072169
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.076 s -
Agent iast 1.209 s 133.227 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.196 s 120.506 ms (11.2%)
Agent iast_TELEMETRY_OFF 1.195 s 119.019 ms (11.1%)
Total tracing 8.56 s -
Total iast 9.096 s 536.4 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.083 s 522.9 ms (6.1%)
Total iast_TELEMETRY_OFF 9.046 s 486.464 ms (5.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent iast 1.203 s 127.534 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.208 s 132.87 ms (12.4%)
Agent iast_TELEMETRY_OFF 1.197 s 122.024 ms (11.3%)
Total tracing 8.552 s -
Total iast 9.095 s 543.253 ms (6.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.068 s 516.404 ms (6.0%)
Total iast_TELEMETRY_OFF 9.072 s 520.661 ms (6.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.608 ms) : 0, 685608
BytebuddyAgent [candidate] (685.402 ms) : 0, 685402
GlobalTracer [baseline] (297.603 ms) : 0, 297603
GlobalTracer [candidate] (297.705 ms) : 0, 297705
AppSec [baseline] (49.369 ms) : 0, 49369
AppSec [candidate] (49.402 ms) : 0, 49402
Remote Config [baseline] (1.056 ms) : 0, 1056
Remote Config [candidate] (1.082 ms) : 0, 1082
Telemetry [baseline] (7.481 ms) : 0, 7481
Telemetry [candidate] (7.495 ms) : 0, 7495
section iast
BytebuddyAgent [baseline] (800.555 ms) : 0, 800555
BytebuddyAgent [candidate] (796.016 ms) : 0, 796016
GlobalTracer [baseline] (290.779 ms) : 0, 290779
GlobalTracer [candidate] (289.987 ms) : 0, 289987
AppSec [baseline] (52.289 ms) : 0, 52289
AppSec [candidate] (50.767 ms) : 0, 50767
IAST [baseline] (23.177 ms) : 0, 23177
IAST [candidate] (22.989 ms) : 0, 22989
Remote Config [baseline] (598.03 µs) : 0, 598
Remote Config [candidate] (594.506 µs) : 0, 595
Telemetry [baseline] (6.741 ms) : 0, 6741
Telemetry [candidate] (8.198 ms) : 0, 8198
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (792.262 ms) : 0, 792262
BytebuddyAgent [candidate] (799.867 ms) : 0, 799867
GlobalTracer [baseline] (288.498 ms) : 0, 288498
GlobalTracer [candidate] (291.177 ms) : 0, 291177
AppSec [baseline] (51.03 ms) : 0, 51030
AppSec [candidate] (50.545 ms) : 0, 50545
IAST [baseline] (22.603 ms) : 0, 22603
IAST [candidate] (24.722 ms) : 0, 24722
Remote Config [baseline] (612.696 µs) : 0, 613
Remote Config [candidate] (589.875 µs) : 0, 590
Telemetry [baseline] (6.657 ms) : 0, 6657
Telemetry [candidate] (6.73 ms) : 0, 6730
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (790.271 ms) : 0, 790271
BytebuddyAgent [candidate] (791.606 ms) : 0, 791606
GlobalTracer [baseline] (289.613 ms) : 0, 289613
GlobalTracer [candidate] (290.095 ms) : 0, 290095
AppSec [baseline] (51.522 ms) : 0, 51522
AppSec [candidate] (50.417 ms) : 0, 50417
IAST [baseline] (21.679 ms) : 0, 21679
IAST [candidate] (23.875 ms) : 0, 23875
Remote Config [baseline] (582.758 µs) : 0, 583
Remote Config [candidate] (578.299 µs) : 0, 578
Telemetry [baseline] (6.569 ms) : 0, 6569
Telemetry [candidate] (6.518 ms) : 0, 6518
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1073972
Total [baseline] (10.468 s) : 0, 10467670
Agent [candidate] (1.077 s) : 0, 1076523
Total [candidate] (10.373 s) : 0, 10373083
section appsec
Agent [baseline] (1.207 s) : 0, 1206724
Total [baseline] (10.544 s) : 0, 10543841
Agent [candidate] (1.2 s) : 0, 1200064
Total [candidate] (10.461 s) : 0, 10460867
section iast
Agent [baseline] (1.196 s) : 0, 1196093
Total [baseline] (10.875 s) : 0, 10874922
Agent [candidate] (1.206 s) : 0, 1206466
Total [candidate] (10.937 s) : 0, 10936876
section profiling
Agent [baseline] (1.275 s) : 0, 1274550
Total [baseline] (10.649 s) : 0, 10649480
Agent [candidate] (1.28 s) : 0, 1279710
Total [candidate] (10.669 s) : 0, 10668523
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent appsec 1.207 s 132.752 ms (12.4%)
Agent iast 1.196 s 122.121 ms (11.4%)
Agent profiling 1.275 s 200.578 ms (18.7%)
Total tracing 10.468 s -
Total appsec 10.544 s 76.171 ms (0.7%)
Total iast 10.875 s 407.252 ms (3.9%)
Total profiling 10.649 s 181.81 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent appsec 1.2 s 123.541 ms (11.5%)
Agent iast 1.206 s 129.944 ms (12.1%)
Agent profiling 1.28 s 203.187 ms (18.9%)
Total tracing 10.373 s -
Total appsec 10.461 s 87.784 ms (0.8%)
Total iast 10.937 s 563.793 ms (5.4%)
Total profiling 10.669 s 295.44 ms (2.8%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.388 ms) : 0, 684388
BytebuddyAgent [candidate] (685.692 ms) : 0, 685692
GlobalTracer [baseline] (297.535 ms) : 0, 297535
GlobalTracer [candidate] (298.519 ms) : 0, 298519
AppSec [baseline] (49.177 ms) : 0, 49177
AppSec [candidate] (49.387 ms) : 0, 49387
Remote Config [baseline] (1.038 ms) : 0, 1038
Remote Config [candidate] (1.029 ms) : 0, 1029
Telemetry [baseline] (7.42 ms) : 0, 7420
Telemetry [candidate] (7.424 ms) : 0, 7424
section appsec
BytebuddyAgent [baseline] (698.646 ms) : 0, 698646
BytebuddyAgent [candidate] (693.813 ms) : 0, 693813
GlobalTracer [baseline] (292.985 ms) : 0, 292985
GlobalTracer [candidate] (292.295 ms) : 0, 292295
AppSec [baseline] (154.285 ms) : 0, 154285
AppSec [candidate] (153.502 ms) : 0, 153502
IAST [baseline] (18.847 ms) : 0, 18847
IAST [candidate] (18.683 ms) : 0, 18683
Remote Config [baseline] (610.248 µs) : 0, 610
Remote Config [candidate] (607.217 µs) : 0, 607
Telemetry [baseline] (6.839 ms) : 0, 6839
Telemetry [candidate] (6.767 ms) : 0, 6767
section iast
BytebuddyAgent [baseline] (791.691 ms) : 0, 791691
BytebuddyAgent [candidate] (799.723 ms) : 0, 799723
GlobalTracer [baseline] (288.243 ms) : 0, 288243
GlobalTracer [candidate] (291.015 ms) : 0, 291015
AppSec [baseline] (50.267 ms) : 0, 50267
AppSec [candidate] (51.288 ms) : 0, 51288
IAST [baseline] (24.343 ms) : 0, 24343
IAST [candidate] (22.598 ms) : 0, 22598
Remote Config [baseline] (577.018 µs) : 0, 577
Remote Config [candidate] (581.317 µs) : 0, 581
Telemetry [baseline] (6.666 ms) : 0, 6666
Telemetry [candidate] (6.65 ms) : 0, 6650
section profiling
BytebuddyAgent [baseline] (678.897 ms) : 0, 678897
BytebuddyAgent [candidate] (680.926 ms) : 0, 680926
GlobalTracer [baseline] (384.273 ms) : 0, 384273
GlobalTracer [candidate] (386.362 ms) : 0, 386362
AppSec [baseline] (50.239 ms) : 0, 50239
AppSec [candidate] (50.599 ms) : 0, 50599
Remote Config [baseline] (900.758 µs) : 0, 901
Remote Config [candidate] (886.018 µs) : 0, 886
Telemetry [baseline] (7.539 ms) : 0, 7539
Telemetry [candidate] (7.527 ms) : 0, 7527
ProfilingAgent [baseline] (96.374 ms) : 0, 96374
ProfilingAgent [candidate] (96.678 ms) : 0, 96678
Profiling [baseline] (96.398 ms) : 0, 96398
Profiling [candidate] (96.702 ms) : 0, 96702

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-03-25T14:50:47 2024-03-25T15:12:49
git_branch master nayeem-kamal/RC-experimental-tracing-enabled-tmp
git_commit_date 1711366976 1711376404
git_commit_sha bfd3a15 c535080
release_version 1.32.0-SNAPSHOT~bfd3a15e17 1.32.0-SNAPSHOT~c53508057e
start_time 2024-03-25T14:50:34 2024-03-25T15:12:36
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1711379914 1711379914
ci_job_id 468440497 468440497
ci_pipeline_id 30736153 30736153
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.41 µs) : 347, 387
.   : milestone, 367,
iast (474.343 µs) : 454, 495
.   : milestone, 474,
iast_FULL (540.018 µs) : 519, 561
.   : milestone, 540,
iast_GLOBAL (499.302 µs) : 478, 521
.   : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (469.248 µs) : 449, 490
.   : milestone, 469,
iast_INACTIVE (439.119 µs) : 419, 459
.   : milestone, 439,
iast_TELEMETRY_OFF (469.887 µs) : 449, 491
.   : milestone, 470,
tracing (435.799 µs) : 416, 456
.   : milestone, 436,
section candidate
no_agent (365.567 µs) : 346, 385
.   : milestone, 366,
iast (473.982 µs) : 453, 495
.   : milestone, 474,
iast_FULL (546.642 µs) : 526, 567
.   : milestone, 547,
iast_GLOBAL (497.289 µs) : 476, 519
.   : milestone, 497,
iast_HARDCODED_SECRET_DISABLED (466.081 µs) : 446, 486
.   : milestone, 466,
iast_INACTIVE (441.19 µs) : 420, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (466.046 µs) : 445, 487
.   : milestone, 466,
tracing (434.045 µs) : 414, 454
.   : milestone, 434,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.41 µs [347.332 µs, 387.488 µs] -
iast 474.343 µs [453.846 µs, 494.839 µs] 106.933 µs (29.1%)
iast_FULL 540.018 µs [519.265 µs, 560.77 µs] 172.608 µs (47.0%)
iast_GLOBAL 499.302 µs [478.015 µs, 520.588 µs] 131.892 µs (35.9%)
iast_HARDCODED_SECRET_DISABLED 469.248 µs [448.683 µs, 489.814 µs] 101.839 µs (27.7%)
iast_INACTIVE 439.119 µs [418.879 µs, 459.358 µs] 71.709 µs (19.5%)
iast_TELEMETRY_OFF 469.887 µs [448.816 µs, 490.958 µs] 102.477 µs (27.9%)
tracing 435.799 µs [415.55 µs, 456.048 µs] 68.389 µs (18.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.567 µs [346.216 µs, 384.919 µs] -
iast 473.982 µs [453.045 µs, 494.918 µs] 108.414 µs (29.7%)
iast_FULL 546.642 µs [526.022 µs, 567.263 µs] 181.075 µs (49.5%)
iast_GLOBAL 497.289 µs [476.021 µs, 518.556 µs] 131.721 µs (36.0%)
iast_HARDCODED_SECRET_DISABLED 466.081 µs [445.709 µs, 486.453 µs] 100.514 µs (27.5%)
iast_INACTIVE 441.19 µs [420.349 µs, 462.03 µs] 75.622 µs (20.7%)
iast_TELEMETRY_OFF 466.046 µs [444.979 µs, 487.114 µs] 100.479 µs (27.5%)
tracing 434.045 µs [414.053 µs, 454.038 µs] 68.478 µs (18.7%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~c53508057e, baseline=1.32.0-SNAPSHOT~bfd3a15e17
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.344 ms) : 1324, 1363
.   : milestone, 1344,
appsec (1.723 ms) : 1698, 1747
.   : milestone, 1723,
appsec_no_iast (1.763 ms) : 1739, 1787
.   : milestone, 1763,
iast (1.487 ms) : 1464, 1510
.   : milestone, 1487,
profiling (1.535 ms) : 1509, 1561
.   : milestone, 1535,
tracing (1.471 ms) : 1446, 1495
.   : milestone, 1471,
section candidate
no_agent (1.351 ms) : 1331, 1371
.   : milestone, 1351,
appsec (1.74 ms) : 1716, 1763
.   : milestone, 1740,
appsec_no_iast (1.759 ms) : 1735, 1784
.   : milestone, 1759,
iast (1.481 ms) : 1458, 1504
.   : milestone, 1481,
profiling (1.49 ms) : 1466, 1515
.   : milestone, 1490,
tracing (1.461 ms) : 1437, 1484
.   : milestone, 1461,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.344 ms [1.324 ms, 1.363 ms] -
appsec 1.723 ms [1.698 ms, 1.747 ms] 378.998 µs (28.2%)
appsec_no_iast 1.763 ms [1.739 ms, 1.787 ms] 419.67 µs (31.2%)
iast 1.487 ms [1.464 ms, 1.51 ms] 143.719 µs (10.7%)
profiling 1.535 ms [1.509 ms, 1.561 ms] 191.349 µs (14.2%)
tracing 1.471 ms [1.446 ms, 1.495 ms] 127.065 µs (9.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.351 ms [1.331 ms, 1.371 ms] -
appsec 1.74 ms [1.716 ms, 1.763 ms] 388.814 µs (28.8%)
appsec_no_iast 1.759 ms [1.735 ms, 1.784 ms] 408.378 µs (30.2%)
iast 1.481 ms [1.458 ms, 1.504 ms] 130.096 µs (9.6%)
profiling 1.49 ms [1.466 ms, 1.515 ms] 139.51 µs (10.3%)
tracing 1.461 ms [1.437 ms, 1.484 ms] 109.841 µs (8.1%)

@nayeem-kamal nayeem-kamal changed the title rc experimental tracing enabled Allow for disablement of tracing via remote config Mar 25, 2024
@nayeem-kamal nayeem-kamal marked this pull request as ready for review March 25, 2024 14:27
@nayeem-kamal nayeem-kamal requested a review from a team as a code owner March 25, 2024 14:27
@nayeem-kamal nayeem-kamal mentioned this pull request Mar 25, 2024
Closed
@nayeem-kamal nayeem-kamal merged commit f56be02 into master Mar 27, 2024
80 checks passed
@nayeem-kamal nayeem-kamal deleted the nayeem-kamal/RC-experimental-tracing-enabled-tmp branch March 27, 2024 14:05
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 27, 2024
@PerfectSlayer PerfectSlayer added the comp: remote config Configuration at Runtime label Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@am312 am312 am312 approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: remote config Configuration at Runtime
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@nayeem-kamal @am312 @PerfectSlayer