Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support ordering of InstrumenterModules #6840

Merged
merged 1 commit into from Mar 27, 2024
Merged

Support ordering of InstrumenterModules #6840

merged 1 commit into from Mar 27, 2024

Conversation

mcculls
Copy link
Contributor

@mcculls mcculls commented Mar 25, 2024 •
edited

Motivation

Allows modules to be applied in a particular order, rather than just the order they appear in the buiild.

Note: this ordering is applied at build-time before generating the instrumenter.index used at runtime.

Jira ticket: AIT-9441

@mcculls mcculls added the comp: core Tracer core label Mar 25, 2024
@mcculls mcculls force-pushed the mcculls/support-instrumenter-module-ordering branch from 5715b62 to 63dfdf5 Compare March 25, 2024 12:27
@mcculls mcculls marked this pull request as ready for review March 25, 2024 12:30
@mcculls mcculls requested a review from a team as a code owner March 25, 2024 12:30
@pr-commenter
Copy link

pr-commenter bot commented Mar 25, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/support-instrumenter-module-ordering
git_commit_date 1711569848 1711569901
git_commit_sha 1a0ea42 e367ec9
release_version 1.32.0-SNAPSHOT~1a0ea421c6 1.32.0-SNAPSHOT~e367ec93f4
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1711573267 1711573267
ci_job_id 471481802 471481802
ci_pipeline_id 30964942 30964942
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1073755
Total [baseline] (10.344 s) : 0, 10344458
Agent [candidate] (1.077 s) : 0, 1076958
Total [candidate] (10.476 s) : 0, 10475693
section appsec
Agent [baseline] (1.197 s) : 0, 1196924
Total [baseline] (10.6 s) : 0, 10600310
Agent [candidate] (1.195 s) : 0, 1194892
Total [candidate] (10.565 s) : 0, 10564512
section iast
Agent [baseline] (1.216 s) : 0, 1215762
Total [baseline] (10.829 s) : 0, 10828770
Agent [candidate] (1.199 s) : 0, 1198804
Total [candidate] (10.821 s) : 0, 10821244
section profiling
Agent [baseline] (1.277 s) : 0, 1276888
Total [baseline] (10.72 s) : 0, 10719655
Agent [candidate] (1.278 s) : 0, 1278316
Total [candidate] (10.694 s) : 0, 10694396
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent appsec 1.197 s 123.169 ms (11.5%)
Agent iast 1.216 s 142.006 ms (13.2%)
Agent profiling 1.277 s 203.133 ms (18.9%)
Total tracing 10.344 s -
Total appsec 10.6 s 255.852 ms (2.5%)
Total iast 10.829 s 484.312 ms (4.7%)
Total profiling 10.72 s 375.197 ms (3.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent appsec 1.195 s 117.934 ms (11.0%)
Agent iast 1.199 s 121.846 ms (11.3%)
Agent profiling 1.278 s 201.358 ms (18.7%)
Total tracing 10.476 s -
Total appsec 10.565 s 88.819 ms (0.8%)
Total iast 10.821 s 345.55 ms (3.3%)
Total profiling 10.694 s 218.702 ms (2.1%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.649 ms) : 0, 684649
BytebuddyAgent [candidate] (685.895 ms) : 0, 685895
GlobalTracer [baseline] (296.791 ms) : 0, 296791
GlobalTracer [candidate] (298.489 ms) : 0, 298489
AppSec [baseline] (49.751 ms) : 0, 49751
AppSec [candidate] (49.734 ms) : 0, 49734
Remote Config [baseline] (668.528 µs) : 0, 669
Remote Config [candidate] (671.263 µs) : 0, 671
Telemetry [baseline] (7.621 ms) : 0, 7621
Telemetry [candidate] (7.657 ms) : 0, 7657
section appsec
BytebuddyAgent [baseline] (694.732 ms) : 0, 694732
BytebuddyAgent [candidate] (693.436 ms) : 0, 693436
GlobalTracer [baseline] (291.224 ms) : 0, 291224
GlobalTracer [candidate] (291.636 ms) : 0, 291636
AppSec [baseline] (149.781 ms) : 0, 149781
AppSec [candidate] (149.389 ms) : 0, 149389
IAST [baseline] (18.768 ms) : 0, 18768
IAST [candidate] (18.772 ms) : 0, 18772
Remote Config [baseline] (605.789 µs) : 0, 606
Remote Config [candidate] (606.709 µs) : 0, 607
Telemetry [baseline] (7.381 ms) : 0, 7381
Telemetry [candidate] (6.764 ms) : 0, 6764
section iast
BytebuddyAgent [baseline] (805.961 ms) : 0, 805961
BytebuddyAgent [candidate] (794.227 ms) : 0, 794227
GlobalTracer [baseline] (292.16 ms) : 0, 292160
GlobalTracer [candidate] (288.988 ms) : 0, 288988
AppSec [baseline] (52.402 ms) : 0, 52402
AppSec [candidate] (49.709 ms) : 0, 49709
IAST [baseline] (23.216 ms) : 0, 23216
IAST [candidate] (23.614 ms) : 0, 23614
Remote Config [baseline] (583.462 µs) : 0, 583
Remote Config [candidate] (586.342 µs) : 0, 586
Telemetry [baseline] (6.685 ms) : 0, 6685
Telemetry [candidate] (7.366 ms) : 0, 7366
section profiling
BytebuddyAgent [baseline] (681.876 ms) : 0, 681876
BytebuddyAgent [candidate] (682.362 ms) : 0, 682362
GlobalTracer [baseline] (382.794 ms) : 0, 382794
GlobalTracer [candidate] (384.031 ms) : 0, 384031
AppSec [baseline] (50.758 ms) : 0, 50758
AppSec [candidate] (50.564 ms) : 0, 50564
Remote Config [baseline] (717.719 µs) : 0, 718
Remote Config [candidate] (711.717 µs) : 0, 712
Telemetry [baseline] (7.55 ms) : 0, 7550
Telemetry [candidate] (7.518 ms) : 0, 7518
ProfilingAgent [baseline] (96.524 ms) : 0, 96524
ProfilingAgent [candidate] (96.48 ms) : 0, 96480
Profiling [baseline] (96.548 ms) : 0, 96548
Profiling [candidate] (96.505 ms) : 0, 96505
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1090073
Total [baseline] (8.613 s) : 0, 8612646
Agent [candidate] (1.075 s) : 0, 1075420
Total [candidate] (8.591 s) : 0, 8591358
section iast
Agent [baseline] (1.207 s) : 0, 1207038
Total [baseline] (9.108 s) : 0, 9107701
Agent [candidate] (1.208 s) : 0, 1208466
Total [candidate] (9.102 s) : 0, 9102010
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.197 s) : 0, 1197307
Total [baseline] (9.039 s) : 0, 9039412
Agent [candidate] (1.202 s) : 0, 1201872
Total [candidate] (9.084 s) : 0, 9083962
section iast_TELEMETRY_OFF
Agent [baseline] (1.203 s) : 0, 1203459
Total [baseline] (9.117 s) : 0, 9117479
Agent [candidate] (1.205 s) : 0, 1205078
Total [candidate] (9.12 s) : 0, 9119900
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent iast 1.207 s 116.965 ms (10.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.197 s 107.234 ms (9.8%)
Agent iast_TELEMETRY_OFF 1.203 s 113.387 ms (10.4%)
Total tracing 8.613 s -
Total iast 9.108 s 495.054 ms (5.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.039 s 426.765 ms (5.0%)
Total iast_TELEMETRY_OFF 9.117 s 504.832 ms (5.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent iast 1.208 s 133.046 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.202 s 126.452 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.205 s 129.658 ms (12.1%)
Total tracing 8.591 s -
Total iast 9.102 s 510.651 ms (5.9%)
Total iast_HARDCODED_SECRET_DISABLED 9.084 s 492.604 ms (5.7%)
Total iast_TELEMETRY_OFF 9.12 s 528.541 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (695.392 ms) : 0, 695392
BytebuddyAgent [candidate] (685.258 ms) : 0, 685258
GlobalTracer [baseline] (301.087 ms) : 0, 301087
GlobalTracer [candidate] (298.046 ms) : 0, 298046
AppSec [baseline] (50.295 ms) : 0, 50295
AppSec [candidate] (49.578 ms) : 0, 49578
Remote Config [baseline] (683.303 µs) : 0, 683
Remote Config [candidate] (653.183 µs) : 0, 653
Telemetry [baseline] (7.724 ms) : 0, 7724
Telemetry [candidate] (7.567 ms) : 0, 7567
section iast
BytebuddyAgent [baseline] (800.07 ms) : 0, 800070
BytebuddyAgent [candidate] (800.689 ms) : 0, 800689
GlobalTracer [baseline] (290.0 ms) : 0, 290000
GlobalTracer [candidate] (290.829 ms) : 0, 290829
AppSec [baseline] (51.778 ms) : 0, 51778
AppSec [candidate] (52.043 ms) : 0, 52043
IAST [baseline] (22.522 ms) : 0, 22522
IAST [candidate] (23.012 ms) : 0, 23012
Remote Config [baseline] (589.431 µs) : 0, 589
Remote Config [candidate] (571.972 µs) : 0, 572
Telemetry [baseline] (7.469 ms) : 0, 7469
Telemetry [candidate] (6.675 ms) : 0, 6675
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (794.577 ms) : 0, 794577
BytebuddyAgent [candidate] (795.154 ms) : 0, 795154
GlobalTracer [baseline] (288.16 ms) : 0, 288160
GlobalTracer [candidate] (289.814 ms) : 0, 289814
AppSec [baseline] (49.324 ms) : 0, 49324
AppSec [candidate] (50.435 ms) : 0, 50435
IAST [baseline] (23.668 ms) : 0, 23668
IAST [candidate] (23.943 ms) : 0, 23943
Remote Config [baseline] (588.655 µs) : 0, 589
Remote Config [candidate] (576.912 µs) : 0, 577
Telemetry [baseline] (6.621 ms) : 0, 6621
Telemetry [candidate] (7.361 ms) : 0, 7361
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (796.644 ms) : 0, 796644
BytebuddyAgent [candidate] (798.558 ms) : 0, 798558
GlobalTracer [baseline] (290.126 ms) : 0, 290126
GlobalTracer [candidate] (290.429 ms) : 0, 290429
AppSec [baseline] (48.386 ms) : 0, 48386
AppSec [candidate] (49.197 ms) : 0, 49197
IAST [baseline] (26.533 ms) : 0, 26533
IAST [candidate] (24.928 ms) : 0, 24928
Remote Config [baseline] (584.689 µs) : 0, 585
Remote Config [candidate] (583.862 µs) : 0, 584
Telemetry [baseline] (6.534 ms) : 0, 6534
Telemetry [candidate] (6.578 ms) : 0, 6578

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-03-27T20:33:16 2024-03-27T20:55:21
git_branch master mcculls/support-instrumenter-module-ordering
git_commit_date 1711569848 1711569901
git_commit_sha 1a0ea42 e367ec9
release_version 1.32.0-SNAPSHOT~1a0ea421c6 1.32.0-SNAPSHOT~e367ec93f4
start_time 2024-03-27T20:33:03 2024-03-27T20:55:08
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1711573267 1711573267
ci_job_id 471481802 471481802
ci_pipeline_id 30964942 30964942
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.35 ms) : 1331, 1370
.   : milestone, 1350,
appsec (1.735 ms) : 1711, 1759
.   : milestone, 1735,
appsec_no_iast (1.714 ms) : 1690, 1739
.   : milestone, 1714,
iast (1.491 ms) : 1468, 1513
.   : milestone, 1491,
profiling (1.508 ms) : 1483, 1533
.   : milestone, 1508,
tracing (1.469 ms) : 1444, 1494
.   : milestone, 1469,
section candidate
no_agent (1.344 ms) : 1324, 1365
.   : milestone, 1344,
appsec (1.719 ms) : 1694, 1745
.   : milestone, 1719,
appsec_no_iast (1.71 ms) : 1685, 1735
.   : milestone, 1710,
iast (1.489 ms) : 1466, 1512
.   : milestone, 1489,
profiling (1.543 ms) : 1518, 1569
.   : milestone, 1543,
tracing (1.466 ms) : 1441, 1490
.   : milestone, 1466,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.35 ms [1.331 ms, 1.37 ms] -
appsec 1.735 ms [1.711 ms, 1.759 ms] 384.409 µs (28.5%)
appsec_no_iast 1.714 ms [1.69 ms, 1.739 ms] 363.988 µs (27.0%)
iast 1.491 ms [1.468 ms, 1.513 ms] 140.424 µs (10.4%)
profiling 1.508 ms [1.483 ms, 1.533 ms] 157.417 µs (11.7%)
tracing 1.469 ms [1.444 ms, 1.494 ms] 118.763 µs (8.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.344 ms [1.324 ms, 1.365 ms] -
appsec 1.719 ms [1.694 ms, 1.745 ms] 374.996 µs (27.9%)
appsec_no_iast 1.71 ms [1.685 ms, 1.735 ms] 365.502 µs (27.2%)
iast 1.489 ms [1.466 ms, 1.512 ms] 144.524 µs (10.8%)
profiling 1.543 ms [1.518 ms, 1.569 ms] 198.945 µs (14.8%)
tracing 1.466 ms [1.441 ms, 1.49 ms] 121.173 µs (9.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e367ec93f4, baseline=1.32.0-SNAPSHOT~1a0ea421c6
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.414 µs) : 348, 389
.   : milestone, 368,
iast (473.886 µs) : 453, 494
.   : milestone, 474,
iast_FULL (542.939 µs) : 521, 565
.   : milestone, 543,
iast_GLOBAL (491.829 µs) : 471, 513
.   : milestone, 492,
iast_HARDCODED_SECRET_DISABLED (476.671 µs) : 456, 498
.   : milestone, 477,
iast_INACTIVE (447.617 µs) : 427, 468
.   : milestone, 448,
iast_TELEMETRY_OFF (473.426 µs) : 452, 494
.   : milestone, 473,
tracing (437.206 µs) : 417, 458
.   : milestone, 437,
section candidate
no_agent (369.498 µs) : 349, 390
.   : milestone, 369,
iast (468.945 µs) : 448, 490
.   : milestone, 469,
iast_FULL (538.526 µs) : 518, 559
.   : milestone, 539,
iast_GLOBAL (496.228 µs) : 476, 517
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (473.575 µs) : 453, 494
.   : milestone, 474,
iast_INACTIVE (449.196 µs) : 428, 471
.   : milestone, 449,
iast_TELEMETRY_OFF (470.263 µs) : 449, 491
.   : milestone, 470,
tracing (445.681 µs) : 425, 467
.   : milestone, 446,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.414 µs [348.076 µs, 388.752 µs] -
iast 473.886 µs [453.349 µs, 494.423 µs] 105.472 µs (28.6%)
iast_FULL 542.939 µs [521.344 µs, 564.534 µs] 174.525 µs (47.4%)
iast_GLOBAL 491.829 µs [470.852 µs, 512.805 µs] 123.415 µs (33.5%)
iast_HARDCODED_SECRET_DISABLED 476.671 µs [455.745 µs, 497.597 µs] 108.257 µs (29.4%)
iast_INACTIVE 447.617 µs [426.813 µs, 468.421 µs] 79.203 µs (21.5%)
iast_TELEMETRY_OFF 473.426 µs [452.381 µs, 494.472 µs] 105.012 µs (28.5%)
tracing 437.206 µs [416.75 µs, 457.662 µs] 68.792 µs (18.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 369.498 µs [348.969 µs, 390.027 µs] -
iast 468.945 µs [448.389 µs, 489.5 µs] 99.447 µs (26.9%)
iast_FULL 538.526 µs [517.954 µs, 559.098 µs] 169.028 µs (45.7%)
iast_GLOBAL 496.228 µs [475.772 µs, 516.684 µs] 126.73 µs (34.3%)
iast_HARDCODED_SECRET_DISABLED 473.575 µs [453.076 µs, 494.074 µs] 104.077 µs (28.2%)
iast_INACTIVE 449.196 µs [427.771 µs, 470.62 µs] 79.698 µs (21.6%)
iast_TELEMETRY_OFF 470.263 µs [449.118 µs, 491.407 µs] 100.765 µs (27.3%)
tracing 445.681 µs [424.561 µs, 466.801 µs] 76.183 µs (20.6%)

dougqh
dougqh approved these changes Mar 27, 2024
View reviewed changes
Copy link
Contributor

@dougqh dougqh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume this is coming from OpenTelemetry.
I worried about the brittleness of such a solution. I would kind of prefer more explicit dependencies.

But if we need to support this, we to support it.

@mcculls
Copy link
Contributor Author

mcculls commented Mar 27, 2024

I presume this is coming from OpenTelemetry.

Yes this is required to align with OpenTelemetry

Base automatically changed from mcculls/generate-instrumenter-index to master March 27, 2024 20:04
@mcculls mcculls force-pushed the mcculls/support-instrumenter-module-ordering branch from 63dfdf5 to e367ec9 Compare March 27, 2024 20:05
@mcculls mcculls merged commit 722243d into master Mar 27, 2024
77 checks passed
@mcculls mcculls deleted the mcculls/support-instrumenter-module-ordering branch March 27, 2024 21:06
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dougqh dougqh dougqh approved these changes

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal is a code owner automatically assigned from DataDog/apm-java

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer

Assignees
No one assigned
Labels
comp: core Tracer core type: enhancement
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mcculls @dougqh @PerfectSlayer