Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add request sampling back in http sources #6810

Merged
merged 2 commits into from Mar 14, 2024
Merged

Add request sampling back in http sources #6810

merged 2 commits into from Mar 14, 2024

Conversation

manuel-alvarez-alvarez
Copy link
Contributor

@manuel-alvarez-alvarez manuel-alvarez-alvarez commented Mar 13, 2024 •
edited

What Does This Do

Adds sampling back to HTTP sources so there will be no tainting if no active IAST context is present. The way it's done is by checking for the presence of an IastRequestContext in the current span at the callsite/addvice level. It brings two benefits:

  • In REQUEST mode the tainting operation happens sooner so less code is run when a request is sampled
  • In GLOBAL mode now sources are sampled too bringing less load to services.

Motivation

When using DD_IAST_CONTEXT_MODE=GLOBAL, tainting all incoming HTTP data is starting to hurt in terms of performance, this PR reduces the amount of work done by IAST ending in less CPU consumption.

@manuel-alvarez-alvarez manuel-alvarez-alvarez added tag: performance Performance related changes comp: asm iast Application Security Management (IAST) labels Mar 13, 2024
@manuel-alvarez-alvarez manuel-alvarez-alvarez force-pushed the malvarez/iast-recover-sources-sampling branch from 3c36750 to e542e02 Compare March 13, 2024 16:16
@pr-commenter
Copy link

pr-commenter bot commented Mar 13, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master malvarez/iast-recover-sources-sampling
git_commit_date 1710407839 1710415194
git_commit_sha f19053f e78a54c
release_version 1.32.0-SNAPSHOT~f19053f7e4 1.32.0-SNAPSHOT~e78a54c6f0
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1710418745 1710418745
ci_job_id 459555840 459555840
ci_pipeline_id 30086610 30086610
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 13 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.089 s) : 0, 1088676
Total [baseline] (9.223 s) : 0, 9222970
Agent [candidate] (1.08 s) : 0, 1079534
Total [candidate] (9.136 s) : 0, 9135867
section appsec
Agent [baseline] (1.2 s) : 0, 1200367
Total [baseline] (9.32 s) : 0, 9319910
Agent [candidate] (1.2 s) : 0, 1200476
Total [candidate] (9.353 s) : 0, 9352828
section iast
Agent [baseline] (1.203 s) : 0, 1202651
Total [baseline] (9.284 s) : 0, 9283875
Agent [candidate] (1.203 s) : 0, 1203111
Total [candidate] (9.317 s) : 0, 9316759
section profiling
Agent [baseline] (1.273 s) : 0, 1272815
Total [baseline] (9.447 s) : 0, 9446614
Agent [candidate] (1.282 s) : 0, 1282296
Total [candidate] (9.474 s) : 0, 9473534
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.089 s -
Agent appsec 1.2 s 111.691 ms (10.3%)
Agent iast 1.203 s 113.975 ms (10.5%)
Agent profiling 1.273 s 184.14 ms (16.9%)
Total tracing 9.223 s -
Total appsec 9.32 s 96.941 ms (1.1%)
Total iast 9.284 s 60.905 ms (0.7%)
Total profiling 9.447 s 223.644 ms (2.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.08 s -
Agent appsec 1.2 s 120.942 ms (11.2%)
Agent iast 1.203 s 123.577 ms (11.4%)
Agent profiling 1.282 s 202.762 ms (18.8%)
Total tracing 9.136 s -
Total appsec 9.353 s 216.961 ms (2.4%)
Total iast 9.317 s 180.892 ms (2.0%)
Total profiling 9.474 s 337.667 ms (3.7%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (703.523 ms) : 0, 703523
BytebuddyAgent [candidate] (695.614 ms) : 0, 695614
GlobalTracer [baseline] (292.835 ms) : 0, 292835
GlobalTracer [candidate] (292.183 ms) : 0, 292183
AppSec [baseline] (49.023 ms) : 0, 49023
AppSec [candidate] (48.923 ms) : 0, 48923
Remote Config [baseline] (747.173 µs) : 0, 747
Remote Config [candidate] (750.924 µs) : 0, 751
Telemetry [baseline] (7.8 ms) : 0, 7800
Telemetry [candidate] (7.763 ms) : 0, 7763
section appsec
BytebuddyAgent [baseline] (696.604 ms) : 0, 696604
BytebuddyAgent [candidate] (696.398 ms) : 0, 696398
GlobalTracer [baseline] (290.826 ms) : 0, 290826
GlobalTracer [candidate] (291.891 ms) : 0, 291891
AppSec [baseline] (153.454 ms) : 0, 153454
AppSec [candidate] (152.698 ms) : 0, 152698
IAST [baseline] (17.827 ms) : 0, 17827
IAST [candidate] (17.849 ms) : 0, 17849
Remote Config [baseline] (605.322 µs) : 0, 605
Remote Config [candidate] (604.29 µs) : 0, 604
Telemetry [baseline] (6.856 ms) : 0, 6856
Telemetry [candidate] (6.848 ms) : 0, 6848
section iast
BytebuddyAgent [baseline] (799.381 ms) : 0, 799381
BytebuddyAgent [candidate] (798.978 ms) : 0, 798978
GlobalTracer [baseline] (288.017 ms) : 0, 288017
GlobalTracer [candidate] (289.007 ms) : 0, 289007
AppSec [baseline] (50.133 ms) : 0, 50133
AppSec [candidate] (50.062 ms) : 0, 50062
IAST [baseline] (23.743 ms) : 0, 23743
IAST [candidate] (23.669 ms) : 0, 23669
Remote Config [baseline] (608.867 µs) : 0, 609
Remote Config [candidate] (578.014 µs) : 0, 578
Telemetry [baseline] (6.579 ms) : 0, 6579
Telemetry [candidate] (6.621 ms) : 0, 6621
section profiling
BytebuddyAgent [baseline] (688.882 ms) : 0, 688882
BytebuddyAgent [candidate] (694.626 ms) : 0, 694626
GlobalTracer [baseline] (375.261 ms) : 0, 375261
GlobalTracer [candidate] (378.262 ms) : 0, 378262
AppSec [baseline] (49.51 ms) : 0, 49510
AppSec [candidate] (49.693 ms) : 0, 49693
Remote Config [baseline] (792.256 µs) : 0, 792
Remote Config [candidate] (794.116 µs) : 0, 794
Telemetry [baseline] (7.65 ms) : 0, 7650
Telemetry [candidate] (7.458 ms) : 0, 7458
ProfilingAgent [baseline] (94.701 ms) : 0, 94701
ProfilingAgent [candidate] (94.971 ms) : 0, 94971
Profiling [baseline] (94.725 ms) : 0, 94725
Profiling [candidate] (94.995 ms) : 0, 94995
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1077833
Total [baseline] (8.542 s) : 0, 8541582
Agent [candidate] (1.081 s) : 0, 1081385
Total [candidate] (8.561 s) : 0, 8560666
section iast
Agent [baseline] (1.204 s) : 0, 1203558
Total [baseline] (9.041 s) : 0, 9041157
Agent [candidate] (1.204 s) : 0, 1203902
Total [candidate] (9.061 s) : 0, 9061425
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.211 s) : 0, 1211263
Total [baseline] (9.052 s) : 0, 9051800
Agent [candidate] (1.222 s) : 0, 1221896
Total [candidate] (9.065 s) : 0, 9065266
section iast_TELEMETRY_OFF
Agent [baseline] (1.195 s) : 0, 1195012
Total [baseline] (9.054 s) : 0, 9054115
Agent [candidate] (1.206 s) : 0, 1206432
Total [candidate] (9.068 s) : 0, 9068242
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent iast 1.204 s 125.725 ms (11.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 133.43 ms (12.4%)
Agent iast_TELEMETRY_OFF 1.195 s 117.179 ms (10.9%)
Total tracing 8.542 s -
Total iast 9.041 s 499.575 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.052 s 510.218 ms (6.0%)
Total iast_TELEMETRY_OFF 9.054 s 512.533 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.081 s -
Agent iast 1.204 s 122.517 ms (11.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.222 s 140.511 ms (13.0%)
Agent iast_TELEMETRY_OFF 1.206 s 125.047 ms (11.6%)
Total tracing 8.561 s -
Total iast 9.061 s 500.759 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.065 s 504.6 ms (5.9%)
Total iast_TELEMETRY_OFF 9.068 s 507.575 ms (5.9%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (695.697 ms) : 0, 695697
BytebuddyAgent [candidate] (696.908 ms) : 0, 696908
GlobalTracer [baseline] (290.785 ms) : 0, 290785
GlobalTracer [candidate] (292.619 ms) : 0, 292619
AppSec [baseline] (48.628 ms) : 0, 48628
AppSec [candidate] (48.999 ms) : 0, 48999
Remote Config [baseline] (742.353 µs) : 0, 742
Remote Config [candidate] (743.327 µs) : 0, 743
Telemetry [baseline] (7.691 ms) : 0, 7691
Telemetry [candidate] (7.816 ms) : 0, 7816
section iast
BytebuddyAgent [baseline] (799.638 ms) : 0, 799638
BytebuddyAgent [candidate] (799.682 ms) : 0, 799682
GlobalTracer [baseline] (288.444 ms) : 0, 288444
GlobalTracer [candidate] (288.875 ms) : 0, 288875
AppSec [baseline] (50.845 ms) : 0, 50845
AppSec [candidate] (51.5 ms) : 0, 51500
Remote Config [baseline] (627.8 µs) : 0, 628
Remote Config [candidate] (617.032 µs) : 0, 617
Telemetry [baseline] (6.652 ms) : 0, 6652
Telemetry [candidate] (7.439 ms) : 0, 7439
IAST [baseline] (22.936 ms) : 0, 22936
IAST [candidate] (21.457 ms) : 0, 21457
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (804.956 ms) : 0, 804956
BytebuddyAgent [candidate] (811.68 ms) : 0, 811680
GlobalTracer [baseline] (289.974 ms) : 0, 289974
GlobalTracer [candidate] (292.793 ms) : 0, 292793
AppSec [baseline] (51.325 ms) : 0, 51325
AppSec [candidate] (49.365 ms) : 0, 49365
Remote Config [baseline] (618.96 µs) : 0, 619
Remote Config [candidate] (609.327 µs) : 0, 609
Telemetry [baseline] (6.657 ms) : 0, 6657
Telemetry [candidate] (7.437 ms) : 0, 7437
IAST [baseline] (23.0 ms) : 0, 23000
IAST [candidate] (25.082 ms) : 0, 25082
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (791.432 ms) : 0, 791432
BytebuddyAgent [candidate] (799.124 ms) : 0, 799124
GlobalTracer [baseline] (288.163 ms) : 0, 288163
GlobalTracer [candidate] (291.082 ms) : 0, 291082
AppSec [baseline] (46.863 ms) : 0, 46863
AppSec [candidate] (50.325 ms) : 0, 50325
Remote Config [baseline] (583.606 µs) : 0, 584
Remote Config [candidate] (592.003 µs) : 0, 592
Telemetry [baseline] (6.545 ms) : 0, 6545
Telemetry [candidate] (6.597 ms) : 0, 6597
IAST [baseline] (27.074 ms) : 0, 27074
IAST [candidate] (24.269 ms) : 0, 24269

Load

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.341 ms) : 1321, 1360
.   : milestone, 1341,
appsec (1.745 ms) : 1722, 1769
.   : milestone, 1745,
iast (1.506 ms) : 1482, 1529
.   : milestone, 1506,
profiling (1.542 ms) : 1517, 1567
.   : milestone, 1542,
tracing (1.51 ms) : 1487, 1534
.   : milestone, 1510,
section candidate
no_agent (1.339 ms) : 1320, 1358
.   : milestone, 1339,
appsec (1.773 ms) : 1749, 1797
.   : milestone, 1773,
iast (1.508 ms) : 1485, 1531
.   : milestone, 1508,
profiling (1.52 ms) : 1496, 1545
.   : milestone, 1520,
tracing (1.5 ms) : 1477, 1523
.   : milestone, 1500,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.341 ms [1.321 ms, 1.36 ms] -
appsec 1.745 ms [1.722 ms, 1.769 ms] 404.848 µs (30.2%)
iast 1.506 ms [1.482 ms, 1.529 ms] 165.254 µs (12.3%)
profiling 1.542 ms [1.517 ms, 1.567 ms] 201.668 µs (15.0%)
tracing 1.51 ms [1.487 ms, 1.534 ms] 169.667 µs (12.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.339 ms [1.32 ms, 1.358 ms] -
appsec 1.773 ms [1.749 ms, 1.797 ms] 434.016 µs (32.4%)
iast 1.508 ms [1.485 ms, 1.531 ms] 169.268 µs (12.6%)
profiling 1.52 ms [1.496 ms, 1.545 ms] 181.605 µs (13.6%)
tracing 1.5 ms [1.477 ms, 1.523 ms] 160.975 µs (12.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e78a54c6f0, baseline=1.32.0-SNAPSHOT~f19053f7e4
    dateFormat X
    axisFormat %s
section baseline
no_agent (360.443 µs) : 341, 380
.   : milestone, 360,
iast (466.048 µs) : 444, 489
.   : milestone, 466,
iast_FULL (532.332 µs) : 512, 553
.   : milestone, 532,
iast_GLOBAL (495.279 µs) : 474, 516
.   : milestone, 495,
iast_HARDCODED_SECRET_DISABLED (475.281 µs) : 455, 496
.   : milestone, 475,
iast_INACTIVE (445.859 µs) : 425, 467
.   : milestone, 446,
iast_TELEMETRY_OFF (465.754 µs) : 445, 486
.   : milestone, 466,
tracing (441.79 µs) : 421, 462
.   : milestone, 442,
section candidate
no_agent (358.395 µs) : 339, 378
.   : milestone, 358,
iast (466.155 µs) : 446, 486
.   : milestone, 466,
iast_FULL (535.33 µs) : 515, 556
.   : milestone, 535,
iast_GLOBAL (497.038 µs) : 476, 518
.   : milestone, 497,
iast_HARDCODED_SECRET_DISABLED (473.726 µs) : 453, 494
.   : milestone, 474,
iast_INACTIVE (448.357 µs) : 428, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (463.767 µs) : 443, 484
.   : milestone, 464,
tracing (445.633 µs) : 425, 466
.   : milestone, 446,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 360.443 µs [340.742 µs, 380.145 µs] -
iast 466.048 µs [443.533 µs, 488.563 µs] 105.605 µs (29.3%)
iast_FULL 532.332 µs [511.981 µs, 552.683 µs] 171.889 µs (47.7%)
iast_GLOBAL 495.279 µs [474.113 µs, 516.445 µs] 134.836 µs (37.4%)
iast_HARDCODED_SECRET_DISABLED 475.281 µs [454.65 µs, 495.913 µs] 114.838 µs (31.9%)
iast_INACTIVE 445.859 µs [424.931 µs, 466.787 µs] 85.416 µs (23.7%)
iast_TELEMETRY_OFF 465.754 µs [445.35 µs, 486.158 µs] 105.311 µs (29.2%)
tracing 441.79 µs [421.189 µs, 462.392 µs] 81.347 µs (22.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 358.395 µs [338.783 µs, 378.007 µs] -
iast 466.155 µs [445.93 µs, 486.379 µs] 107.76 µs (30.1%)
iast_FULL 535.33 µs [515.086 µs, 555.573 µs] 176.935 µs (49.4%)
iast_GLOBAL 497.038 µs [476.177 µs, 517.898 µs] 138.643 µs (38.7%)
iast_HARDCODED_SECRET_DISABLED 473.726 µs [453.006 µs, 494.445 µs] 115.331 µs (32.2%)
iast_INACTIVE 448.357 µs [427.597 µs, 469.117 µs] 89.962 µs (25.1%)
iast_TELEMETRY_OFF 463.767 µs [443.098 µs, 484.436 µs] 105.372 µs (29.4%)
tracing 445.633 µs [424.827 µs, 466.439 µs] 87.238 µs (24.3%)

@manuel-alvarez-alvarez manuel-alvarez-alvarez force-pushed the malvarez/iast-recover-sources-sampling branch 2 times, most recently from 7180447 to ee9940d Compare March 13, 2024 19:40
@manuel-alvarez-alvarez manuel-alvarez-alvarez marked this pull request as ready for review March 13, 2024 19:40
@smola smola changed the title Add request sampling back in http sources Add request sampling back in http sources in global context mode Mar 14, 2024
@manuel-alvarez-alvarez manuel-alvarez-alvarez changed the title Add request sampling back in http sources in global context mode Add request sampling back in http sources Mar 14, 2024
@manuel-alvarez-alvarez manuel-alvarez-alvarez force-pushed the malvarez/iast-recover-sources-sampling branch 5 times, most recently from daca71a to ba61dd0 Compare March 14, 2024 10:37
@manuel-alvarez-alvarez manuel-alvarez-alvarez force-pushed the malvarez/iast-recover-sources-sampling branch from ba61dd0 to bf8602f Compare March 14, 2024 10:57
@manuel-alvarez-alvarez manuel-alvarez-alvarez merged commit 8d4f9c2 into master Mar 14, 2024
130 checks passed
@manuel-alvarez-alvarez manuel-alvarez-alvarez deleted the malvarez/iast-recover-sources-sampling branch March 14, 2024 12:35
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smola smola smola approved these changes

@dougqh dougqh dougqh approved these changes

@jandro996 jandro996 Awaiting requested review from jandro996 jandro996 is a code owner automatically assigned from DataDog/asm-java

@am312 am312 Awaiting requested review from am312 am312 was automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: asm iast Application Security Management (IAST) run-tests: all Run all tests tag: performance Performance related changes
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@manuel-alvarez-alvarez @smola @dougqh