Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved API Security schema computation performance #6765

Merged
merged 3 commits into from Mar 8, 2024
Merged

Improved API Security schema computation performance #6765

merged 3 commits into from Mar 8, 2024

Conversation

ValentinZakharov
Copy link
Contributor

@ValentinZakharov ValentinZakharov commented Mar 1, 2024 •
edited

What Does This Do

Computation of API Security schema moved to the end of request (after submit response)

Motivation

Computing schema imposed additional latency overhead.

Request latency with enabled API Security schema collection p90 (baseline)

  p50 p90
No API Sec 28,1655 32,90775
API Sec 28,9915 85,73242

API Sec (baseline) (1)

Request latency with enabled API Security schema collection p90 (candidate)

  p50 p90
No API Sec 28,5375 32,24396
API Sec 28,4485 32,06919

API Sec (candidate) (1)

Additional Notes

This is preparation work for improvement of API Security Sampling mechanism.

@ValentinZakharov ValentinZakharov added the comp: asm waf Application Security Management (WAF) label Mar 1, 2024
@ValentinZakharov ValentinZakharov self-assigned this Mar 1, 2024
@pr-commenter
Copy link

pr-commenter bot commented Mar 1, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master vzakharov/end_req_schema_extraction
git_commit_date 1709742193 1709893759
git_commit_sha d20df91 37e2a2f
release_version 1.32.0-SNAPSHOT~d20df91089 1.32.0-SNAPSHOT~37e2a2fae1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1709896932 1709896932
ci_job_id 454846642 454846642
ci_pipeline_id 29789846 29789846
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.082 s) : 0, 1082316
Total [baseline] (8.568 s) : 0, 8568331
Agent [candidate] (1.081 s) : 0, 1081285
Total [candidate] (8.574 s) : 0, 8574027
section iast
Agent [baseline] (1.21 s) : 0, 1210357
Total [baseline] (9.063 s) : 0, 9063202
Agent [candidate] (1.21 s) : 0, 1210053
Total [candidate] (9.057 s) : 0, 9056974
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.211 s) : 0, 1210897
Total [baseline] (9.06 s) : 0, 9059648
Agent [candidate] (1.211 s) : 0, 1210524
Total [candidate] (9.042 s) : 0, 9042162
section iast_TELEMETRY_OFF
Agent [baseline] (1.199 s) : 0, 1199063
Total [baseline] (9.033 s) : 0, 9032750
Agent [candidate] (1.207 s) : 0, 1207310
Total [candidate] (9.109 s) : 0, 9108774
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.082 s -
Agent iast 1.21 s 128.041 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 128.582 ms (11.9%)
Agent iast_TELEMETRY_OFF 1.199 s 116.747 ms (10.8%)
Total tracing 8.568 s -
Total iast 9.063 s 494.871 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.06 s 491.317 ms (5.7%)
Total iast_TELEMETRY_OFF 9.033 s 464.419 ms (5.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.081 s -
Agent iast 1.21 s 128.768 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 129.239 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.207 s 126.025 ms (11.7%)
Total tracing 8.574 s -
Total iast 9.057 s 482.947 ms (5.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.042 s 468.135 ms (5.5%)
Total iast_TELEMETRY_OFF 9.109 s 534.747 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (697.501 ms) : 0, 697501
BytebuddyAgent [candidate] (695.77 ms) : 0, 695770
GlobalTracer [baseline] (291.138 ms) : 0, 291138
GlobalTracer [candidate] (292.018 ms) : 0, 292018
AppSec [baseline] (50.658 ms) : 0, 50658
AppSec [candidate] (50.66 ms) : 0, 50660
Remote Config [baseline] (736.283 µs) : 0, 736
Remote Config [candidate] (729.966 µs) : 0, 730
Telemetry [baseline] (7.786 ms) : 0, 7786
Telemetry [candidate] (7.789 ms) : 0, 7789
section iast
BytebuddyAgent [baseline] (803.78 ms) : 0, 803780
BytebuddyAgent [candidate] (802.992 ms) : 0, 802992
GlobalTracer [baseline] (288.852 ms) : 0, 288852
GlobalTracer [candidate] (288.956 ms) : 0, 288956
AppSec [baseline] (52.348 ms) : 0, 52348
AppSec [candidate] (51.871 ms) : 0, 51871
Remote Config [baseline] (582.983 µs) : 0, 583
Remote Config [candidate] (578.349 µs) : 0, 578
Telemetry [baseline] (6.639 ms) : 0, 6639
Telemetry [candidate] (6.695 ms) : 0, 6695
IAST [baseline] (23.711 ms) : 0, 23711
IAST [candidate] (24.616 ms) : 0, 24616
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.74 ms) : 0, 802740
BytebuddyAgent [candidate] (802.843 ms) : 0, 802843
GlobalTracer [baseline] (290.346 ms) : 0, 290346
GlobalTracer [candidate] (289.075 ms) : 0, 289075
AppSec [baseline] (54.42 ms) : 0, 54420
AppSec [candidate] (53.136 ms) : 0, 53136
Remote Config [baseline] (597.475 µs) : 0, 597
Remote Config [candidate] (578.655 µs) : 0, 579
Telemetry [baseline] (6.779 ms) : 0, 6779
Telemetry [candidate] (6.696 ms) : 0, 6696
IAST [baseline] (21.716 ms) : 0, 21716
IAST [candidate] (23.816 ms) : 0, 23816
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (792.748 ms) : 0, 792748
BytebuddyAgent [candidate] (797.355 ms) : 0, 797355
GlobalTracer [baseline] (288.419 ms) : 0, 288419
GlobalTracer [candidate] (291.957 ms) : 0, 291957
AppSec [baseline] (51.358 ms) : 0, 51358
AppSec [candidate] (55.21 ms) : 0, 55210
Remote Config [baseline] (595.909 µs) : 0, 596
Remote Config [candidate] (589.051 µs) : 0, 589
Telemetry [baseline] (7.328 ms) : 0, 7328
Telemetry [candidate] (6.607 ms) : 0, 6607
IAST [baseline] (24.256 ms) : 0, 24256
IAST [candidate] (21.059 ms) : 0, 21059
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1089733
Total [baseline] (9.197 s) : 0, 9197378
Agent [candidate] (1.093 s) : 0, 1092733
Total [candidate] (9.208 s) : 0, 9207765
section appsec
Agent [baseline] (1.212 s) : 0, 1212092
Total [baseline] (9.344 s) : 0, 9343586
Agent [candidate] (1.221 s) : 0, 1220587
Total [candidate] (9.408 s) : 0, 9408280
section iast
Agent [baseline] (1.209 s) : 0, 1209177
Total [baseline] (9.403 s) : 0, 9402557
Agent [candidate] (1.209 s) : 0, 1209224
Total [candidate] (9.368 s) : 0, 9367648
section profiling
Agent [baseline] (1.28 s) : 0, 1280343
Total [baseline] (9.376 s) : 0, 9375786
Agent [candidate] (1.275 s) : 0, 1274587
Total [candidate] (9.399 s) : 0, 9398565
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent appsec 1.212 s 122.359 ms (11.2%)
Agent iast 1.209 s 119.444 ms (11.0%)
Agent profiling 1.28 s 190.61 ms (17.5%)
Total tracing 9.197 s -
Total appsec 9.344 s 146.208 ms (1.6%)
Total iast 9.403 s 205.179 ms (2.2%)
Total profiling 9.376 s 178.407 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.093 s -
Agent appsec 1.221 s 127.853 ms (11.7%)
Agent iast 1.209 s 116.491 ms (10.7%)
Agent profiling 1.275 s 181.854 ms (16.6%)
Total tracing 9.208 s -
Total appsec 9.408 s 200.515 ms (2.2%)
Total iast 9.368 s 159.882 ms (1.7%)
Total profiling 9.399 s 190.799 ms (2.1%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (702.423 ms) : 0, 702423
BytebuddyAgent [candidate] (703.602 ms) : 0, 703602
GlobalTracer [baseline] (293.251 ms) : 0, 293251
GlobalTracer [candidate] (294.488 ms) : 0, 294488
AppSec [baseline] (50.883 ms) : 0, 50883
AppSec [candidate] (51.4 ms) : 0, 51400
Remote Config [baseline] (732.278 µs) : 0, 732
Remote Config [candidate] (738.549 µs) : 0, 739
Telemetry [baseline] (7.77 ms) : 0, 7770
Telemetry [candidate] (7.772 ms) : 0, 7772
section appsec
BytebuddyAgent [baseline] (702.174 ms) : 0, 702174
BytebuddyAgent [candidate] (706.974 ms) : 0, 706974
GlobalTracer [baseline] (293.322 ms) : 0, 293322
GlobalTracer [candidate] (295.855 ms) : 0, 295855
AppSec [baseline] (156.296 ms) : 0, 156296
AppSec [candidate] (157.295 ms) : 0, 157295
Remote Config [baseline] (618.155 µs) : 0, 618
Remote Config [candidate] (616.505 µs) : 0, 617
Telemetry [baseline] (6.983 ms) : 0, 6983
Telemetry [candidate] (7.0 ms) : 0, 7000
IAST [baseline] (18.015 ms) : 0, 18015
IAST [candidate] (18.092 ms) : 0, 18092
section iast
BytebuddyAgent [baseline] (802.678 ms) : 0, 802678
BytebuddyAgent [candidate] (801.945 ms) : 0, 801945
GlobalTracer [baseline] (288.777 ms) : 0, 288777
GlobalTracer [candidate] (288.963 ms) : 0, 288963
AppSec [baseline] (54.025 ms) : 0, 54025
AppSec [candidate] (52.888 ms) : 0, 52888
Remote Config [baseline] (569.323 µs) : 0, 569
Remote Config [candidate] (581.118 µs) : 0, 581
Telemetry [baseline] (7.429 ms) : 0, 7429
Telemetry [candidate] (6.644 ms) : 0, 6644
IAST [baseline] (21.263 ms) : 0, 21263
IAST [candidate] (23.875 ms) : 0, 23875
section profiling
BytebuddyAgent [baseline] (693.921 ms) : 0, 693921
BytebuddyAgent [candidate] (690.299 ms) : 0, 690299
GlobalTracer [baseline] (375.231 ms) : 0, 375231
GlobalTracer [candidate] (373.84 ms) : 0, 373840
AppSec [baseline] (52.881 ms) : 0, 52881
AppSec [candidate] (52.836 ms) : 0, 52836
Remote Config [baseline] (764.032 µs) : 0, 764
Remote Config [candidate] (744.836 µs) : 0, 745
Telemetry [baseline] (7.482 ms) : 0, 7482
Telemetry [candidate] (7.455 ms) : 0, 7455
ProfilingAgent [baseline] (93.7 ms) : 0, 93700
ProfilingAgent [candidate] (93.295 ms) : 0, 93295
Profiling [baseline] (93.724 ms) : 0, 93724
Profiling [candidate] (93.319 ms) : 0, 93319

Load

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.271 µs) : 348, 389
.   : milestone, 368,
iast (478.301 µs) : 458, 499
.   : milestone, 478,
iast_FULL (540.143 µs) : 520, 561
.   : milestone, 540,
iast_GLOBAL (496.005 µs) : 475, 517
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (471.323 µs) : 451, 492
.   : milestone, 471,
iast_INACTIVE (447.912 µs) : 427, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (465.444 µs) : 445, 486
.   : milestone, 465,
tracing (443.477 µs) : 423, 464
.   : milestone, 443,
section candidate
no_agent (368.015 µs) : 348, 388
.   : milestone, 368,
iast (470.975 µs) : 451, 491
.   : milestone, 471,
iast_FULL (534.276 µs) : 514, 555
.   : milestone, 534,
iast_GLOBAL (502.11 µs) : 479, 525
.   : milestone, 502,
iast_HARDCODED_SECRET_DISABLED (474.666 µs) : 454, 495
.   : milestone, 475,
iast_INACTIVE (446.992 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (468.577 µs) : 448, 489
.   : milestone, 469,
tracing (445.316 µs) : 425, 466
.   : milestone, 445,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.271 µs [347.555 µs, 388.986 µs] -
iast 478.301 µs [457.582 µs, 499.019 µs] 110.03 µs (29.9%)
iast_FULL 540.143 µs [519.58 µs, 560.707 µs] 171.873 µs (46.7%)
iast_GLOBAL 496.005 µs [474.953 µs, 517.057 µs] 127.734 µs (34.7%)
iast_HARDCODED_SECRET_DISABLED 471.323 µs [450.544 µs, 492.103 µs] 103.053 µs (28.0%)
iast_INACTIVE 447.912 µs [427.194 µs, 468.631 µs] 79.642 µs (21.6%)
iast_TELEMETRY_OFF 465.444 µs [444.963 µs, 485.926 µs] 97.174 µs (26.4%)
tracing 443.477 µs [422.907 µs, 464.048 µs] 75.207 µs (20.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.015 µs [347.849 µs, 388.181 µs] -
iast 470.975 µs [450.6 µs, 491.349 µs] 102.96 µs (28.0%)
iast_FULL 534.276 µs [513.986 µs, 554.566 µs] 166.261 µs (45.2%)
iast_GLOBAL 502.11 µs [478.928 µs, 525.293 µs] 134.095 µs (36.4%)
iast_HARDCODED_SECRET_DISABLED 474.666 µs [453.972 µs, 495.359 µs] 106.651 µs (29.0%)
iast_INACTIVE 446.992 µs [425.589 µs, 468.395 µs] 78.977 µs (21.5%)
iast_TELEMETRY_OFF 468.577 µs [447.939 µs, 489.216 µs] 100.563 µs (27.3%)
tracing 445.316 µs [424.54 µs, 466.092 µs] 77.301 µs (21.0%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~37e2a2fae1, baseline=1.32.0-SNAPSHOT~d20df91089
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.364 ms) : 1345, 1383
.   : milestone, 1364,
appsec (1.762 ms) : 1738, 1785
.   : milestone, 1762,
iast (1.511 ms) : 1488, 1535
.   : milestone, 1511,
profiling (1.562 ms) : 1537, 1587
.   : milestone, 1562,
tracing (1.516 ms) : 1493, 1539
.   : milestone, 1516,
section candidate
no_agent (1.35 ms) : 1331, 1370
.   : milestone, 1350,
appsec (1.801 ms) : 1778, 1825
.   : milestone, 1801,
iast (1.533 ms) : 1509, 1556
.   : milestone, 1533,
profiling (1.536 ms) : 1511, 1561
.   : milestone, 1536,
tracing (1.511 ms) : 1488, 1535
.   : milestone, 1511,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.364 ms [1.345 ms, 1.383 ms] -
appsec 1.762 ms [1.738 ms, 1.785 ms] 397.725 µs (29.2%)
iast 1.511 ms [1.488 ms, 1.535 ms] 147.368 µs (10.8%)
profiling 1.562 ms [1.537 ms, 1.587 ms] 197.671 µs (14.5%)
tracing 1.516 ms [1.493 ms, 1.539 ms] 151.938 µs (11.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.35 ms [1.331 ms, 1.37 ms] -
appsec 1.801 ms [1.778 ms, 1.825 ms] 451.062 µs (33.4%)
iast 1.533 ms [1.509 ms, 1.556 ms] 182.283 µs (13.5%)
profiling 1.536 ms [1.511 ms, 1.561 ms] 185.625 µs (13.7%)
tracing 1.511 ms [1.488 ms, 1.535 ms] 160.99 µs (11.9%)

@ValentinZakharov ValentinZakharov force-pushed the vzakharov/end_req_schema_extraction branch from da0a108 to b18b49f Compare March 1, 2024 15:38
@ValentinZakharov ValentinZakharov marked this pull request as ready for review March 1, 2024 16:58
@ValentinZakharov ValentinZakharov requested a review from a team as a code owner March 1, 2024 16:58
jandro996
jandro996 reviewed Mar 5, 2024
View reviewed changes
dd-java-agent/appsec/src/main/java/com/datadog/appsec/gateway/GatewayBridge.java Outdated
extractSchema = requestSampler.sampleRequest();
}

if (!extractSchema) return;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO use always {} in if statements is more readable

@smola smola added the tag: performance Performance related changes label Mar 8, 2024
@ValentinZakharov ValentinZakharov changed the title API Security schema computes at the end of request Improved API Security schema computes performance Mar 8, 2024
@ValentinZakharov ValentinZakharov changed the title Improved API Security schema computes performance Improved API Security schema computation performance Mar 8, 2024
@ValentinZakharov ValentinZakharov merged commit b174715 into master Mar 8, 2024
79 checks passed
@ValentinZakharov ValentinZakharov deleted the vzakharov/end_req_schema_extraction branch March 8, 2024 13:04
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smola smola smola approved these changes

@jandro996 jandro996 jandro996 approved these changes

Assignees

@ValentinZakharov ValentinZakharov

Labels
comp: asm waf Application Security Management (WAF) tag: performance Performance related changes
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ValentinZakharov @smola @jandro996