Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Exception debugging instrumentation async #6829

Merged
merged 2 commits into from Mar 22, 2024
Merged

Make Exception debugging instrumentation async #6829

merged 2 commits into from Mar 22, 2024

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Mar 21, 2024 •
edited by jira bot

What Does This Do

Rename config option for exception debugging from
DD_DYNAMIC_INSTRUMENTATION_EXCEPTION_ENABLED to DD_EXCEPTION_DEBUGGING_ENABLED

Fix frame index following the change in Error Tracking UI

Motivation

Additional Notes

Jira ticket: DEBUG-2068

@jpbempel jpbempel added the comp: debugger Dynamic Instrumentation label Mar 21, 2024
@jpbempel jpbempel requested review from a team as code owners March 21, 2024 09:47
@jpbempel jpbempel requested review from shatzi, mcculls and nayeem-kamal and removed request for a team March 21, 2024 09:47
@pr-commenter
Copy link

pr-commenter bot commented Mar 21, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/async-instrumentation
git_commit_date 1711037161 1711044199
git_commit_sha 78b3d6c e53c4bc
release_version 1.32.0-SNAPSHOT~78b3d6c5d9 1.32.0-SNAPSHOT~e53c4bc9b7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1711047380 1711047380
ci_job_id 466301838 466301838
ci_pipeline_id 30573957 30573957
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 47 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast_HARDCODED_SECRET_DISABLED:Remote Config worse
[+11.551µs; +40.707µs] or [+2.080%; +7.332%]		 581.347µs 555.218µs
scenario:startup:petclinic:appsec:IAST better
[-1.987ms; -1.078ms] or [-10.174%; -5.520%]		 18.001ms 19.534ms
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1078395
Total [baseline] (8.57 s) : 0, 8570408
Agent [candidate] (1.08 s) : 0, 1080000
Total [candidate] (8.568 s) : 0, 8567789
section iast
Agent [baseline] (1.202 s) : 0, 1201603
Total [baseline] (9.038 s) : 0, 9038203
Agent [candidate] (1.221 s) : 0, 1220994
Total [candidate] (9.101 s) : 0, 9100668
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.211 s) : 0, 1211188
Total [baseline] (9.011 s) : 0, 9011185
Agent [candidate] (1.211 s) : 0, 1210643
Total [candidate] (9.056 s) : 0, 9055853
section iast_TELEMETRY_OFF
Agent [baseline] (1.207 s) : 0, 1206924
Total [baseline] (9.127 s) : 0, 9127244
Agent [candidate] (1.202 s) : 0, 1201735
Total [candidate] (9.043 s) : 0, 9043142
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent iast 1.202 s 123.208 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 132.793 ms (12.3%)
Agent iast_TELEMETRY_OFF 1.207 s 128.529 ms (11.9%)
Total tracing 8.57 s -
Total iast 9.038 s 467.795 ms (5.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.011 s 440.777 ms (5.1%)
Total iast_TELEMETRY_OFF 9.127 s 556.836 ms (6.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.08 s -
Agent iast 1.221 s 140.994 ms (13.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 130.643 ms (12.1%)
Agent iast_TELEMETRY_OFF 1.202 s 121.735 ms (11.3%)
Total tracing 8.568 s -
Total iast 9.101 s 532.879 ms (6.2%)
Total iast_HARDCODED_SECRET_DISABLED 9.056 s 488.064 ms (5.7%)
Total iast_TELEMETRY_OFF 9.043 s 475.353 ms (5.5%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (687.414 ms) : 0, 687414
BytebuddyAgent [candidate] (696.24 ms) : 0, 696240
GlobalTracer [baseline] (298.679 ms) : 0, 298679
GlobalTracer [candidate] (291.837 ms) : 0, 291837
AppSec [baseline] (49.496 ms) : 0, 49496
AppSec [candidate] (49.217 ms) : 0, 49217
Remote Config [baseline] (1.086 ms) : 0, 1086
Remote Config [candidate] (1.085 ms) : 0, 1085
Telemetry [baseline] (7.435 ms) : 0, 7435
Telemetry [candidate] (7.352 ms) : 0, 7352
section iast
BytebuddyAgent [baseline] (796.799 ms) : 0, 796799
BytebuddyAgent [candidate] (811.732 ms) : 0, 811732
GlobalTracer [baseline] (288.029 ms) : 0, 288029
GlobalTracer [candidate] (292.547 ms) : 0, 292547
AppSec [baseline] (48.878 ms) : 0, 48878
AppSec [candidate] (50.726 ms) : 0, 50726
IAST [baseline] (25.564 ms) : 0, 25564
IAST [candidate] (23.965 ms) : 0, 23965
Remote Config [baseline] (575.677 µs) : 0, 576
Remote Config [candidate] (571.398 µs) : 0, 571
Telemetry [baseline] (7.35 ms) : 0, 7350
Telemetry [candidate] (6.68 ms) : 0, 6680
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (803.493 ms) : 0, 803493
BytebuddyAgent [candidate] (803.872 ms) : 0, 803872
GlobalTracer [baseline] (290.891 ms) : 0, 290891
GlobalTracer [candidate] (290.698 ms) : 0, 290698
AppSec [baseline] (49.269 ms) : 0, 49269
AppSec [candidate] (50.755 ms) : 0, 50755
IAST [baseline] (25.886 ms) : 0, 25886
IAST [candidate] (21.026 ms) : 0, 21026
Remote Config [baseline] (555.218 µs) : 0, 555
Remote Config [candidate] (581.347 µs) : 0, 581
Telemetry [baseline] (6.479 ms) : 0, 6479
Telemetry [candidate] (9.109 ms) : 0, 9109
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (799.365 ms) : 0, 799365
BytebuddyAgent [candidate] (795.286 ms) : 0, 795286
GlobalTracer [baseline] (291.222 ms) : 0, 291222
GlobalTracer [candidate] (290.573 ms) : 0, 290573
AppSec [baseline] (51.27 ms) : 0, 51270
AppSec [candidate] (50.2 ms) : 0, 50200
IAST [baseline] (23.221 ms) : 0, 23221
IAST [candidate] (24.128 ms) : 0, 24128
Remote Config [baseline] (593.977 µs) : 0, 594
Remote Config [candidate] (605.521 µs) : 0, 606
Telemetry [baseline] (6.688 ms) : 0, 6688
Telemetry [candidate] (6.613 ms) : 0, 6613
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1078322
Total [baseline] (9.182 s) : 0, 9181902
Agent [candidate] (1.085 s) : 0, 1085185
Total [candidate] (9.138 s) : 0, 9138175
section appsec
Agent [baseline] (1.21 s) : 0, 1209720
Total [baseline] (9.307 s) : 0, 9306897
Agent [candidate] (1.209 s) : 0, 1209080
Total [candidate] (9.282 s) : 0, 9282425
section iast
Agent [baseline] (1.206 s) : 0, 1206244
Total [baseline] (9.362 s) : 0, 9361636
Agent [candidate] (1.208 s) : 0, 1208309
Total [candidate] (9.365 s) : 0, 9365265
section profiling
Agent [baseline] (1.275 s) : 0, 1275419
Total [baseline] (9.388 s) : 0, 9387944
Agent [candidate] (1.276 s) : 0, 1275930
Total [candidate] (9.396 s) : 0, 9395961
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent appsec 1.21 s 131.398 ms (12.2%)
Agent iast 1.206 s 127.921 ms (11.9%)
Agent profiling 1.275 s 197.097 ms (18.3%)
Total tracing 9.182 s -
Total appsec 9.307 s 124.995 ms (1.4%)
Total iast 9.362 s 179.733 ms (2.0%)
Total profiling 9.388 s 206.042 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.085 s -
Agent appsec 1.209 s 123.895 ms (11.4%)
Agent iast 1.208 s 123.125 ms (11.3%)
Agent profiling 1.276 s 190.746 ms (17.6%)
Total tracing 9.138 s -
Total appsec 9.282 s 144.25 ms (1.6%)
Total iast 9.365 s 227.089 ms (2.5%)
Total profiling 9.396 s 257.785 ms (2.8%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.824 ms) : 0, 686824
BytebuddyAgent [candidate] (699.966 ms) : 0, 699966
GlobalTracer [baseline] (298.917 ms) : 0, 298917
GlobalTracer [candidate] (292.705 ms) : 0, 292705
AppSec [baseline] (49.613 ms) : 0, 49613
AppSec [candidate] (49.155 ms) : 0, 49155
Remote Config [baseline] (1.061 ms) : 0, 1061
Remote Config [candidate] (1.088 ms) : 0, 1088
Telemetry [baseline] (7.543 ms) : 0, 7543
Telemetry [candidate] (7.503 ms) : 0, 7503
section appsec
BytebuddyAgent [baseline] (699.873 ms) : 0, 699873
BytebuddyAgent [candidate] (701.038 ms) : 0, 701038
GlobalTracer [baseline] (293.959 ms) : 0, 293959
GlobalTracer [candidate] (294.008 ms) : 0, 294008
AppSec [baseline] (154.511 ms) : 0, 154511
AppSec [candidate] (153.907 ms) : 0, 153907
Remote Config [baseline] (616.605 µs) : 0, 617
Remote Config [candidate] (623.376 µs) : 0, 623
Telemetry [baseline] (6.857 ms) : 0, 6857
Telemetry [candidate] (6.917 ms) : 0, 6917
IAST [baseline] (19.534 ms) : 0, 19534
IAST [candidate] (18.001 ms) : 0, 18001
section iast
BytebuddyAgent [baseline] (799.939 ms) : 0, 799939
BytebuddyAgent [candidate] (802.451 ms) : 0, 802451
GlobalTracer [baseline] (289.629 ms) : 0, 289629
GlobalTracer [candidate] (290.265 ms) : 0, 290265
AppSec [baseline] (50.512 ms) : 0, 50512
AppSec [candidate] (49.615 ms) : 0, 49615
Remote Config [baseline] (571.696 µs) : 0, 572
Remote Config [candidate] (570.861 µs) : 0, 571
Telemetry [baseline] (6.607 ms) : 0, 6607
Telemetry [candidate] (6.571 ms) : 0, 6571
IAST [baseline] (24.587 ms) : 0, 24587
IAST [candidate] (24.522 ms) : 0, 24522
section profiling
BytebuddyAgent [baseline] (681.008 ms) : 0, 681008
BytebuddyAgent [candidate] (689.811 ms) : 0, 689811
GlobalTracer [baseline] (382.915 ms) : 0, 382915
GlobalTracer [candidate] (376.573 ms) : 0, 376573
AppSec [baseline] (50.369 ms) : 0, 50369
AppSec [candidate] (50.107 ms) : 0, 50107
Remote Config [baseline] (885.048 µs) : 0, 885
Remote Config [candidate] (860.033 µs) : 0, 860
Telemetry [baseline] (7.49 ms) : 0, 7490
Telemetry [candidate] (7.377 ms) : 0, 7377
ProfilingAgent [baseline] (96.541 ms) : 0, 96541
ProfilingAgent [candidate] (95.078 ms) : 0, 95078
Profiling [baseline] (96.564 ms) : 0, 96564
Profiling [candidate] (95.101 ms) : 0, 95101

Load

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9
    dateFormat X
    axisFormat %s
section baseline
no_agent (361.707 µs) : 342, 382
.   : milestone, 362,
iast (472.735 µs) : 452, 493
.   : milestone, 473,
iast_FULL (544.998 µs) : 525, 565
.   : milestone, 545,
iast_GLOBAL (487.249 µs) : 467, 508
.   : milestone, 487,
iast_HARDCODED_SECRET_DISABLED (473.55 µs) : 453, 494
.   : milestone, 474,
iast_INACTIVE (444.07 µs) : 424, 464
.   : milestone, 444,
iast_TELEMETRY_OFF (471.861 µs) : 450, 493
.   : milestone, 472,
tracing (442.796 µs) : 422, 464
.   : milestone, 443,
section candidate
no_agent (362.881 µs) : 343, 383
.   : milestone, 363,
iast (484.598 µs) : 464, 505
.   : milestone, 485,
iast_FULL (540.189 µs) : 519, 561
.   : milestone, 540,
iast_GLOBAL (495.967 µs) : 475, 517
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (472.917 µs) : 452, 493
.   : milestone, 473,
iast_INACTIVE (456.233 µs) : 435, 478
.   : milestone, 456,
iast_TELEMETRY_OFF (462.383 µs) : 442, 483
.   : milestone, 462,
tracing (443.164 µs) : 422, 464
.   : milestone, 443,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 361.707 µs [341.779 µs, 381.636 µs] -
iast 472.735 µs [452.196 µs, 493.275 µs] 111.028 µs (30.7%)
iast_FULL 544.998 µs [524.602 µs, 565.394 µs] 183.291 µs (50.7%)
iast_GLOBAL 487.249 µs [466.934 µs, 507.564 µs] 125.542 µs (34.7%)
iast_HARDCODED_SECRET_DISABLED 473.55 µs [452.869 µs, 494.231 µs] 111.842 µs (30.9%)
iast_INACTIVE 444.07 µs [423.844 µs, 464.296 µs] 82.363 µs (22.8%)
iast_TELEMETRY_OFF 471.861 µs [450.28 µs, 493.441 µs] 110.153 µs (30.5%)
tracing 442.796 µs [421.803 µs, 463.79 µs] 81.089 µs (22.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.881 µs [342.753 µs, 383.009 µs] -
iast 484.598 µs [463.791 µs, 505.404 µs] 121.717 µs (33.5%)
iast_FULL 540.189 µs [519.342 µs, 561.036 µs] 177.308 µs (48.9%)
iast_GLOBAL 495.967 µs [474.965 µs, 516.969 µs] 133.086 µs (36.7%)
iast_HARDCODED_SECRET_DISABLED 472.917 µs [452.44 µs, 493.394 µs] 110.036 µs (30.3%)
iast_INACTIVE 456.233 µs [434.558 µs, 477.908 µs] 93.352 µs (25.7%)
iast_TELEMETRY_OFF 462.383 µs [442.222 µs, 482.545 µs] 99.502 µs (27.4%)
tracing 443.164 µs [422.265 µs, 464.063 µs] 80.283 µs (22.1%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~e53c4bc9b7, baseline=1.32.0-SNAPSHOT~78b3d6c5d9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.359 ms) : 1340, 1378
.   : milestone, 1359,
appsec (1.786 ms) : 1763, 1809
.   : milestone, 1786,
iast (1.494 ms) : 1471, 1518
.   : milestone, 1494,
profiling (1.541 ms) : 1517, 1565
.   : milestone, 1541,
tracing (1.512 ms) : 1488, 1536
.   : milestone, 1512,
section candidate
no_agent (1.349 ms) : 1330, 1368
.   : milestone, 1349,
appsec (1.789 ms) : 1766, 1812
.   : milestone, 1789,
iast (1.538 ms) : 1515, 1561
.   : milestone, 1538,
profiling (1.516 ms) : 1492, 1539
.   : milestone, 1516,
tracing (1.505 ms) : 1481, 1528
.   : milestone, 1505,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.359 ms [1.34 ms, 1.378 ms] -
appsec 1.786 ms [1.763 ms, 1.809 ms] 427.416 µs (31.5%)
iast 1.494 ms [1.471 ms, 1.518 ms] 135.355 µs (10.0%)
profiling 1.541 ms [1.517 ms, 1.565 ms] 182.016 µs (13.4%)
tracing 1.512 ms [1.488 ms, 1.536 ms] 153.169 µs (11.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.349 ms [1.33 ms, 1.368 ms] -
appsec 1.789 ms [1.766 ms, 1.812 ms] 439.403 µs (32.6%)
iast 1.538 ms [1.515 ms, 1.561 ms] 188.243 µs (14.0%)
profiling 1.516 ms [1.492 ms, 1.539 ms] 166.139 µs (12.3%)
tracing 1.505 ms [1.481 ms, 1.528 ms] 155.305 µs (11.5%)

@jpbempel jpbempel force-pushed the jpbempel/async-instrumentation branch from 5d73b78 to 1679925 Compare March 21, 2024 13:24
@jpbempel
Make Exception debugging instrumentation async
1c59b38 
Rename config option for exception debugging from
DD_DYNAMIC_INSTRUMENTATION_EXCEPTION_ENABLED to
DD_EXCEPTION_DEBUGGING_ENABLED

Fix frame index following the change in Error Tracking UI
@jpbempel jpbempel force-pushed the jpbempel/async-instrumentation branch from 1679925 to 1c59b38 Compare March 21, 2024 13:34
shatzi
shatzi approved these changes Mar 21, 2024
View reviewed changes
...nt/agent-debugger/src/main/java/com/datadog/debugger/exception/DefaultExceptionDebugger.java
}
}

private void applyExceptionConfiguration(String fingerprint) {
configurationUpdater.accept(EXCEPTION, exceptionProbeManager.getProbes());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add check: if (exceptionProbeManager.isAlreadyInstrumented(fingerprint)) { to avoid race condition?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm no because IsAlreadyInstrumented rely on the fact we add the fingerprint which is now done after applying the config...

dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/util/ExceptionHelper.java Outdated
@@ -73,7 +73,7 @@ public static Throwable getInnerMostThrowable(Throwable t) {

public static StackTraceElement[] flattenStackTrace(Throwable t) {
List<StackTraceElement> result = new ArrayList<>();
result.add(null); // add a stack frame representing the exception message
// result.add(null); // add a stack frame representing the exception message
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right!

...t/agent-debugger/src/test/java/com/datadog/debugger/exception/ExceptionProbeManagerTest.java Outdated
@@ -52,4 +53,19 @@ void filterAllFrames() {
exceptionProbeManager.createProbesForException("", exception.getStackTrace());
assertEquals(0, exceptionProbeManager.getProbes().size());
}

static void waitForInstrumentation(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a great utility function to have

assertWithTimeout(()->exceptionProbeManager.isAlreadyInstrumented(fingerprint), Duration.OfSeconds(30))

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@jpbempel jpbempel merged commit 46fdeca into master Mar 22, 2024
80 checks passed
@jpbempel jpbempel deleted the jpbempel/async-instrumentation branch March 22, 2024 09:26
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shatzi shatzi shatzi approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@jpbempel @shatzi