Capture exception snapshots only once an hour #6983

jpbempel · 2024-05-03T07:49:07Z

What Does This Do

Keep a timestamp associated to the fingerprint to know when last time we capture snapshots for this exception. Hardcoded once an hour will probably make it a config token for this later

Motivation

Additional Notes

Jira ticket: DEBUG-2068

ojung · 2024-05-06T11:16:12Z

...debugger/src/test/java/com/datadog/debugger/exception/ExceptionProbeInstrumentationTest.java

+  public void captureOncePerHour() throws Exception {
+    Config config = createConfig();
+    Clock clockMock = mock(Clock.class);
+    when(clockMock.instant()).thenReturn(Instant.now());


When using mockito to mock clocks I got weird flaky tests in the past about unfinished mockings. Have you ever had that? I decided to have a method with two signatures, one with clock and the other one without the clock using a "default clock" to fix it.

no problem with mocking so far 🤞

ojung · 2024-05-06T11:50:46Z

...agent/agent-debugger/src/main/java/com/datadog/debugger/exception/ExceptionProbeManager.java

@@ -59,7 +75,7 @@ public void createProbesForException(String fingerprint, StackTraceElement[] sta
  }

  void addFingerprint(String fingerprint) {
-    fingerprints.add(fingerprint);
+    fingerprints.put(fingerprint, Instant.ofEpochSecond(0));


There is also Instant.MIN: https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html#MIN

done, thanks 👌

ojung · 2024-05-06T11:51:37Z

...agent/agent-debugger/src/main/java/com/datadog/debugger/exception/ExceptionProbeManager.java

+    if (lastCapture == null) {
+      return false;
+    }
+    // only capture once an hour


Suggested change

// only capture once an hour

// only capture once a captureInterval

Or delete the comment all together.

ojung

Thanks 👍

Keep a timestamp associated to the fingerprint to know when last time we capture snapshots for this exception. Hardcoded once an hour will probably make it a config token for this later

remove comment

pr-commenter · 2024-05-07T09:21:42Z

Benchmarks

Startup

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-05-07T08:56:19	2024-05-07T09:03:11
git_branch	master	jpbempel/exception-capture-1h
git_commit_date	1715068887	1715071287
git_commit_sha	`6978c8e`	`d3f5525`
release_version	1.34.0-SNAPSHOT~6978c8eed3	1.34.0-SNAPSHOT~d3f5525450
start_time	2024-05-07T08:56:06	2024-05-07T09:02:58

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1715072939	1715072939
ci_job_id	505846840	505846840
ci_pipeline_id	33743192	33743192
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~d3f5525450, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.351 ms) : 1332, 1370
.   : milestone, 1351,
appsec (1.727 ms) : 1704, 1750
.   : milestone, 1727,
appsec_no_iast (1.728 ms) : 1703, 1752
.   : milestone, 1728,
iast (1.491 ms) : 1469, 1514
.   : milestone, 1491,
profiling (1.535 ms) : 1509, 1561
.   : milestone, 1535,
tracing (1.487 ms) : 1463, 1511
.   : milestone, 1487,
section candidate
no_agent (1.347 ms) : 1327, 1366
.   : milestone, 1347,
appsec (1.745 ms) : 1721, 1770
.   : milestone, 1745,
appsec_no_iast (1.726 ms) : 1701, 1750
.   : milestone, 1726,
iast (1.512 ms) : 1489, 1535
.   : milestone, 1512,
profiling (1.558 ms) : 1532, 1584
.   : milestone, 1558,
tracing (1.486 ms) : 1463, 1509
.   : milestone, 1486,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.332 ms, 1.37 ms]	-
appsec	1.727 ms [1.704 ms, 1.75 ms]	376.201 µs (27.8%)
appsec_no_iast	1.728 ms [1.703 ms, 1.752 ms]	377.017 µs (27.9%)
iast	1.491 ms [1.469 ms, 1.514 ms]	140.168 µs (10.4%)
profiling	1.535 ms [1.509 ms, 1.561 ms]	184.433 µs (13.7%)
tracing	1.487 ms [1.463 ms, 1.511 ms]	136.389 µs (10.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.347 ms [1.327 ms, 1.366 ms]	-
appsec	1.745 ms [1.721 ms, 1.77 ms]	398.83 µs (29.6%)
appsec_no_iast	1.726 ms [1.701 ms, 1.75 ms]	379.107 µs (28.2%)
iast	1.512 ms [1.489 ms, 1.535 ms]	165.626 µs (12.3%)
profiling	1.558 ms [1.532 ms, 1.584 ms]	211.393 µs (15.7%)
tracing	1.486 ms [1.463 ms, 1.509 ms]	139.535 µs (10.4%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.34.0-SNAPSHOT~d3f5525450, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (371.82 µs) : 352, 392
.   : milestone, 372,
iast (486.892 µs) : 466, 508
.   : milestone, 487,
iast_FULL (552.68 µs) : 532, 574
.   : milestone, 553,
iast_GLOBAL (499.895 µs) : 479, 520
.   : milestone, 500,
iast_HARDCODED_SECRET_DISABLED (481.198 µs) : 461, 502
.   : milestone, 481,
iast_INACTIVE (457.387 µs) : 435, 479
.   : milestone, 457,
iast_TELEMETRY_OFF (479.318 µs) : 459, 500
.   : milestone, 479,
tracing (455.008 µs) : 434, 476
.   : milestone, 455,
section candidate
no_agent (377.428 µs) : 358, 397
.   : milestone, 377,
iast (481.482 µs) : 461, 502
.   : milestone, 481,
iast_FULL (558.277 µs) : 537, 579
.   : milestone, 558,
iast_GLOBAL (509.135 µs) : 488, 530
.   : milestone, 509,
iast_HARDCODED_SECRET_DISABLED (487.207 µs) : 466, 509
.   : milestone, 487,
iast_INACTIVE (453.428 µs) : 433, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (485.347 µs) : 463, 508
.   : milestone, 485,
tracing (458.73 µs) : 437, 480
.   : milestone, 459,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	371.82 µs [351.661 µs, 391.979 µs]	-
iast	486.892 µs [466.216 µs, 507.568 µs]	115.072 µs (30.9%)
iast_FULL	552.68 µs [531.753 µs, 573.607 µs]	180.86 µs (48.6%)
iast_GLOBAL	499.895 µs [479.307 µs, 520.484 µs]	128.075 µs (34.4%)
iast_HARDCODED_SECRET_DISABLED	481.198 µs [460.516 µs, 501.879 µs]	109.378 µs (29.4%)
iast_INACTIVE	457.387 µs [435.412 µs, 479.361 µs]	85.566 µs (23.0%)
iast_TELEMETRY_OFF	479.318 µs [458.626 µs, 500.01 µs]	107.498 µs (28.9%)
tracing	455.008 µs [434.133 µs, 475.883 µs]	83.188 µs (22.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	377.428 µs [357.949 µs, 396.908 µs]	-
iast	481.482 µs [460.848 µs, 502.115 µs]	104.053 µs (27.6%)
iast_FULL	558.277 µs [537.339 µs, 579.215 µs]	180.849 µs (47.9%)
iast_GLOBAL	509.135 µs [488.069 µs, 530.2 µs]	131.706 µs (34.9%)
iast_HARDCODED_SECRET_DISABLED	487.207 µs [465.906 µs, 508.507 µs]	109.779 µs (29.1%)
iast_INACTIVE	453.428 µs [433.02 µs, 473.837 µs]	76.0 µs (20.1%)
iast_TELEMETRY_OFF	485.347 µs [462.881 µs, 507.814 µs]	107.919 µs (28.6%)
tracing	458.73 µs [437.389 µs, 480.071 µs]	81.302 µs (21.5%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/exception-capture-1h
git_commit_date	1715068887	1715071287
git_commit_sha	`6978c8e`	`d3f5525`
release_version	1.34.0-SNAPSHOT~6978c8eed3	1.34.0-SNAPSHOT~d3f5525450

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1715073569	1715073569
ci_job_id	505846841	505846841
ci_pipeline_id	33743192	33743192
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~d3f5525450, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.463 ms) : 1452, 1475
.   : milestone, 1463,
appsec (2.193 ms) : 2160, 2227
.   : milestone, 2193,
iast (1.881 ms) : 1846, 1916
.   : milestone, 1881,
iast_GLOBAL (1.912 ms) : 1878, 1947
.   : milestone, 1912,
profiling (1.851 ms) : 1816, 1885
.   : milestone, 1851,
tracing (1.836 ms) : 1804, 1868
.   : milestone, 1836,
section candidate
no_agent (1.46 ms) : 1448, 1471
.   : milestone, 1460,
appsec (2.205 ms) : 2170, 2239
.   : milestone, 2205,
iast (1.891 ms) : 1856, 1927
.   : milestone, 1891,
iast_GLOBAL (1.915 ms) : 1880, 1950
.   : milestone, 1915,
profiling (1.857 ms) : 1824, 1891
.   : milestone, 1857,
tracing (1.834 ms) : 1802, 1867
.   : milestone, 1834,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.463 ms [1.452 ms, 1.475 ms]	-
appsec	2.193 ms [2.16 ms, 2.227 ms]	729.767 µs (49.9%)
iast	1.881 ms [1.846 ms, 1.916 ms]	417.584 µs (28.5%)
iast_GLOBAL	1.912 ms [1.878 ms, 1.947 ms]	449.094 µs (30.7%)
profiling	1.851 ms [1.816 ms, 1.885 ms]	387.339 µs (26.5%)
tracing	1.836 ms [1.804 ms, 1.868 ms]	372.593 µs (25.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.46 ms [1.448 ms, 1.471 ms]	-
appsec	2.205 ms [2.17 ms, 2.239 ms]	744.725 µs (51.0%)
iast	1.891 ms [1.856 ms, 1.927 ms]	431.545 µs (29.6%)
iast_GLOBAL	1.915 ms [1.88 ms, 1.95 ms]	455.035 µs (31.2%)
profiling	1.857 ms [1.824 ms, 1.891 ms]	397.688 µs (27.2%)
tracing	1.834 ms [1.802 ms, 1.867 ms]	374.679 µs (25.7%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.34.0-SNAPSHOT~d3f5525450, baseline=1.34.0-SNAPSHOT~6978c8eed3
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.398 s) : 15398000, 15398000
.   : milestone, 15398000,
appsec (15.186 s) : 15186000, 15186000
.   : milestone, 15186000,
iast (18.837 s) : 18837000, 18837000
.   : milestone, 18837000,
iast_GLOBAL (17.774 s) : 17774000, 17774000
.   : milestone, 17774000,
profiling (15.361 s) : 15361000, 15361000
.   : milestone, 15361000,
tracing (15.01 s) : 15010000, 15010000
.   : milestone, 15010000,
section candidate
no_agent (14.856 s) : 14856000, 14856000
.   : milestone, 14856000,
appsec (15.224 s) : 15224000, 15224000
.   : milestone, 15224000,
iast (18.79 s) : 18790000, 18790000
.   : milestone, 18790000,
iast_GLOBAL (17.727 s) : 17727000, 17727000
.   : milestone, 17727000,
profiling (15.094 s) : 15094000, 15094000
.   : milestone, 15094000,
tracing (15.325 s) : 15325000, 15325000
.   : milestone, 15325000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.398 s [15.398 s, 15.398 s]	-
appsec	15.186 s [15.186 s, 15.186 s]	-212.0 ms (-1.4%)
iast	18.837 s [18.837 s, 18.837 s]	3.439 s (22.3%)
iast_GLOBAL	17.774 s [17.774 s, 17.774 s]	2.376 s (15.4%)
profiling	15.361 s [15.361 s, 15.361 s]	-37.0 ms (-0.2%)
tracing	15.01 s [15.01 s, 15.01 s]	-388.0 ms (-2.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.856 s [14.856 s, 14.856 s]	-
appsec	15.224 s [15.224 s, 15.224 s]	368.0 ms (2.5%)
iast	18.79 s [18.79 s, 18.79 s]	3.934 s (26.5%)
iast_GLOBAL	17.727 s [17.727 s, 17.727 s]	2.871 s (19.3%)
profiling	15.094 s [15.094 s, 15.094 s]	238.0 ms (1.6%)
tracing	15.325 s [15.325 s, 15.325 s]	469.0 ms (3.2%)

jpbempel added the comp: debugger Dynamic Instrumentation label May 3, 2024

jpbempel requested a review from a team as a code owner May 3, 2024 07:49

jpbempel requested review from shatzi and removed request for a team May 3, 2024 07:49

ojung reviewed May 6, 2024

View reviewed changes

ojung approved these changes May 6, 2024

View reviewed changes

jpbempel added 3 commits May 7, 2024 10:30

Capture exception snapshots only once an hour

290608e

Keep a timestamp associated to the fingerprint to know when last time we capture snapshots for this exception. Hardcoded once an hour will probably make it a config token for this later

Instant.MIN

c4e0247

remove comment

use seconds instead of millisecs with Instant.MIN

d3f5525

jpbempel force-pushed the jpbempel/exception-capture-1h branch from afc1d37 to d3f5525 Compare May 7, 2024 08:41

jpbempel merged commit 1e473e3 into master May 7, 2024
80 checks passed

jpbempel deleted the jpbempel/exception-capture-1h branch May 7, 2024 09:22

github-actions bot added this to the 1.34.0 milestone May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture exception snapshots only once an hour #6983

Capture exception snapshots only once an hour #6983

jpbempel commented May 3, 2024 •

edited by jira bot

ojung May 6, 2024

jpbempel May 6, 2024

ojung May 6, 2024

jpbempel May 6, 2024

ojung May 6, 2024

ojung May 6, 2024

jpbempel May 6, 2024

ojung left a comment

pr-commenter bot commented May 7, 2024

	// only capture once an hour
	// only capture once a captureInterval

Capture exception snapshots only once an hour #6983

Capture exception snapshots only once an hour #6983

Conversation

jpbempel commented May 3, 2024 • edited by jira bot

What Does This Do

Motivation

Additional Notes

ojung May 6, 2024

Choose a reason for hiding this comment

jpbempel May 6, 2024

Choose a reason for hiding this comment

ojung May 6, 2024

Choose a reason for hiding this comment

jpbempel May 6, 2024

Choose a reason for hiding this comment

ojung May 6, 2024

Choose a reason for hiding this comment

ojung May 6, 2024

Choose a reason for hiding this comment

jpbempel May 6, 2024

Choose a reason for hiding this comment

ojung left a comment

Choose a reason for hiding this comment

pr-commenter bot commented May 7, 2024

Benchmarks

Startup

Load

Parameters

Summary

Dacapo

Parameters

Summary

jpbempel commented May 3, 2024 •

edited by jira bot