Add support for sending OOME events #7253
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jbachorik
changed the title
jbachorik
force-pushed
the
jb/oome_notifier
branch
2 times, most recently
from
5773294
to
6a15242
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 47 metrics, 15 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1074358
Total [baseline] (8.572 s) : 0, 8572193
Agent [candidate] (1.062 s) : 0, 1062361
Total [candidate] (8.554 s) : 0, 8554317
section iast
Agent [baseline] (1.182 s) : 0, 1182307
Total [baseline] (9.033 s) : 0, 9033476
Agent [candidate] (1.169 s) : 0, 1169161
Total [candidate] (8.993 s) : 0, 8992622
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.182 s) : 0, 1181900
Total [baseline] (8.972 s) : 0, 8971519
Agent [candidate] (1.178 s) : 0, 1178314
Total [candidate] (8.985 s) : 0, 8984813
section iast_TELEMETRY_OFF
Agent [baseline] (1.187 s) : 0, 1187254
Total [baseline] (9.075 s) : 0, 9074521
Agent [candidate] (1.174 s) : 0, 1174445
Total [candidate] (9.042 s) : 0, 9041535
gantt
title insecure-bank - break down per module: candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.241 ms) : 0, 673241
BytebuddyAgent [candidate] (665.891 ms) : 0, 665891
GlobalTracer [baseline] (307.12 ms) : 0, 307120
GlobalTracer [candidate] (303.515 ms) : 0, 303515
AppSec [baseline] (50.67 ms) : 0, 50670
AppSec [candidate] (50.221 ms) : 0, 50221
Remote Config [baseline] (738.01 µs) : 0, 738
Remote Config [candidate] (692.362 µs) : 0, 692
Telemetry [baseline] (7.752 ms) : 0, 7752
Telemetry [candidate] (7.627 ms) : 0, 7627
section iast
BytebuddyAgent [baseline] (787.903 ms) : 0, 787903
BytebuddyAgent [candidate] (779.871 ms) : 0, 779871
GlobalTracer [baseline] (297.236 ms) : 0, 297236
GlobalTracer [candidate] (293.43 ms) : 0, 293430
AppSec [baseline] (47.706 ms) : 0, 47706
AppSec [candidate] (46.925 ms) : 0, 46925
Remote Config [baseline] (595.591 µs) : 0, 596
Remote Config [candidate] (635.074 µs) : 0, 635
Telemetry [baseline] (7.806 ms) : 0, 7806
Telemetry [candidate] (6.981 ms) : 0, 6981
IAST [baseline] (27.538 ms) : 0, 27538
IAST [candidate] (27.998 ms) : 0, 27998
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (787.974 ms) : 0, 787974
BytebuddyAgent [candidate] (785.959 ms) : 0, 785959
GlobalTracer [baseline] (297.682 ms) : 0, 297682
GlobalTracer [candidate] (295.963 ms) : 0, 295963
AppSec [baseline] (47.685 ms) : 0, 47685
AppSec [candidate] (47.832 ms) : 0, 47832
Remote Config [baseline] (618.232 µs) : 0, 618
Remote Config [candidate] (650.355 µs) : 0, 650
Telemetry [baseline] (7.069 ms) : 0, 7069
Telemetry [candidate] (7.849 ms) : 0, 7849
IAST [baseline] (27.331 ms) : 0, 27331
IAST [candidate] (26.606 ms) : 0, 26606
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (791.267 ms) : 0, 791267
BytebuddyAgent [candidate] (783.596 ms) : 0, 783596
GlobalTracer [baseline] (298.899 ms) : 0, 298899
GlobalTracer [candidate] (295.296 ms) : 0, 295296
AppSec [baseline] (47.603 ms) : 0, 47603
AppSec [candidate] (47.564 ms) : 0, 47564
Remote Config [baseline] (636.65 µs) : 0, 637
Remote Config [candidate] (607.647 µs) : 0, 608
Telemetry [baseline] (6.978 ms) : 0, 6978
Telemetry [candidate] (6.909 ms) : 0, 6909
IAST [baseline] (28.263 ms) : 0, 28263
IAST [candidate] (27.047 ms) : 0, 27047
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065810
Total [baseline] (10.349 s) : 0, 10348945
Agent [candidate] (1.072 s) : 0, 1072435
Total [candidate] (10.338 s) : 0, 10338158
section appsec
Agent [baseline] (1.186 s) : 0, 1185526
Total [baseline] (10.479 s) : 0, 10479290
Agent [candidate] (1.182 s) : 0, 1182194
Total [candidate] (10.514 s) : 0, 10514242
section iast
Agent [baseline] (1.183 s) : 0, 1182779
Total [baseline] (10.757 s) : 0, 10756737
Agent [candidate] (1.171 s) : 0, 1170991
Total [candidate] (10.819 s) : 0, 10819213
section profiling
Agent [baseline] (1.269 s) : 0, 1268510
Total [baseline] (10.667 s) : 0, 10667327
Agent [candidate] (1.262 s) : 0, 1262274
Total [candidate] (10.62 s) : 0, 10619799
gantt
title petclinic - break down per module: candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (667.273 ms) : 0, 667273
BytebuddyAgent [candidate] (672.779 ms) : 0, 672779
GlobalTracer [baseline] (305.151 ms) : 0, 305151
GlobalTracer [candidate] (305.948 ms) : 0, 305948
AppSec [baseline] (50.431 ms) : 0, 50431
AppSec [candidate] (50.611 ms) : 0, 50611
Remote Config [baseline] (750.105 µs) : 0, 750
Remote Config [candidate] (686.673 µs) : 0, 687
Telemetry [baseline] (7.654 ms) : 0, 7654
Telemetry [candidate] (7.595 ms) : 0, 7595
section appsec
BytebuddyAgent [baseline] (677.791 ms) : 0, 677791
BytebuddyAgent [candidate] (676.77 ms) : 0, 676770
GlobalTracer [baseline] (298.88 ms) : 0, 298880
GlobalTracer [candidate] (297.561 ms) : 0, 297561
AppSec [baseline] (154.037 ms) : 0, 154037
AppSec [candidate] (154.005 ms) : 0, 154005
Remote Config [baseline] (634.14 µs) : 0, 634
Remote Config [candidate] (645.509 µs) : 0, 646
Telemetry [baseline] (8.663 ms) : 0, 8663
Telemetry [candidate] (7.792 ms) : 0, 7792
IAST [baseline] (21.728 ms) : 0, 21728
IAST [candidate] (21.755 ms) : 0, 21755
section iast
BytebuddyAgent [baseline] (789.204 ms) : 0, 789204
BytebuddyAgent [candidate] (780.082 ms) : 0, 780082
GlobalTracer [baseline] (297.114 ms) : 0, 297114
GlobalTracer [candidate] (293.788 ms) : 0, 293788
AppSec [baseline] (47.816 ms) : 0, 47816
AppSec [candidate] (47.131 ms) : 0, 47131
Remote Config [baseline] (580.641 µs) : 0, 581
Remote Config [candidate] (621.879 µs) : 0, 622
Telemetry [baseline] (7.024 ms) : 0, 7024
Telemetry [candidate] (8.545 ms) : 0, 8545
IAST [baseline] (27.491 ms) : 0, 27491
IAST [candidate] (27.497 ms) : 0, 27497
section profiling
BytebuddyAgent [baseline] (665.725 ms) : 0, 665725
BytebuddyAgent [candidate] (663.446 ms) : 0, 663446
GlobalTracer [baseline] (389.492 ms) : 0, 389492
GlobalTracer [candidate] (386.492 ms) : 0, 386492
AppSec [baseline] (52.08 ms) : 0, 52080
AppSec [candidate] (51.35 ms) : 0, 51350
Remote Config [baseline] (661.374 µs) : 0, 661
Remote Config [candidate] (742.211 µs) : 0, 742
Telemetry [baseline] (7.472 ms) : 0, 7472
Telemetry [candidate] (7.389 ms) : 0, 7389
ProfilingAgent [baseline] (95.826 ms) : 0, 95826
ProfilingAgent [candidate] (95.79 ms) : 0, 95790
Profiling [baseline] (95.851 ms) : 0, 95851
Profiling [candidate] (95.815 ms) : 0, 95815
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section baseline
no_agent (365.431 µs) : 345, 386
. : milestone, 365,
iast (479.781 µs) : 459, 501
. : milestone, 480,
iast_FULL (547.824 µs) : 527, 569
. : milestone, 548,
iast_GLOBAL (506.587 µs) : 485, 528
. : milestone, 507,
iast_HARDCODED_SECRET_DISABLED (484.691 µs) : 463, 506
. : milestone, 485,
iast_INACTIVE (459.159 µs) : 438, 481
. : milestone, 459,
iast_TELEMETRY_OFF (467.226 µs) : 446, 489
. : milestone, 467,
tracing (436.626 µs) : 417, 457
. : milestone, 437,
section candidate
no_agent (372.889 µs) : 353, 392
. : milestone, 373,
iast (484.26 µs) : 463, 506
. : milestone, 484,
iast_FULL (551.934 µs) : 531, 573
. : milestone, 552,
iast_GLOBAL (504.637 µs) : 484, 526
. : milestone, 505,
iast_HARDCODED_SECRET_DISABLED (484.293 µs) : 463, 505
. : milestone, 484,
iast_INACTIVE (452.621 µs) : 432, 474
. : milestone, 453,
iast_TELEMETRY_OFF (471.456 µs) : 450, 493
. : milestone, 471,
tracing (448.279 µs) : 427, 470
. : milestone, 448,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section baseline
no_agent (1.339 ms) : 1320, 1359
. : milestone, 1339,
appsec (1.744 ms) : 1721, 1767
. : milestone, 1744,
appsec_no_iast (1.734 ms) : 1710, 1758
. : milestone, 1734,
iast (1.472 ms) : 1450, 1494
. : milestone, 1472,
profiling (1.499 ms) : 1475, 1522
. : milestone, 1499,
tracing (1.468 ms) : 1444, 1491
. : milestone, 1468,
section candidate
no_agent (1.347 ms) : 1328, 1367
. : milestone, 1347,
appsec (1.713 ms) : 1688, 1738
. : milestone, 1713,
appsec_no_iast (1.711 ms) : 1686, 1736
. : milestone, 1711,
iast (1.472 ms) : 1449, 1495
. : milestone, 1472,
profiling (1.472 ms) : 1447, 1497
. : milestone, 1472,
tracing (1.467 ms) : 1443, 1491
. : milestone, 1467,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section baseline
no_agent (1.461 ms) : 1449, 1472
. : milestone, 1461,
appsec (2.213 ms) : 2177, 2249
. : milestone, 2213,
iast (1.973 ms) : 1930, 2016
. : milestone, 1973,
iast_GLOBAL (2.017 ms) : 1973, 2061
. : milestone, 2017,
profiling (1.865 ms) : 1829, 1901
. : milestone, 1865,
tracing (1.848 ms) : 1814, 1883
. : milestone, 1848,
section candidate
no_agent (1.455 ms) : 1444, 1467
. : milestone, 1455,
appsec (2.189 ms) : 2155, 2223
. : milestone, 2189,
iast (1.963 ms) : 1921, 2005
. : milestone, 1963,
iast_GLOBAL (1.988 ms) : 1947, 2028
. : milestone, 1988,
profiling (2.308 ms) : 2132, 2484
. : milestone, 2308,
tracing (1.819 ms) : 1788, 1851
. : milestone, 1819,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.36.0-SNAPSHOT~838fd83bff, baseline=1.37.0-SNAPSHOT~ffe2b6b64d
dateFormat X
axisFormat %s
section baseline
no_agent (15.0 s) : 15000000, 15000000
. : milestone, 15000000,
appsec (15.062 s) : 15062000, 15062000
. : milestone, 15062000,
iast (18.856 s) : 18856000, 18856000
. : milestone, 18856000,
iast_GLOBAL (17.88 s) : 17880000, 17880000
. : milestone, 17880000,
profiling (15.59 s) : 15590000, 15590000
. : milestone, 15590000,
tracing (14.874 s) : 14874000, 14874000
. : milestone, 14874000,
section candidate
no_agent (15.461 s) : 15461000, 15461000
. : milestone, 15461000,
appsec (15.081 s) : 15081000, 15081000
. : milestone, 15081000,
iast (18.843 s) : 18843000, 18843000
. : milestone, 18843000,
iast_GLOBAL (17.911 s) : 17911000, 17911000
. : milestone, 17911000,
profiling (15.093 s) : 15093000, 15093000
. : milestone, 15093000,
tracing (15.059 s) : 15059000, 15059000
. : milestone, 15059000,
|
jbachorik
force-pushed
the
jb/oome_notifier
branch
from
6a15242
to
e9de765
Compare
richardstartin
approved these changes
Jul 1, 2024
jbachorik
force-pushed
the
jb/oome_notifier
branch
3 times, most recently
from
ac6f724
to
cfa6d4e
Compare
jbachorik
force-pushed
the
jb/oome_notifier
branch
from
cfa6d4e
to
838fd83
Compare
| @@ -91,6 +91,7 @@ public CrashUploader() { | |||
|
|
|||
| ConfigProvider configProvider = config.configProvider(); | |||
|
|
|||
| System.out.println("===> telemetryUrl: " + telemetryUrl); | |||
Comment on lines
+100
to
+113
| public void recordEvent( | ||
| String type, String source, String eventName, String message, String... tags) { | ||
| Event.AlertType alertType = Event.AlertType.valueOf(type.toUpperCase()); | ||
| log.debug( | ||
| "Recording event: {} - {} - {} - {} [{}]", alertType, source, eventName, message, tags); | ||
| Event.Builder eventBuilder = | ||
| Event.builder() | ||
| .withTitle(eventName) | ||
| .withText(message) | ||
| .withSourceTypeName(source) | ||
| .withDate(System.currentTimeMillis()) | ||
| .withAlertType(alertType); | ||
| connection.statsd.recordEvent(eventBuilder.build(), tagMapping.apply(tags)); | ||
| } |
There was a problem hiding this comment.
Is this new method be part of the StatsD client API?
I would rather not expose the :communication implementation and revert it back to package private.
There was a problem hiding this comment.
I agree, this method should really have been added to StatsDClient interface in the internal-api so you didn't then need to expose and cast to the implementation.
PerfectSlayer
changed the title
mcculls
reviewed
Jul 2, 2024
| @SuppressForbidden | ||
| public static void sendOomeEvent(String taglist) { | ||
| try (DDAgentStatsDClient client = | ||
| (DDAgentStatsDClient) |
There was a problem hiding this comment.
We should be using the interface rather than the implementation here
|
How do we enable this new capability? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
It adds the ability to configure OOME reporting via
OnOutOfMemoryErrorJVM arg script.The script is auto-generated in the same way as the crash tracking script is.
The OOME is captured and forwarded as a Statsd event.
Motivation
Provide more data for the planned memory leak page for Java
Additional Notes
While adding the second script some issues were identified also in the crash tracking script - eg. when there are many JVM processes running on the same host and they are all crashing we might not report all of the crashes. This was fixed and the common code was refactored.
Jira ticket: PROF-10025