[aws_otel] update assets aggregation functions and calculations by jakubgalecki0 · Pull Request #19695 · elastic/integrations

jakubgalecki0 · 2026-06-22T15:56:18Z

Proposed commit message

Update assets to match AWS guide https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

github-actions · 2026-06-22T15:58:04Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

ishleenk17 · 2026-06-22T16:09:07Z

+  changes:
+    - description: Update aggregation function and remove division by zero in SLO metric filter.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/19637


ishleenk17 · 2026-06-22T16:11:09Z

    "artifacts": {
      "investigation_guide": {
-        "blob": "## Database Connections High\n\n### Triage\n1. Compare peak connections to the engine `max_connections` setting (not in CloudWatch).\n2. Check for connection leaks — idle sessions accumulating over time.\n3. Review application deploys or traffic spikes that increased connection demand.\n4. Inspect `FreeableMemory` — each connection consumes memory.\n\n### Mitigation\n- Enable connection pooling (PgBouncer, RDS Proxy, application pool).\n- Fix connection leaks in application code.\n- Increase `max_connections` only if memory allows; prefer pooling first."
+        "blob": "## Database Connections High\n\n### Triage\n1. Compare average connections to the engine `max_connections` setting (not in CloudWatch).\n2. Check for connection leaks — idle sessions accumulating over time.\n3. Review application deploys or traffic spikes that increased connection demand.\n4. Inspect `FreeableMemory` — each connection consumes memory.\n\n### Mitigation\n- Enable connection pooling (PgBouncer, RDS Proxy, application pool).\n- Fix connection leaks in application code.\n- Increase `max_connections` only if memory allows; prefer pooling first."


The places where we are using avg, max of a metric are similar to the aggregation we are using in dashboards for those metrics ?

Yeah. most of the metrics match have same aggregation function but we have some discrepancies. I did base the aggregation function in alerts on this resource https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html

Copilot

Pull request overview

Updates multiple AWS OpenTelemetry “content-only” integration packages to better align alert/SLO aggregations and calculations with AWS recommended CloudWatch alarm guidance, and links SLO templates back to their overview dashboards.

Changes:

Bump package versions to 0.4.0 and add corresponding changelog.yml entries.
AWS SQS OTel: replace the “backlog growth” alert with a “high backlog” alert and update DLQ matching to *dlq* across relevant alerts/SLOs.
Link several SLO templates (SQS/RDS/Lambda/ELB/EC2) to their package dashboards; add guard filters to avoid divide-by-zero in ratio SLOs.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/aws_sqs_otel/manifest.yml	Version bump to 0.4.0.
packages/aws_sqs_otel/kibana/slo_template/aws_sqs_otel-oldest-message-age-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_sqs_otel/kibana/slo_template/aws_sqs_otel-dlq-empty-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_sqs_otel/kibana/alerting_rule_template/aws_sqs_otel-oldest-message-age-high.json	Update DLQ exclusion pattern to `dlq`.
packages/aws_sqs_otel/kibana/alerting_rule_template/aws_sqs_otel-in-flight-saturation.json	Switch to `Average`/`AVG` aggregation and align DLQ exclusion pattern.
packages/aws_sqs_otel/kibana/alerting_rule_template/aws_sqs_otel-dlq-messages-present.json	Update DLQ inclusion pattern to `dlq` and adjust guidance text.
packages/aws_sqs_otel/kibana/alerting_rule_template/aws_sqs_otel-high-backlog.json	Add new sustained high-backlog alert template.
packages/aws_sqs_otel/kibana/alerting_rule_template/aws_sqs_otel-backlog-growth.json	Remove backlog-growth alert template.
packages/aws_sqs_otel/docs/README.md	Update alert template list to reflect high-backlog replacement.
packages/aws_sqs_otel/changelog.yml	Add 0.4.0 entry describing alert/aggregation updates.
packages/aws_rds_otel/manifest.yml	Version bump to 0.4.0.
packages/aws_rds_otel/kibana/slo_template/aws_rds_otel-avg-read-latency-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_rds_otel/kibana/alerting_rule_template/aws_rds_otel-database-connections-high.json	Switch to `Average`/`AVG` aggregation and update messaging accordingly.
packages/aws_rds_otel/changelog.yml	Add 0.4.0 entry for aggregation change.
packages/aws_lambda_otel/manifest.yml	Version bump to 0.4.0.
packages/aws_lambda_otel/kibana/slo_template/aws_lambda_otel-invocation-success-rate-99.5-Rolling30Days.json	Add denominator `> 0` filter and dashboard artifact link.
packages/aws_lambda_otel/kibana/slo_template/aws_lambda_otel-avg-duration-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_lambda_otel/changelog.yml	Add 0.4.0 entry for aggregation/div-by-zero guard update.
packages/aws_elb_metrics_otel/manifest.yml	Version bump to 0.4.0.
packages/aws_elb_metrics_otel/kibana/slo_template/aws_elb_metrics_otel-target-response-time-avg-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_elb_metrics_otel/kibana/slo_template/aws_elb_metrics_otel-request-availability-99.5-Rolling30Days.json	Add denominator `> 0` filter and dashboard artifact link.
packages/aws_elb_metrics_otel/changelog.yml	Add 0.4.0 entry for aggregation/div-by-zero guard update.
packages/aws_ec2_otel/manifest.yml	Version bump to 0.4.0.
packages/aws_ec2_otel/kibana/slo_template/aws_ec2_otel-status-check-availability-99.5-Rolling30Days.json	Add dashboard artifact link.
packages/aws_ec2_otel/changelog.yml	Add 0.4.0 entry for dashboard linkage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ishleenk17 · 2026-06-23T12:07:12Z

+    ],
+    "artifacts": {
+      "dashboards": [
+        { "id": "aws_ec2_otel-45f45575-d519-4a70-a50c-2195c0c9f916" }


Hoping these id;s stay the same as the updates on the dashboards are ongoing.

ishleenk17 · 2026-06-23T12:08:38Z

      "searchType": "esqlQuery",
      "esqlQuery": {
-        "esql": "FROM metrics-aws.rds.otel-*\n| WHERE Namespace == \"AWS/RDS\"\n    AND MetricName == \"DatabaseConnections\"\n    AND stat == \"Maximum\"\n| STATS max_connections = MAX(`metrics.amazonaws.com/AWS/RDS/DatabaseConnections`)\n    BY DBInstanceIdentifier, cloud.region\n// Peak connection count — set relative to max_connections and baseline (default: 100)\n| WHERE max_connections > 100\n| SORT max_connections DESC"
+        "esql": "FROM metrics-aws.rds.otel-*\n| WHERE Namespace == \"AWS/RDS\"\n    AND MetricName == \"DatabaseConnections\"\n    AND stat == \"Average\"\n| STATS avg_connections = AVG(`metrics.amazonaws.com/AWS/RDS/DatabaseConnections`)\n    BY DBInstanceIdentifier, cloud.region\n// Average connection count — set relative to max_connections and baseline (default: 100)\n| WHERE avg_connections > 100\n| SORT avg_connections DESC"


Was this change as per what we are using for dashboards. ?
Are we collecting both but using average for the database connections ?
Do we have a reasoning for which aggregation ispreferred when ?

@ishleenk17 reason for the change was to match aggregation recommended by the AWS documentation

Dashboards are using MAX:

"esql": "FROM metrics-aws.rds.otel-* | WHERE LENGTH(TO_STRING(DBInstanceIdentifier)) > 0 | WHERE stat == \"Maximum\" | STATS total_conn = MAX(CASE(stat == \"Maximum\", `metrics.amazonaws.com/AWS/RDS/DatabaseConnections`, NULL))"

kibana/dashboard/aws_rds_otel-c92e1216-83f6-4de7-9d7f-1bcb42d5988a.json

In this case I used AVG instead of MAX to avoid alerting on single spikes, and instead alert on sustained connection saturation.

agithomas · 2026-06-24T05:23:10Z

Recommendation: For the ELB, as the alerts are configured for the ALB, it would be best to add "Application Load Balancer" or similar in the title itself. Presently, it is [AWS ELB OTel], and the "Application Load Balancer" appears in the description. Kindly check the experience when the alert is triggered.

agithomas · 2026-06-24T05:27:31Z

Can these be TS queries?

jakubgalecki0 · 2026-06-24T10:04:00Z

@agithomas here are the alarms and SLOs being triggered

jakubgalecki0 · 2026-06-24T10:31:44Z

Can these be TS queries?

@agithomas we could use TS queries but I don't think there is need for that. They are pre-aggregated and we don't need time-series functions. However if you think we should go with TS queries I am happy to convert them. If so it would be the best to apply it to all other alerts.

agithomas

LGTM!

mergify · 2026-06-25T06:31:08Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

agithomas · 2026-06-25T06:34:50Z

    "artifacts": {
      "investigation_guide": {
-        "blob": "## In-Flight Saturation\n\n`ApproximateNumberOfMessagesNotVisible` counts messages received but not yet deleted (hidden by visibility timeout). Standard queues cap in-flight messages at ~120,000; approaching this limit blocks new deliveries.\n\n### Triage Steps\n1. Check whether consumers are deleting messages — compare `NumberOfMessagesDeleted` to `NumberOfMessagesReceived`.\n2. Look for stuck or crashed consumers holding messages until visibility timeout expires.\n3. Verify visibility timeout is not excessively long relative to processing time.\n4. Inspect consumer error rates — are messages failing without being deleted or sent to a DLQ?\n5. Consider scaling consumers or reducing batch receive sizes if processing is slow.\n\n### Customisation\n- Tune the `in_flight > 100000` threshold (~83% of the ~120k standard-queue limit).\n- FIFO queues have different in-flight limits — adjust threshold per queue type if needed.\n- This rule uses `Maximum` stat to catch peak in-flight during the window."
+        "blob": "## In-Flight Saturation\n\n`ApproximateNumberOfMessagesNotVisible` counts messages received but not yet deleted (hidden by visibility timeout). Standard queues cap in-flight messages at ~120,000; approaching this limit blocks new deliveries.\n\n### Triage Steps\n1. Check whether consumers are deleting messages — compare `NumberOfMessagesDeleted` to `NumberOfMessagesReceived`.\n2. Look for stuck or crashed consumers holding messages until visibility timeout expires.\n3. Verify visibility timeout is not excessively long relative to processing time.\n4. Inspect consumer error rates — are messages failing without being deleted or sent to a DLQ?\n5. Consider scaling consumers or reducing batch receive sizes if processing is slow.\n\n### Customisation\n- Tune the `in_flight > 100000` threshold (~83% of the ~120k standard-queue limit).\n- FIFO queues have different in-flight limits — adjust threshold per queue type if needed.\n- This rule uses the `Average` stat, per AWS recommended alarms for `ApproximateNumberOfMessagesNotVisible`."


nit: When the threshold (10,000) is customisable, it would be best not to mention static threshold values in the investigation guide.

agithomas · 2026-06-25T06:39:01Z

    "artifacts": {
      "investigation_guide": {
-        "blob": "## DLQ Has Messages\n\nAny message in a dead-letter queue means the pipeline dropped work after exceeding `maxReceiveCount` on the source queue.\n\n### Triage Steps\n1. Identify the source queue (DLQ name usually mirrors the primary queue, e.g. `orders-dlq` → `orders`).\n2. Inspect DLQ messages in the AWS console or via `ReceiveMessage` to find poison payloads or recurring error patterns.\n3. Check consumer logs and error rates for the service that processes the source queue.\n4. Verify redrive policy and `maxReceiveCount` are configured as expected.\n5. After fixing the root cause, redrive or manually reprocess DLQ messages.\n\n### Customisation\n- Tune the `LIKE \"*-dlq\"` pattern in the ES|QL query to match your DLQ naming convention (e.g. `*deadletter*`).\n- Threshold is `> 0` by design — even one DLQ message is actionable."
+        "blob": "## DLQ Has Messages\n\nAny message in a dead-letter queue means the pipeline dropped work after exceeding `maxReceiveCount` on the source queue.\n\n### Triage Steps\n1. Identify the source queue (DLQ name usually mirrors the primary queue, e.g. `orders-dlq` → `orders`).\n2. Inspect DLQ messages in the AWS console or via `ReceiveMessage` to find poison payloads or recurring error patterns.\n3. Check consumer logs and error rates for the service that processes the source queue.\n4. Verify redrive policy and `maxReceiveCount` are configured as expected.\n5. After fixing the root cause, redrive or manually reprocess DLQ messages.\n\n### Customisation\n- Tune the `LIKE \"*dlq*\"` pattern in the ES|QL query to match your DLQ naming convention (e.g. `*deadletter*`). This matches the `QueueName: *dlq*` filter used by the DLQ-empty SLO so the two stay consistent.\n- Threshold is `> 0` by design — even one DLQ message is actionable."


This matches the QueueName: *dlq* filter used by the DLQ-empty SLO so the two stay consistent.

nit: Is this aspect relevant enough, if the user chose to install only the alerts? Revisit .

elastic-vault-github-plugin-prod · 2026-06-25T08:09:34Z

✅ All changelog entries have the correct PR link.

infra-vault-gh-plugin-prod · 2026-06-25T08:30:38Z

💚 Build Succeeded

Buildkite Build
Commit: de4091f

History

💚 Build #44993 succeeded ce838ae
💚 Build #44987 succeeded 71fff3d
💚 Build #44985 succeeded 943073f
💚 Build #44958 succeeded 976d603
💚 Build #44954 succeeded 590cdc9
💚 Build #44936 succeeded d2ed201

elastic-vault-github-plugin-prod · 2026-06-25T08:45:05Z

Package aws_ec2_otel - 0.4.0 containing this change is available at https://epr.elastic.co/package/aws_ec2_otel/0.4.0/

elastic-vault-github-plugin-prod · 2026-06-25T08:45:06Z

Package aws_elb_metrics_otel - 0.4.0 containing this change is available at https://epr.elastic.co/package/aws_elb_metrics_otel/0.4.0/

elastic-vault-github-plugin-prod · 2026-06-25T08:45:07Z

Package aws_lambda_otel - 0.4.0 containing this change is available at https://epr.elastic.co/package/aws_lambda_otel/0.4.0/

elastic-vault-github-plugin-prod · 2026-06-25T08:45:08Z

Package aws_rds_otel - 0.4.0 containing this change is available at https://epr.elastic.co/package/aws_rds_otel/0.4.0/

elastic-vault-github-plugin-prod · 2026-06-25T08:45:09Z

Package aws_sqs_otel - 0.4.0 containing this change is available at https://epr.elastic.co/package/aws_sqs_otel/0.4.0/

jakubgalecki0 added 3 commits June 22, 2026 17:41

[aws_otel] update assets aggregation functions and calculations

dbc6fb5

strip prefix from passthrough fields

8269aeb

add changelog entrty

b19b506

jakubgalecki0 requested a review from a team as a code owner June 22, 2026 15:56

update dlq pattern in in-flight-saturation

3684f35

jakubgalecki0 changed the title ~~Aws otel alert slo fixes~~ [aws_otel] update assets aggregation functions and calculations Jun 22, 2026

jakubgalecki0 added 2 commits June 22, 2026 18:05

Merge branch 'main' into aws_otel_alert_slo_fixes

e45805f

update pr link

3d60213

ishleenk17 reviewed Jun 22, 2026

View reviewed changes

link dashboard to slos

cc71af2

muthu-mps requested a review from Copilot June 23, 2026 10:36

Copilot started reviewing on behalf of muthu-mps June 23, 2026 10:36 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Merge branch 'main' into aws_otel_alert_slo_fixes

d2ed201

andrewkroh added the Integration:aws_ec2_otel AWS EC2 Metrics OpenTelemetry Assets label Jun 23, 2026

ishleenk17 reviewed Jun 23, 2026

View reviewed changes

jakubgalecki0 added 2 commits June 23, 2026 15:49

Merge branch 'main' into aws_otel_alert_slo_fixes

590cdc9

Merge branch 'main' into aws_otel_alert_slo_fixes

976d603

update ELB assets title

943073f

infra-vault-gh-plugin-prod Bot deleted a comment from elasticmachine Jun 24, 2026

fix elb tags

71fff3d

Merge branch 'main' into aws_otel_alert_slo_fixes

ce838ae

agithomas approved these changes Jun 25, 2026

View reviewed changes

agithomas reviewed Jun 25, 2026

View reviewed changes

mykola-elastic approved these changes Jun 25, 2026

View reviewed changes

address nits

de4091f

jakubgalecki0 merged commit a9278e9 into elastic:main Jun 25, 2026
10 checks passed

jakubgalecki0 mentioned this pull request Jul 23, 2026

[aws_otel] add missing recommended alerts #19887

Merged

5 tasks

Uh oh!

Conversation

jakubgalecki0 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented Jun 22, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakubgalecki0 Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agithomas commented Jun 24, 2026

Uh oh!

agithomas commented Jun 24, 2026

Uh oh!

jakubgalecki0 commented Jun 24, 2026

Uh oh!

jakubgalecki0 commented Jun 24, 2026

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 25, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jakubgalecki0 commented Jun 22, 2026 •

edited

Loading

jakubgalecki0 Jun 23, 2026 •

edited

Loading