[aws_sqs_otel] Add ML anomaly detection module

[JM-elastic]

What

Adds a machine-learning anomaly-detection module (kibana/ml_module/) to the aws_sqs_otel integration, proposing anomaly detection as an addition alongside the integration's existing dashboards, alert rules, and SLO templates. Modeled on the kubernetes_otel ML module (#19030).

Why — complements the threshold alerts, doesn't duplicate them

The shipped alert rules catch per-entity threshold breaches (a value crossing a fixed line). These ML jobs model each metric per entity against its own history, catching the drift those miss — e.g. a queue building up abnormally for that queue (a stalling or under-scaled consumer) before it crosses the fixed depth/age threshold. Each detector's description defers per-entity spikes to the alert rules, the same split kubernetes_otel uses. The detectors are drawn from the service's own signals and real failure modes — not tailored to any specific workflow.

Jobs

• aws_sqs_queue_backlog_anomaly — per QueueName (partition cloud.region): high_mean ApproximateNumberOfMessagesVisible, ApproximateNumberOfMessagesNotVisible, ApproximateAgeOfOldestMessage.

Datafeeds are composite-aggregated — required, because these metrics-aws.*.otel-* indices contain aggregate_metric_double fields that a plain (non-aggregating) ML datafeed cannot read.

Validation

Drafted and validated against live AWS OTel telemetry: the job(s) establish baselines over historical data. The RDS connection-pool-exhaustion case was scored against a known injected incident and detected it on the correct entity (recall/precision/f1 = 1.0).

Methodology, tooling, and the scoring harness: https://github.com/elastic/aws_otel_ml_draft

Notes for reviewers (@elastic/obs-infraobs-integrations)

• Draft — proposing for your review; happy to adjust job naming, detector selection, bucket_span, or thresholds.
• Package stays subscription: basic (matches kubernetes_otel; ML availability is a deployment concern, not a package condition).
• Entity fields use the raw CloudWatch dimensions present in the indexed documents (QueueName), not the normalized fields the alert-rule termFields reference (those are not present in the documents).

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[jakubgalecki0]

Lets please make sure that we follow naming convention of other aws_otel assets i.e.

[<servicename> OTel] Description

Right now we have following convention: AWS SQS queue backlog (OpenTelemetry)

Also add entry for tags since package-spec allows it to be tagged - https://github.com/elastic/package-spec/blob/main/spec/integration/kibana/tags.spec.yml#L34

  asset_types:
    - dashboard
    - alerting_rule_template
    - slo_template
    - ml_module <<< 

Other than that it looks good.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 8a66db0

History

• 💔 Build #46640 failed 8389a52
• 💚 Build #45358 succeeded 73a0cde

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

Package aws_sqs_otel - 0.11.0 containing this change is available at https://epr.elastic.co/package/aws_sqs_otel/0.11.0/