[AWS] Add Amazon EKS audit data stream - #21051
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
bryans3c
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
For support, reach out in the #docs Slack channel. cc @elastic/integration-docs |
Elastic Docs Style Checker (Vale)Summary: 8 suggestions found 💡 Suggestions (8): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
There was a problem hiding this comment.
🟡 Changes recommended
The current implementation has user-facing inconsistencies (notably preserve_original_event behavior and changelog PR link correctness) that should be resolved before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new AWS integration data stream (aws.eks_audit) to collect and parse Amazon EKS Kubernetes API audit logs from CloudWatch Logs, including a dedicated Fleet policy template and full ingest pipeline/field mapping/test coverage.
Changes:
- Register a new Amazon EKS policy template in the AWS package and bump package version/changelog.
- Add the
eks_auditdata stream (agent stream template, ingest pipelines, fields, sample event) to parse EKS audit events intoaws.eks.audit.*+ ECS with redaction/normalization. - Add docs and pipeline/policy fixtures to validate configuration precedence and parsing behavior.
File summaries
| File | Description |
|---|---|
| packages/aws/manifest.yml | Bump package version; register Amazon EKS policy template. |
| packages/aws/docs/README.md | Add EKS row to service/data coverage table. |
| packages/aws/docs/eks.md | New EKS audit stream documentation and field reference. |
| packages/aws/data_stream/eks_audit/sample_event.json | New sample event for the dataset. |
| packages/aws/data_stream/eks_audit/manifest.yml | Define the eks_audit data stream and its vars. |
| packages/aws/data_stream/eks_audit/fields/fields.yml | Define aws.eks.audit.* and related AWS fields. |
| packages/aws/data_stream/eks_audit/fields/ecs.yml | Declare ECS fields used by the data stream. |
| packages/aws/data_stream/eks_audit/fields/base-fields.yml | Define base data stream fields/constants. |
| packages/aws/data_stream/eks_audit/elasticsearch/ingest_pipeline/default.yml | Default pipeline to enrich CloudWatch metadata and route to audit parsing. |
| packages/aws/data_stream/eks_audit/elasticsearch/ingest_pipeline/audit.yml | Core audit JSON parsing, normalization, redaction, ECS projection. |
| packages/aws/data_stream/eks_audit/agent/stream/aws-cloudwatch.yml.hbs | Agent stream template for aws-cloudwatch input selection/vars. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-static-credentials.yml | Policy test fixture for static credentials + ARN mode. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-static-credentials.expected | Expected rendered policy output for static credentials fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-shared-profile-assume-role.yml | Policy test fixture for shared profile + role assume. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-shared-profile-assume-role.expected | Expected rendered policy output for shared profile + role assume. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-name-precedence.yml | Policy test fixture for name-vs-prefix precedence. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-name-precedence.expected | Expected rendered policy output for name precedence fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-default-prefix.yml | Policy test fixture for default /aws/eks/ prefix mode. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-default-prefix.expected | Expected rendered policy output for default prefix fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-arn-precedence.yml | Policy test fixture for ARN precedence over name/prefix. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-arn-precedence.expected | Expected rendered policy output for ARN precedence fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-agentless-cloud-connector.yml | Policy test fixture for agentless cloud connector mode. |
| packages/aws/data_stream/eks_audit/_dev/test/policy/test-agentless-cloud-connector.expected | Expected rendered policy output for agentless cloud connector fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit.log-expected.json | Pipeline expected outputs for primary fixture set. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit.log-config.yml | Pipeline test config (fields/tags/input metadata). |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit.log | Pipeline input events (JSON audit lines). |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-no-preserve.log-expected.json | Pipeline expected outputs for “no preserve” fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-no-preserve.log-config.yml | Pipeline config for “no preserve” fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-no-preserve.log | Pipeline input events for “no preserve” fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-arn.log-expected.json | Pipeline expected outputs for ARN-style log_group metadata fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-arn.log-config.yml | Pipeline config for ARN-style log_group metadata fixture. |
| packages/aws/data_stream/eks_audit/_dev/test/pipeline/test-eks-audit-arn.log | Pipeline input event for ARN-style log_group metadata fixture. |
| packages/aws/changelog.yml | Add 8.2.0 changelog entry for EKS audit stream. |
| packages/aws/_dev/build/docs/README.md | Generated docs update (service/data coverage table). |
| packages/aws/_dev/build/docs/eks.md | Generated docs page for EKS audit stream. |
| .github/CODEOWNERS | Add ownership for the new eks_audit data stream. |
Review details
- Files reviewed: 33/36 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - rename: | ||
| tag: rename_message_to_event_original | ||
| field: message | ||
| if: ctx.event?.original == null | ||
| ignore_missing: true | ||
| target_field: event.original | ||
| - json: |
There was a problem hiding this comment.
This is handled centrally by Fleet, not per-integration. Verified end-to-end against a live logs-aws.eks_audit-* data stream: a document without the tag is indexed without event.original and the same document with the tag retains it. Pipeline test fixtures show event.original only because elastic-package test pipeline simulates the package pipeline in isolation.
|
✅ All changelog entries have the correct PR link. |
There was a problem hiding this comment.
🔵 Needs a closer look
The ingest pipeline currently retains event.original even when “Preserve original event” is disabled, which contradicts the documented behavior and increases sensitive-data retention risk.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
packages/aws/docs/eks.md:31
- Grammar: “looks as following” should be “looks as follows”.
packages/aws/data_stream/eks_audit/elasticsearch/ingest_pipeline/audit.yml:25
preserve_original_eventis documented as controlling whetherevent.originalis populated, but this pipeline always renamesmessagetoevent.originalunconditionally (wheneverevent.originalis absent). As a result, raw audit payloads can be retained even when users leave Preserve original event disabled, which undermines the intent to minimize sensitive-data retention (notably for Secret events) and contradicts the stream var description.
- rename:
tag: rename_message_to_event_original
field: message
if: ctx.event?.original == null
ignore_missing: true
- Files reviewed: 33/36 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
🟢 No issues across the latest commits 1b65e59. Review summaryIssues found across earlier commits 0d64e2f — 1 medium
🤖 AI-Generated Review | Vera Review Bot - v0.3.0 | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
|
|
@bryans3c - This PR has merge conflicts with the base branch. Please resolve them. |
Adds the
aws.eks_auditdata stream to the AWS integration to collect Amazon EKS Kubernetes API audit logs from CloudWatch Logs, parallel to how theazureandgcppackages expose provider-native Kubernetes audit streams. Without this, EKS audit events are only reachable through the provider-neutralkubernetes.audit_logsstream, so AWS-native detection rules have nologs-aws.*index to target and noaws.eks.audit.*schema to query.Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
123456789012, RFC 5737 IPs, placeholder users and cluster names). No real account IDs, ARNs, IPs, or identities.elastic-package build,check,test static,test pipeline,test policy,test assetall pass.pods/exec, RBAC objects, core API group normalization, Secret and SecretList redaction, SubjectAccessReview,event.originalpreservation modes, ARN-style log group metadata, dotteduser.extrakeys, and malformed field shapes.elastic-package test system.event.originalis disabled by default;preserve_original_eventretains the unredacted payload and the docs say so.logs:GetLogEventsis not required; the docs list onlylogs:DescribeLogGroupsandlogs:FilterLogEvents.changelog.ymllink with this PR's URL after opening.How to test this PR locally
Optional end-to-end against a real cluster (requires an EKS cluster with control-plane
auditlogging enabled and read access to its CloudWatch log group):Then in Discover, data view
logs-aws.eks_audit-*, confirmaws.eks.audit.verb,aws.eks.audit.objectRef.*,user.name,source.ip,user_agent.original,event.outcome,event.category,event.type, andorchestrator.cluster.namepopulate, and thataws.eks.audit.requestObject/responseObjectare present in_sourcebut only declared sub-fields are searchable.Related issues
kubernetesintegration covers the provider-neutral path)kubernetes.audit_logsCloudWatch input)feature/*branch that cannot accept pushes)Screenshots