Skip to content

[AWS-cloudtrail] set host.id field#17827

Merged
alexreal1314 merged 4 commits intoelastic:mainfrom
alexreal1314:16343-aws-set-host
Mar 18, 2026
Merged

[AWS-cloudtrail] set host.id field#17827
alexreal1314 merged 4 commits intoelastic:mainfrom
alexreal1314:16343-aws-set-host

Conversation

@alexreal1314
Copy link
Contributor

@alexreal1314 alexreal1314 commented Mar 15, 2026

Proposed commit message

Fix CloudTrail host entity classification to only use EC2 instance IDs.

The entity classification logic incorrectly treated AMI, EBS volume, and snapshot IDs as host resources. Only EC2 instance IDs (i-*) are genuine host identifiers.
Removed volume, snapshot, and image from hostResourceTypes, keeping only instance. Removed vol-, snap-, and ami- from hostIdPrefixes, keeping only i-. These non-instance resource IDs now fall through to genericTargets and are stored in entity.target.id instead of host.target.entity.id.
Added host.id as a new field set to a single string value (the first EC2 instance ID) rather than an array, since host.id should represent one host per document.

Updated test expected files accordingly: replaced host blocks with entity.target.id for snapshot and AMI tests, and set host.id as a single string in the five EC2 instance test files.

closes https://github.com/elastic/security-team/issues/16343.


Fix host entity classification in CloudTrail ingest pipeline

Problem

The CloudTrail ingest pipeline's entity classification logic treated all resources matching hostIdPrefixes (i-, vol-, snap-, ami-) and hostResourceTypes (instance, volume, snapshot, image) as host entities. This caused AMI IDs, EBS volume IDs, and snapshot IDs to be incorrectly stored in host.target.entity.id and host.id.

These resource types are not hosts in any meaningful sense — only EC2 instance IDs (i-*) represent actual compute hosts. For example, ModifySnapshotAttribute events were producing documents with host.id: ["snap-0a392d80692e2526a"], which is semantically wrong.

Changes

  1. hostResourceTypes: Removed volume, snapshot, and image — only instance remains.
  2. hostIdPrefixes: Removed vol-, snap-, ami- — only i- remains.
  3. host.id: New field added in this PR to surface the target host identifier. Set to a single string value (first EC2 instance ID) rather than an array, since host.id represents one host per document.
  4. Routing: Non-instance resource IDs (vol-*, snap-*, ami-*) now fall through to genericTargets and are stored in entity.target.id instead.

Affected fields

Resource type Before After
EC2 instance (i-*) host.target.entity.id (array) host.target.entity.id (array), host.id (string)
AMI (ami-*) host.target.entity.id entity.target.id
EBS volume (vol-*) host.target.entity.id entity.target.id
Snapshot (snap-*) host.target.entity.id entity.target.id

Test updates

  • test-modify-snapshot-attribute-json.log-expected.jsonhost block replaced with entity.target.id
  • test-modify-image-attribute-json.log-expected.jsonhost block replaced with entity.target.id
  • 5 EC2 instance test files — added host.id as a single string value

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

  1. cd packages/aws
  2. run elastic-package test pipeline -v --data-streams cloudtrail --generate

Related issues

Screenshots

@alexreal1314 alexreal1314 self-assigned this Mar 15, 2026
@alexreal1314 alexreal1314 added enhancement New feature or request Integration:aws AWS Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] labels Mar 15, 2026
@alexreal1314 alexreal1314 marked this pull request as ready for review March 15, 2026 21:51
@alexreal1314 alexreal1314 requested review from a team as code owners March 15, 2026 21:51
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Mar 16, 2026

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@alexreal1314 alexreal1314 changed the title [AWS-cloudtrail] set host.id field for entity id generation and fetching [AWS-cloudtrail] set host.id field Mar 16, 2026
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 16, 2026
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, the PR body only links to a private security-team issue, no commit message explaining what or why. This should be the squash commit message and the long-term record in git history. Please fill in the "Proposed commit message" section per the PR template.

@alexreal1314 alexreal1314 requested a review from andrewkroh March 17, 2026 16:19
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @alexreal1314

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host.id value seems better aligned to ECS now.

Regarding the proposed commit message, those are going to be viewed as plain text by tools like git log so do not use Markdown. Some recent examples: #17508, #17379. So please edit that to avoid markdown, and then be sure to paste it during the squash and merge.

@alexreal1314 alexreal1314 merged commit 7d3b60b into elastic:main Mar 18, 2026
10 checks passed
@elastic-vault-github-plugin-prod

Package aws - 6.4.0 containing this change is available at https://epr.elastic.co/package/aws/6.4.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:aws AWS Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants