[aws] cloudtrail: move assumed-role session name from user.changes.* to related.user - #20318
Conversation
chemamartinez
added
enhancement
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
andrewkroh
added
the
Team:obs-ds-hosted-services
|
Failed build seems unrelated. |
| - grok: | ||
| tag: extract_session_name_email_prefix | ||
| field: _tmp.session_name | ||
| patterns: | ||
| - "%{DATA:_tmp.session_name_prefix}@" | ||
| ignore_missing: true | ||
| if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@') |
There was a problem hiding this comment.
Using grok for this is quite expensive.
| - grok: | |
| tag: extract_session_name_email_prefix | |
| field: _tmp.session_name | |
| patterns: | |
| - "%{DATA:_tmp.session_name_prefix}@" | |
| ignore_missing: true | |
| if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@') | |
| - dissect: | |
| tag: extract_session_name_email_prefix | |
| field: _tmp.session_name | |
| pattern: '%{_tmp.session_name_prefix}@%{}' | |
| if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@') |
|
@vera-review-bot review |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| - version: "7.1.0" | ||
| changes: | ||
| - description: "Stop storing the assumed-role ARN session name in `user.changes.*`. The session name is now added to `related.user` for cross-source correlation; when it is an email, both the full email and the local-part prefix are added. `user.name` continues to hold the IAM role name for detection rules." | ||
| type: enhancement |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/aws/changelog.yml:5
This entry stops populating user.changes.name for assumed-role/federated-user CloudTrail events, which is a field removal from the normalized output, but it is typed enhancement and shipped as a minor bump. Type it breaking-change and bump the package to 8.0.0.
Details
The pipeline change replaces the grok target user.changes.name with the scratch field _tmp.session_name, so for every AssumedRole / FederatedUser event the user.changes.name field disappears from the indexed document. The package versioning rules classify 'field removals or renames' and 'default behavior changes that alter collected/normalized output' as breaking changes requiring a major bump, and breaking-change is a valid changelog type. Anyone whose saved searches, dashboards or custom detection rules key on user.changes.name for CloudTrail (the field has been populated this way since PR 15601) silently loses that data on what looks like a routine minor upgrade. Note this is orthogonal to user.name, which the linked discussion agreed must not change - that part of the change is correct.
Recommendation:
Mark the entry as a breaking change and bump packages/aws/manifest.yml to 8.0.0 (and the changelog version to match):
- version: "8.0.0"
changes:
- description: |
Stop storing the assumed-role ARN session name in `user.changes.*`. The session name is now added to `related.user` for cross-source correlation; when it is an email, both the full email and the local-part prefix are added.
`user.name` continues to hold the IAM role name for detection rules. Users relying on `user.changes.name` for CloudTrail assumed-role events must switch to `related.user`.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20318🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
I don't think this should be treated as a breaking change as this fields that is not populated anymore was not used in any official asset.
| field: | ||
| - json | ||
| - _conf | ||
| - _tmp |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:2174
_tmp is added to the success-path remove processor but not to the pipeline's on_failure remove, so the scratch session-name fields are indexed on every document that hits the error path. Add _tmp to the on_failure remove list as well.
Details
This remove processor cleans json, _conf and now _tmp on the success path, but the pipeline-level on_failure block at the bottom of the file only removes json and _conf. _tmp.session_name (and _tmp.session_name_prefix) are written near line 1093, well before the many later processors that carry no ignore_failure (for example dissect_json_tlsDetails_tlsVersion, rename_additionalEventData_UserName_to_username, set_user_email, and the retain/remove_nulls scripts). When any of those fails, on_failure runs and the document is indexed with the internal scratch object still attached, exposing an undeclared _tmp.* field to users and to any dynamic mapping on the data stream. The existing removal of json/_conf in the same block shows this cleanup is intended.
Recommendation:
Add _tmp to the on_failure remove processor at the end of the pipeline so the scratch field is cleaned on both paths:
on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: error.message
value: >-
Processor '{{{ _ingest.on_failure_processor_type }}}'
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{{/_ingest.on_failure_processor_tag}}}in pipeline '{{{ _ingest.pipeline }}}' failed with message '{{{ _ingest.on_failure_message }}}'
- remove:
field:
- json
- _conf
- _tmp
ignore_missing: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| if: ctx._tmp?.session_name != null | ||
| allow_duplicates: false | ||
| ignore_failure: true | ||
| - dissect: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: low path: packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:1107
This dissect is the only one of the four new processors without ignore_failure, so a session name the pattern cannot split (for example one whose local part is empty, such as @example.com) fails the whole document into the pipeline error path. Add ignore_failure: true.
Details
The if guard only checks that the session name contains an '@'; it does not guarantee the pattern %{_tmp.session_name_prefix}@%{} produces a non-empty capture. AWS allows RoleSessionName to be any 2-64 character string from [\w+=,.@-], so a caller-supplied session name beginning with '@' is possible. Because the processor carries neither ignore_failure nor an on_failure, such an event would be tagged event.kind: pipeline_error and lose the rest of its enrichment, rather than just skipping the optional prefix extraction. The three sibling processors added alongside it (the grok and both appends) all tolerate failure, so this one is inconsistent with the block it belongs to.
Recommendation:
Make the optional prefix extraction non-fatal:
- dissect:
tag: extract_session_name_email_prefix
field: _tmp.session_name
pattern: '%{_tmp.session_name_prefix}@%{}'
if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@')
ignore_failure: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| "expected": [ | ||
| { | ||
| "@timestamp": "2010-02-08T12.00.00", | ||
| "@timestamp": "2026-07-23T08:05:06.557540923Z", |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/aws/data_stream/cloudwatch_logs/_dev/test/pipeline/test-cloudwatch-ec2.log-expected.json:4
These expected @timestamp values were regenerated to the time the test suite was last run and have nothing to do with the CloudTrail change; revert this file and the other unrelated fixtures so the diff stays scoped to the change.
Details
This PR touches three data streams whose pipelines it does not modify: cloudwatch_logs (all six @timestamp values rewritten to 2026-07-23T08:05:06.x), securityhub_findings_full_posture (17 @timestamp values rewritten to 2026-07-23T08:05:21.x), and the entity/event key reordering in test-modify-image-attribute-json.log-expected.json and test-modify-snapshot-attribute-json.log-expected.json. Both affected data streams declare "@timestamp": ".*" under dynamic_fields in their test-common-config.yml, so the recorded values are never compared and the tests pass either way - the churn is pure diff noise that will re-appear as a conflict for the next person who regenerates these fixtures, and it makes the review of the actual CloudTrail change harder. The two reordering-only CloudTrail fixtures are similarly no-op.
Recommendation:
Restore the committed values for the unrelated data streams (git checkout main -- packages/aws/data_stream/cloudwatch_logs/_dev/test/pipeline/ packages/aws/data_stream/securityhub_findings_full_posture/_dev/test/pipeline/) and regenerate only the CloudTrail fixtures, so this file keeps its original content:
{
"expected": [
{
"@timestamp": "2010-02-08T12.00.00",
"cloud": {
"provider": "aws"
}
}
]
}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: _tmp.session_name | ||
| pattern: '%{_tmp.session_name_prefix}@%{}' | ||
| if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@') | ||
| - append: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: low path: packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:1112
When the session name is an email the pipeline adds it to related.user but leaves user.email unset, so the human identity is still not queryable on a dedicated field; consider also copying it to user.email, which is additive and does not touch user.name.
Details
For an SSO/Identity Center ARN such as arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_.../firstname.lastname@example.com, the new block establishes that _tmp.session_name is an email address, but the only consumer is related.user. user.email stays empty, because the existing set_user_email processor copies from user.name, which for assumed roles holds the IAM role name and never contains an '@'. The linked issue's ECS reviewer proposed exactly this pairing ("leverage related.user and put there extracted user data from the ARN and optional user.email"), and setting user.email is purely additive: it does not change user.name, so it does not affect the detection rules that the discussion agreed to protect. The new expected fixture test-assume-role-email-session-json.log-expected.json confirms the field is currently absent.
Recommendation:
Populate user.email from the session name when it is an email, before _tmp is removed:
- set:
tag: set_user_email_from_session_name
field: user.email
copy_from: _tmp.session_name
if: ctx.user?.email == null && ctx._tmp?.session_name_prefix != null
ignore_empty_value: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
🚀 Benchmarks reportTo see the full report comment with |
|
✅ All changelog entries have the correct PR link. |
| - version: "7.1.0" | ||
| changes: | ||
| - description: "Stop storing the assumed-role ARN session name in `user.changes.*`. The session name is now added to `related.user` for cross-source correlation; when it is an email, both the full email and the local-part prefix are added. `user.name` continues to hold the IAM role name for detection rules." | ||
| type: enhancement |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/aws/changelog.yml:5
This entry is typed enhancement with a minor bump, but the change stops populating user.changes.name on existing events — retype it breaking-change and bump the package to 8.0.0.
Details
The changed pipeline no longer writes the assumed-role session name to user.changes.name (packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:1096 renames the grok capture to the scratch field _tmp.session_name). Every AssumedRole/FederatedUser CloudTrail document that previously carried user.changes.name will stop carrying it, and the 14 updated -expected.json fixtures in this PR confirm that scope. The same block also newly populates user.email (default.yml:1119-1124) and reorders related.user. Per the changelog rules, breaking-change applies to "field removals or renames" and "default behavior changes that alter collected/normalized output" — both apply here — and a breaking change requires a major version bump. The entry is currently type: enhancement at packages/aws/changelog.yml:5 with version: "7.1.0" (and packages/aws/manifest.yml:4 set to 7.1.0), so users upgrading on a minor get a silent field removal. Note the field itself is still legitimately produced elsewhere (default.yml:2034-2038 from requestParameters.newUserName), so only the assumed-role source is dropped — that nuance is worth stating in the description too.
Recommendation:
Retype the entry and bump to the next major in both files.
# packages/aws/changelog.yml
- version: "8.0.0"
changes:
- description: |
The assumed-role ARN session name is no longer stored in `user.changes.name`. It is now added to
`related.user` for cross-source correlation, and when it is an email both the full email and the
local-part prefix are added; `user.email` is also populated in that case. `user.name` continues to
hold the IAM role name. `user.changes.name` is still populated from `requestParameters.newUserName`
for user-rename events. Update any custom rules, dashboards, or saved queries that read
`user.changes.name` for assumed-role activity to use `related.user` instead.
type: breaking-change
link: https://github.com/elastic/integrations/pull/20318# packages/aws/manifest.yml
version: 8.0.0🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
Explained above why I don't consider this a breaking change.
| ignore_failure: true | ||
| - set: | ||
| tag: set_user_email_from_session_name | ||
| field: user.email |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:1121
This processor newly populates user.email on assumed-role events, but the changelog entry does not mention it — document the new field (or drop the processor if it was not intended).
Details
set_user_email_from_session_name sets user.email from the session name whenever the session name contains @. This is a new field on documents that previously had none: packages/aws/data_stream/cloudtrail/_dev/test/pipeline/test-tls-details-json.log-expected.json gains "email": "some.user@example.com" in this PR, and the new email-session fixture asserts it too. The changelog entry added at packages/aws/changelog.yml:4 enumerates the behavior changes (stop using user.changes.*, add to related.user, add the local-part prefix, keep user.name) but says nothing about user.email, so users will see an undocumented new field appear on assumed-role events. The guard is also looser than the pipeline's existing set_user_email at default.yml:2095-2099, which requires indexOf("@") > 0; this one accepts any session name containing @, including one that starts with @.
Recommendation:
Document the new field in the changelog description (see the suggested wording in the changelog finding), and tighten the guard to match the existing set_user_email convention:
- set:
tag: set_user_email_from_session_name
field: user.email
copy_from: _tmp.session_name
if: ctx.user?.email == null && ctx._tmp?.session_name != null && ctx._tmp.session_name.indexOf('@') > 0
ignore_empty_value: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
I don't think it is worth it to expand the changelog with this information.
| if: ctx._tmp?.session_name != null | ||
| allow_duplicates: false | ||
| ignore_failure: true | ||
| - dissect: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: low path: packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml:1107
The new dissect is the only processor in this block without ignore_failure, so a dissect failure escalates to the pipeline on_failure and stamps the whole event pipeline_error — add ignore_failure: true.
Details
The three sibling processors added in the same block (append_related_user_session_name at line 1100, append_related_user_session_name_prefix at line 1112, and the grok at line 1093 via ignore_missing) all tolerate failure, but extract_session_name_email_prefix does not. Any failure of this dissect is handled by the pipeline-level on_failure at default.yml:2209, which sets event.kind: pipeline_error and appends to error.message — turning an otherwise fully parsed CloudTrail event into an error document over an optional, best-effort enrichment. roleSessionName accepts @ anywhere in the value ([\w+=,.@-]*), so the if guard only proves an @ is present, not that the value splits into a usable prefix and suffix.
Recommendation:
Make the enrichment non-fatal, consistent with the surrounding processors:
- dissect:
tag: extract_session_name_email_prefix
field: _tmp.session_name
pattern: '%{_tmp.session_name_prefix}@%{}'
if: ctx._tmp?.session_name != null && ctx._tmp.session_name.contains('@')
ignore_failure: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
I think it is better to know when an error happens here.
Review summaryIssues found across the latest commits f1ab611 — 1 high, 1 medium, 1 low
Issues found across earlier commits d469229 — 1 high, 1 medium, 3 low
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
|
|
Tick the box to add this pull request to the merge queue (same as
|
|
Package aws - 7.1.0 containing this change is available at https://epr.elastic.co/package/aws/7.1.0/ |
4 participants
Proposed commit message
Checklist
changelog.ymlfile.Related issues