Skip to content

[aws] Enable Identity Federation for SecurityHub data streams #20529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
seanrathier wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

[aws] Enable Identity Federation for SecurityHub data streams #20529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
da5e049
[aws] Enable Identity Federation for SecurityHub data streams (ingest…
seanrathier Aug 4, 2026
3579193
Fix version to 7.5.0 (follows 7.4.0 from #20527)
seanrathier Aug 4, 2026
0d97cd3
Fix version to 7.3.0 (follows consolidated 7.2.0 from #20527)
seanrathier Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/aws/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "7.3.0"
changes:
- description: Enable Identity Federation (Cloud Connectors) for the SecurityHub data streams (securityhub_findings, securityhub_findings_full_posture, securityhub_insights) by migrating their HTTPJSON templates from hand-rolled SigV4 signing to the input's auth.aws configuration. Fixes silent session-token breakage in the hand-rolled path.
type: enhancement
link: https://github.com/elastic/integrations/pull/20529
- version: "7.1.1"
changes:
- description: Add `data_stream.namespace` to the Amazon Inspector vulnerability latest transform's unique key so findings are tracked per namespace, preventing findings ingested into non-default namespaces from being dropped or conflated in the latest index. Bump transform's destination suffix to `-v2`.
Expand Down
43 changes: 31 additions & 12 deletions packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,6 @@ request.ssl: {{ssl}}

request.url: https://securityhub.{{aws_region}}.{{tld}}/findings
request.transforms:
- set:
target: header.X-Amz-Date
value: '[[formatDate (now) "20060102T150405Z"]]'
- set:
target: body.MaxResults
value: 100
Expand All @@ -36,21 +33,12 @@ request.transforms:
value: '[{ "Start": "[[formatDate (parseDate .cursor.last_execution_datetime "RFC3339") "2006-01-02T15"]]", "End": "2099-01-01T00" }]'
default: '[{ "Start": "[[formatDate (now (parseDuration "-{{initial_interval}}")) "2006-01-02T15"]]", "End": "2099-01-01T00" }]'
value_type: json
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
{{!-- https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html --}}
response.pagination:
- set:
target: body.NextToken
value: '[[if (eq (len .last_response.body.Findings) 100)]][[.last_response.body.NextToken]][[end]]'
fail_on_template_error: true
do_not_log_failure: true
- delete:
target: header.Authorization
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
cursor:
last_execution_datetime:
value: '[[if (ne (len .last_response.body.Findings) 100)]][[.last_event.UpdatedAt]][[end]]'
Expand All @@ -74,3 +62,34 @@ publisher_pipeline.disable_host: true
processors:
{{processors}}
{{/if}}
auth.aws:
{{#if access_key_id}}
access_key_id: {{access_key_id}}
{{/if}}
{{#if secret_access_key}}
secret_access_key: {{secret_access_key}}
{{/if}}
{{#if session_token}}
session_token: {{session_token}}
{{/if}}
{{#if shared_credential_file}}
shared_credential_file: {{shared_credential_file}}
{{/if}}
{{#if credential_profile_name}}
credential_profile_name: {{credential_profile_name}}
{{/if}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if external_id}}
external_id: {{external_id}}
{{/if}}
{{#if assume_role_duration}}
assume_role.duration: {{assume_role_duration}}
{{/if}}
{{#if assume_role_expiry_window}}
assume_role.expiry_window: {{assume_role_expiry_window}}
{{/if}}
{{#if supports_identity_federation}}
use_cloud_connectors: {{supports_identity_federation}}
{{/if}}
47 changes: 33 additions & 14 deletions packages/aws/data_stream/securityhub_findings_full_posture/agent/stream/httpjson.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,36 +16,24 @@ request.ssl: {{ssl}}

request.url: https://securityhub.{{aws_region}}.{{tld}}/findings
request.transforms:
- set:
target: header.X-Amz-Date
value: '[[formatDate (now) "20060102T150405Z"]]'
- set:
target: body.MaxResults
value: 100
value_type: int
- set:
target: body.Filters.RecordState
target: body.Filters.RecordState
value: '[{"Comparison":"NOT_EQUALS","Value":"ARCHIVED"}]'
value_type: json
- set:
target: body.Filters.WorkflowStatus
target: body.Filters.WorkflowStatus
value: '[{"Comparison":"NOT_EQUALS","Value":"SUPPRESSED"}]'
value_type: json
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
{{!-- https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html --}}
response.pagination:
- set:
target: body.NextToken
value: '[[if (eq (len .last_response.body.Findings) 100)]][[.last_response.body.NextToken]][[end]]'
fail_on_template_error: true
do_not_log_failure: true
- delete:
target: header.Authorization
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
response.split:
target: body.Findings
ignore_empty_value: true
Expand All @@ -66,3 +54,34 @@ publisher_pipeline.disable_host: true
processors:
{{processors}}
{{/if}}
auth.aws:
{{#if access_key_id}}
access_key_id: {{access_key_id}}
{{/if}}
{{#if secret_access_key}}
secret_access_key: {{secret_access_key}}
{{/if}}
{{#if session_token}}
session_token: {{session_token}}
{{/if}}
{{#if shared_credential_file}}
shared_credential_file: {{shared_credential_file}}
{{/if}}
{{#if credential_profile_name}}
credential_profile_name: {{credential_profile_name}}
{{/if}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if external_id}}
external_id: {{external_id}}
{{/if}}
{{#if assume_role_duration}}
assume_role.duration: {{assume_role_duration}}
{{/if}}
{{#if assume_role_expiry_window}}
assume_role.expiry_window: {{assume_role_expiry_window}}
{{/if}}
{{#if supports_identity_federation}}
use_cloud_connectors: {{supports_identity_federation}}
{{/if}}
43 changes: 31 additions & 12 deletions packages/aws/data_stream/securityhub_insights/agent/stream/httpjson.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,6 @@ request.ssl: {{ssl}}

request.url: https://securityhub.{{aws_region}}.{{tld}}/insights/get
request.transforms:
- set:
target: header.X-Amz-Date
value: '[[(formatDate (now) "20060102T150405Z")]]'
- set:
target: body.MaxResults
value: 100
Expand Down Expand Up @@ -69,10 +66,6 @@ request.transforms:
"arn:aws:securityhub:::insight/securityhub/default/39",
"arn:aws:securityhub:::insight/securityhub/default/40"
]
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/insights/get\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
{{!-- https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html --}}
response.pagination:
- delete:
# Only request specific ARNs in the first request (for managed insights).
Expand All @@ -97,11 +90,6 @@ response.pagination:
value: '[[with (index .last_response.body "NextToken")]][[.]][[end]]'
fail_on_template_error: false
do_not_log_failure: true
- delete:
target: header.Authorization
- set:
target: header.Authorization
value: '[[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate ($now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate ($now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/insights/get\n" "\n" "host:securityhub.{{aws_region}}.{{tld}}\n" (sprintf "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'
response.split:
target: body.Insights
ignore_empty_value: true
Expand All @@ -122,3 +110,34 @@ publisher_pipeline.disable_host: true
processors:
{{processors}}
{{/if}}
auth.aws:
{{#if access_key_id}}
access_key_id: {{access_key_id}}
{{/if}}
{{#if secret_access_key}}
secret_access_key: {{secret_access_key}}
{{/if}}
{{#if session_token}}
session_token: {{session_token}}
{{/if}}
{{#if shared_credential_file}}
shared_credential_file: {{shared_credential_file}}
{{/if}}
{{#if credential_profile_name}}
credential_profile_name: {{credential_profile_name}}
{{/if}}
{{#if role_arn}}
role_arn: {{role_arn}}
{{/if}}
{{#if external_id}}
external_id: {{external_id}}
{{/if}}
{{#if assume_role_duration}}
assume_role.duration: {{assume_role_duration}}
{{/if}}
{{#if assume_role_expiry_window}}
assume_role.expiry_window: {{assume_role_expiry_window}}
{{/if}}
{{#if supports_identity_federation}}
use_cloud_connectors: {{supports_identity_federation}}
{{/if}}
4 changes: 1 addition & 3 deletions packages/aws/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.6.1
name: aws
title: AWS
version: 7.1.1
version: 7.3.0
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
categories:
Expand Down Expand Up @@ -1017,8 +1017,6 @@ policy_templates:
- type: httpjson
title: Collect AWS Security Hub CSPM logs via API
description: Collecting AWS Security Hub CSPM logs via API.
hide_in_var_group_options:
credential_type: [identity_federation]
screenshots:
- src: /img/securityhub_cspm_findings_insights_dashboard.png
title: Security Hub CSPM Findings and Insights dashboard screenshot
Expand Down
Loading