[aws] Enable Identity Federation for the Config data stream

[seanrathier]

Proposed commit message

[aws] Enable Identity Federation for the Config data stream

Migrate the Config CEL program from hand-rolled SigV4 signing to the
input's auth.aws configuration, and remove the config policy
template's hide_in_var_group_options gate so Identity Federation is
selectable on the agentless path.

Background

Part of https://github.com/elastic/ingest-dev/issues/8802 — first of the per-stream PRs extending the GuardDuty federation pattern to the remaining agentless-eligible aws streams. Split per stream to keep review and e2e validation scoped.

What changed

• CEL program migrated off hand-rolled SigV4 (−263 net lines): the program previously derived signing keys and built canonical requests inline, with access keys passed through CEL state — and a hard credential-validation gate rejecting every method except static access keys. With auth.aws the input signs requests transparently; the program keeps only its real logic (rule pagination, service-linked rule filtering, per-rule compliance collection with non-fatal errors).
• Full auth.aws block added including the use_cloud_connectors hook driven by supports_identity_federation — same pattern as the GuardDuty HTTPJSON stream from [aws] Add var_groups for credential type selection with Identity Federation #19828.
• Gate removed: hide_in_var_group_options dropped from the config policy template's cel input.
• Side benefit: Role ARN, shared credentials, and IAM instance profiles now work for Config — the hand-rolled signing never supported them.
• No format_version change (stays 3.6.1); no provider_permissions — the static fallback template's permissions are mirrored from the patch sets in [WIP] Support dynamic IaC for integrations #20240 via the paired cloudbeat PR.

IAM permissions (for the paired cloudbeat PR)

The program calls exactly two operations, each mapping 1:1 to its IAM action:

API call (X-Amz-Target)	IAM action
StarlingDoveService.DescribeConfigRules	config:DescribeConfigRules
StarlingDoveService.GetComplianceDetailsByConfigRule	config:GetComplianceDetailsByConfigRule

Note: #20240's config patch also lists config:DescribeComplianceByConfigRule — the program does not call it, so the CFT mirror omits it (least privilege).

Testing

• elastic-package lint / build pass
• The existing system test (TLS mock of the Config API with static dummy keys) exercises the new auth.aws path — static-key SigV4 signs offline, no STS involved
• Hold-merge: per the [Cisco FTD] Fix the handling of spaces in 113005 messages #8798 discipline, merge only after the e2e validation below confirms the federated flow

E2E test plan

The critical thing this run proves: the input's auth.aws signer correctly signs the program's do_request calls with custom X-Amz-Target headers against real AWS — the system-test mock does not verify signatures.

1. AWS-side prerequisites (test account):

• AWS Config enabled in the target region (configuration recorder + delivery channel)
• At least one customer-created managed rule (e.g. s3-bucket-versioning-enabled). Service-linked rules don't count — the program filters them out by their populated CreatedBy field.

2. Generate evaluations (findings don't exist until a rule evaluates):

• Point the rule at a deliberately noncompliant resource (e.g. an unversioned S3 bucket)
• Force evaluation instead of waiting for the schedule:

  aws configservice start-config-rules-evaluation --config-rule-names <rule-name>
  

3. Latency: on-demand evaluation completes in minutes; periodic rules can take up to 24h — always use the on-demand trigger before concluding failure.

4. Onboard in Kibana (agentless + Identity Federation):

• Add the AWS Config integration on the agentless path, select Identity Federation, deploy the role, paste Role ARN + External ID
• Verify documents in logs-aws.config-<namespace>; each evaluation result should carry ConfigRuleInfo merged in (populated aws.config fields, compliance type present)

5. Failure probes:

• Authorization failure: CEL error event with DescribeConfigRules: POST … 403 and AccessDeniedException in the message — means the deployed role is missing an action (check the paired cloudbeat PR's policy)
• Authorized-but-empty: no documents at all (internal retry placeholder events are dropped by the processor) — usually means no non-service-linked rules exist or no evaluations have run yet (see step 2)

Also validate the legacy paths still work (this PR touches shared request code): a default-mode (agent-based) policy with static access keys, and one with Role ARN — the latter never worked before this PR, so it doubles as a feature check.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have added an entry to my package's changelog.yml file.

🤖 Generated with Claude Code

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 9370eda

History

• 💔 Build #46979 failed b481b6f