aws_securityhub: latest_findings transform fails once total_fields.limit (2000) is exhausted

[kcreddy]

Summary

The latest_findings transform in the AWS Security Hub package (OCSF / GetFindingsV2) goes health red once its destination index runs out of mapping budget:

Limit of total fields [2000] has been exceeded while adding new fields [1]
Destination index mappings are incompatible with the transform configuration

After this, documents stop reaching the latest view (logs-aws_securityhub_latest.dest_finding-1) and the transform retry-loops. Ingest into the finding data stream keeps working, so the problem is easy to miss until the latest view goes stale.

How close the package is to the limit

Counting mapping entries the way Elasticsearch counts them (objects, leaves and multi-fields all count toward total_fields), the packaged field definitions alone consume:

Index	Declared mapping entries	Limit
finding data stream	1,569	2,000
latest_findings destination	1,569	2,000
latest_cdr_vulnerabilities destination	1,489	2,000

That leaves roughly 430 slots before a single document is indexed. The 2000 limit is set in the finding data stream manifest, the latest_findings destination template and the latest_cdr_vulnerabilities destination template.

What consumes the remaining budget

aws_securityhub.finding.resources is a declared group, but two of its subtrees grow with the data rather than with the package:

• resources.data declares only awsEc2InstanceDetails and awsLambdaFunctionDetails. Security Hub emits a *Details object per resource type (S3 buckets, RDS instances, IAM roles, EKS clusters and many more), and every undeclared one maps dynamically.
• resources.tags.* is a wildcard object, so each customer-defined tag key becomes its own mapping entry.

For scale, the ten pipeline test fixtures in the repository already produce 37 distinct leaf paths under resources.

Why ingest survives and the transform does not

Fleet composes the stack templates logs@mappings and logs@settings into logs data stream index templates (template.ts#L148), and logs@settings sets index.mapping.total_fields.ignore_dynamic_beyond_limit: true. Past the limit, dynamic fields are silently left unmapped instead of failing the write.

Transform destination templates are built by a different code path that composes only the package component templates plus ecs@mappings, so they get no such escape hatch and the mapping update fails hard. The data stream side is therefore still a problem, just a quieter one: fields beyond 2,000 stop being searchable and aggregatable without any error surfaced.

Proposed fix

Raise total_fields.limit from 2000 to 5000 on the finding data stream and both transform destinations, following the same approach as #20395 for CrowdStrike FDR.

Raising the limit is not sufficient on its own for the transforms. Fleet keys transform reinstallation on _meta.fleet_transform_version and skips reinstalling the destination index template when it is unchanged (install.ts#L554). Even when the version is bumped, the previous transform is deleted with delete_dest_index: false (install.ts#L521), so the existing destination index keeps its old settings. The destination index name therefore also needs bumping, as done for ibm_qradar and tenable_io in #16180 and for qualys_gav in #20080:

• latest_findings: dest_finding-1 → dest_finding-2, fleet_transform_version 0.1.0 → 0.2.0
• latest_cdr_vulnerabilities: vulnerability_latest-v1 → vulnerability_latest-v2, fleet_transform_version 0.2.0 → 0.3.0

The destination indices are rebuilt from the source data stream after the upgrade, and the read aliases move to the new indices on creation.

Known limitation

5,000 is headroom, not a bound. Tag keys are customer-defined and the set of ASFF detail schemas is large, so a big enough estate can exhaust 5,000 as well. Mapping resources.data and resources.tags as flattened would cap each subtree at a single mapping entry, consistent with the 147 flattened fields the package already uses. That is a larger change and is deliberately out of scope here.

Workaround for an index already in this state

index.mapping.total_fields.limit is a dynamic setting, so an affected deployment can unblock the transform before upgrading:

PUT logs-aws_securityhub_latest.dest_finding-1/_settings
{ "index.mapping.total_fields.limit": 5000 }

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)