awscloudwatch input drops data

[faec]

The awscloudwatch input can skip data in its target log groups, with severity depending on configuration and log size. This seems to apply to all platforms and all versions at least since 8.0.

Easy reproduction:

• Set number_of_workers to 1 (the default)
• Set start_position to beginning (the default)
• Set log_group_name_prefix to a value matching 2 or more log groups

The first matching log group will ingest data starting from the beginning, but all other log groups will only include data from after ingestion began.

This loss continues during ingestion: events from any time span will only include data from at most one log group at a time.

More finicky reproduction with a single log group:

• Set number_of_workers to 1 (the default)
• Set start_position to beginning (the default)
• Target a single log group with a significant amount of past data (enough to require significantly longer than the scan_frequency to ingest -- optionally set scan_frequency to 1s to make this easier)

Data added to the log group between the start of ingestion and the completion of the first scan will be skipped.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)