Filebeat input cloudwatch doesn't use registry file

[badbatcat]

Registry file usage for filebeat cloudwatch input

I'm trying to use filebeat as a log shipper from AWS Cloudwatch to ELK trough Logstash.

Filebeat configuration:
`
filebeat.registry.path: /efs/${NAMESPACE}/registry
filebeat.shutdown_timeout: 5s
registry.flush: 5s
filebeat.inputs:

• type: aws-cloudwatch
  log_group_name: /aws/lambda/some-service-lambda
  region_name: us-east-1
  enabled: true
  start_position: beginning
  scan_frequency: 1m
  access_key_id: ${ACCESS_KEY}
  secret_access_key: ${SECRET_KEY}
  fields_under_root: true
  fields:
  type: generic
  componentGroup: lambda
  environment: dev
  partition: cw2log
  region: na
  clientId: somevalue
  component: some-service-lambda

output.logstash:
hosts:
- logstash:443
ssl:
verification_mode : none
enabled: true
certificate_authorities: ["/etc/logstash-crt/ssl_cert"]
certificate: "/etc/logstash-crt/ssl_cert"
key: "/etc/logstash-crt/ssl_cert_key"
`

during data shipping registry files is empty:
filebeat@cw2li-filebeat-554bb695d8-kh84x:~$ ls -la /efs/namespace/registry/filebeat/ total 16 drwxr-x--- 2 filebeat filebeat 6144 Jan 11 20:35 . drwxr-x--- 3 filebeat filebeat 6144 Jan 11 20:35 .. -rw------- 1 filebeat filebeat 0 Jan 11 20:35 log.json -rw------- 1 filebeat filebeat 15 Jan 11 20:35 meta.json

and with restart or tempoprary infrastructure outage, filebeat starts shipping data again producing duplicates.
Case with start_position: end also does not allow to have significant outage and can lead to data loss.

Cloud you please check possibility to store the latest delivered timestamp or any other metrics in the registry file and use it for resume shipping from this moment after restart ?

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[botelastic]

This issue doesn't have a Team:<team> label.