Filebeat input cloudwatch doesn't use registry file #37620
Labels
Filebeat
Filebeat
Team:Cloud-Monitoring
Label for the Cloud Monitoring team
Team:obs-ds-hosted-services
Label for the Observability Hosted Services team
Registry file usage for filebeat cloudwatch input
I'm trying to use filebeat as a log shipper from AWS Cloudwatch to ELK trough Logstash.
Filebeat configuration:
`
filebeat.registry.path: /efs/${NAMESPACE}/registry
filebeat.shutdown_timeout: 5s
registry.flush: 5s
filebeat.inputs:
log_group_name: /aws/lambda/some-service-lambda
region_name: us-east-1
enabled: true
start_position: beginning
scan_frequency: 1m
access_key_id: ${ACCESS_KEY}
secret_access_key: ${SECRET_KEY}
fields_under_root: true
fields:
type: generic
componentGroup: lambda
environment: dev
partition: cw2log
region: na
clientId: somevalue
component: some-service-lambda
output.logstash:
hosts:
- logstash:443
ssl:
verification_mode : none
enabled: true
certificate_authorities: ["/etc/logstash-crt/ssl_cert"]
certificate: "/etc/logstash-crt/ssl_cert"
key: "/etc/logstash-crt/ssl_cert_key"
`
during data shipping registry files is empty:
filebeat@cw2li-filebeat-554bb695d8-kh84x:~$ ls -la /efs/namespace/registry/filebeat/ total 16 drwxr-x--- 2 filebeat filebeat 6144 Jan 11 20:35 . drwxr-x--- 3 filebeat filebeat 6144 Jan 11 20:35 .. -rw------- 1 filebeat filebeat 0 Jan 11 20:35 log.json -rw------- 1 filebeat filebeat 15 Jan 11 20:35 meta.json
and with restart or tempoprary infrastructure outage, filebeat starts shipping data again producing duplicates.
Case with
start_position: end
also does not allow to have significant outage and can lead to data loss.Cloud you please check possibility to store the latest delivered timestamp or any other metrics in the registry file and use it for resume shipping from this moment after restart ?
The text was updated successfully, but these errors were encountered: