-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AWS] Drop header log line in CloudFront events #5017
Conversation
CloudFront stores these two lines as a header in each log file. Before this change it fails the pipeline and add message to error.message field. #Version: 1.0 #Fields: date time x-edge-location (TRUNCATED)
Hey @0ccupi3R, thank you for taking the time to work on this contribution to the AWS integration! You also created #5016 with a sample file for the tests. Could you please move that sample file to this PR? Having them all together in the same PR allows us to work and merge the change as a single unit of work. |
Added header events which is available in each Cloud Front log file. `DROP` filter has added pipeline.
Hello @zmoog Thanks for picking it. Even though it was just a header line, I have added it to the sample log line within the same PR. |
For the records, I see the Elastic Agent integration uses a 'local' processor to filter out the logs comments before sending: integrations/packages/aws/data_stream/cloudfront_logs/agent/stream/aws-s3.yml.hbs Lines 48 to 52 in ef38d55
So when the Agent runs with the DEBUG log level we can see the following message:
I can see it contains @0ccupi3R, are you using the Elastic Agent to ingest the CloudFront standard logs? |
It's worth dropping invalid loglines to avoid errors. |
/test |
🌐 Coverage report
|
/test |
/test |
2 similar comments
/test |
/test |
Package aws - 1.29.1 containing this change is available at https://epr.elastic.co/search?package=aws |
By default the CloudFront stores these two lines as a header in each log file. Before this change it fails the pipeline and add message to error.message field.
Added the sample log line
#5016 6c95e22