New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CloudWatch Log Group and CloudWatch Log Stream fields to AWS Flow Logs and Logs #54

Closed
danielgrant opened this Issue Nov 2, 2017 · 0 comments

Comments

Projects
None yet
3 participants
@danielgrant
Contributor

danielgrant commented Nov 2, 2017

Currently, org.graylog.aws.cloudwatch.CloudWatchLogData only extracts the logEvents field from a CloudWatch payload, and ignores additional useful metadata - specifically, the logGroup and logStream fields.

We have encountered use cases where this information is not only useful, but essential, e.g.:

  • When running ECS tasks, the task definition can be configured to use the awslogs driver and write to a CloudWatch Log Group. This results in multiple tasks writing to different streams within the same group. There is currently no way to distinguish between these streams (and therefore, the individual tasks responsible for generating the log entries) in Graylog.

  • When using Auto Scaling Groups that create and destroy EC2 instances based on CloudWatch Alarms, the user data defined in the Launch Configuration attached to the Auto Scaling Group can install and configure the CloudWatch Logs Agent to stream various system logs from the EC2 instance to a CloudWatch Log Group. Each individual EC2 instance writes to its own stream within the log group. There is currently no way to distinguish between these streams (and therefore, the different EC2 instances responsible for generating the log entries) in Graylog.

We would propose that the Graylog AWS plugin be updated to consume the logGroup and logStream fields from the CloudWatch payload, and apply these fields to the log entries in Graylog, so that Graylog is capable of distinguishing between the constituent streams of a log group.

@jalogisch jalogisch added the feature label Nov 6, 2017

@joschi joschi closed this in 8e53cca Nov 9, 2017

joschi added a commit that referenced this issue Nov 9, 2017

@joschi joschi added this to the 2.4.0 milestone Nov 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment