Skip to content
Permalink
Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
63 lines (42 sloc) 3.12 KB

AWS CloudWatch to S3

Logging infrastructure for exporting all CloudWatch logs from multiple accounts to a single S3 bucket.

Overview

Architecture diagram

This project supplies a CloudFormation template that setups Kinesis stream that takes log records from CloudWatch and writes them to a specific S3 bucket as they are arrive. Log records can be retrieved from multiple AWS accounts using the second CloudFormation template.

Log records are batched together across log groups and partitioned into folders based on time of ingestion. Log format can be configured to either be raw log lines or compressed CloudWatch JSON. The raw log format is:

LOG_GROUP:LOG_STREAM\tTIMESTAMP\tRAW_LOG_LINE

For example:

/aws/lambda/SomeLambdaFunction:2019/02/05/[$LATEST]b346f603d7bb4b6aa77b53bc4050bc37 1549428326  INFO hello world

Subscription of CloudWatch log groups is done in two ways. If CloudTrail is enabled, every new log group will immediacy be subscribed. In addition, every hour a subscription Lambda is executed to look for new log groups and subscribe them. Finally, the same subscription Lambda is executed during deployment of the CloudFormation stack so all log groups matching the configured prefix will be subscribed immediately on deployment.

If CloudTrail is not enabled, it may take up to an hour for new log groups to be subscribed. This time can be configured in the CloudFormation stack using the SubscribeSchedule parameter. In CloudFormation UI it may be named Look for New Logs Schedule.

Deploy

If you have just one AWS account, simply deploy CloudWatch2S3.template in CloudFormation.

If you have multiple AWS accounts, choose a central account where all logs will be stored in S3 and deploy CloudWatch2S3.template in CloudFormation. Once done, go to the outputs tab and copy the value of LogDestination. Then go to the other accounts and deploy CloudWatch2S3-additional-account.template in CloudFormation. You will need to supply the value you copied as the LogDestination parameter.

Parameters

There are a lot of parameters to play with, but the defaults should be good enough for most. If you have a lot of log records coming in (more than 1000/s or 1MB/s), you might want to increase Kinesis shard count.

Troubleshooting

  • Make sure the right CloudWatch log groups are subscribed
  • Look for errors in CloudWatch log group /aws/kinesisfirehose/<STACK NAME>-DeliveryStream
  • Look for errors in CloudWatch log group /aws/lambda/<STACK NAME>-LogProcessor-<RANDOM>
  • Make sure Kinesis, Firehose and S3 can using your KMS key when using encryption
  • Increase Kinesis shard count with ShardCount (Kinesis Shard Count) CloudFormation parameter

Origin

This project is based on Amazon's Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis but adds:

  • One step installation
  • Zero scripts
  • Works out of the box
  • Easier configuration without editing files
  • No hard dependency on CloudTrail
  • Optional unpacking of CloudWatch JSON format
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.