Skip to content
Logging infrastructure for exporting all CloudWatch logs from multiple accounts to a single S3 bucket
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
CloudWatch2S3-additional-account.template
CloudWatch2S3.template
LICENSE
README.md
architecture.svg

README.md

AWS CloudWatch to S3

Logging infrastructure for exporting all CloudWatch logs from multiple accounts to a single S3 bucket.

Overview

Architecture diagram

This project supplies a CloudFormation template that setups Kinesis stream that takes log records from CloudWatch and writes them to a specific S3 bucket as they are arrive. Log records can be retrieved from multiple AWS accounts using the second CloudFormation template.

Log records are batched together across log groups and partitioned into folders based on time of ingestion. Log format can be configured to either be raw log lines or compressed CloudWatch JSON. The raw log format is:

LOG_GROUP:LOG_STREAM\tTIMESTAMP\tRAW_LOG_LINE

For example:

/aws/lambda/SomeLambdaFunction:2019/02/05/[$LATEST]b346f603d7bb4b6aa77b53bc4050bc37 1549428326  INFO hello world

Subscription of CloudWatch log groups is done in two ways. If CloudTrail is enabled, every new log group will immediacy be subscribed. In addition, every hour a subscription Lambda is executed to look for new log groups and subscribe them. Finally, the same subscription Lambda is executed during deployment of the CloudFormation stack so all log groups matching the configured prefix will be subscribed immediately on deployment.

If CloudTrail is not enabled, it may take up to an hour for new log groups to be subscribed. This time can be configured in the CloudFormation stack using the SubscribeSchedule parameter. In CloudFormation UI it may be named Look for New Logs Schedule.

Deploy

If you have just one AWS account, simply deploy CloudWatch2S3.template in CloudFormation.

If you have multiple AWS accounts, choose a central account where all logs will be stored in S3 and deploy CloudWatch2S3.template in CloudFormation. Once done, go to the outputs tab and copy the value of LogDestination. Then go to the other accounts and deploy CloudWatch2S3-additional-account.template in CloudFormation. You will need to supply the value you copied as the LogDestination parameter.

Parameters

There are a lot of parameters to play with, but the defaults should be good enough for most. If you have a lot of log records coming in (more than 1000/s or 1MB/s), you might want to increase Kinesis shard count.

Troubleshooting

  • Make sure the right CloudWatch log groups are subscribed
  • Look for errors in CloudWatch log group /aws/kinesisfirehose/<STACK NAME>-DeliveryStream
  • Look for errors in CloudWatch log group /aws/lambda/<STACK NAME>-LogProcessor-<RANDOM>
  • Make sure Kinesis, Firehose and S3 can using your KMS key when using encryption
  • Increase Kinesis shard count with ShardCount (Kinesis Shard Count) CloudFormation parameter

Origin

This project is based on Amazon's Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis but adds:

  • One step installation
  • Zero scripts
  • Works out of the box
  • Easier configuration without editing files
  • No hard dependency on CloudTrail
  • Optional unpacking of CloudWatch JSON format
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.