Lambda Function for Cloudtrail to Kinesis streaming
This is a Lambda function that will stream Cloudtrail logs saved to an S3 bucket to either a single Kinesis stream, a single GCP PubSub topic, a single GCP Stackdriver log stream, or any combination of the three.
It can support any number of S3 buckets, as it executes based off of any S3 notification events sent either directly to the lambda func or to an SNS topic that the lambda func subscribes to, but the code is specific to Cloudtrail logs. It will decode the Cloudtrail JSON and send one "Record" at a time to the kinesis stream.
Any single lambda function running this code can only support EITHER S3 events or SNS events.
This is controlled by the
CT_EVENT_TYPE environment variable, and defaults to S3.
make package can be used to package the function in a zip file. A docker container is
temporarily used to generate the Linux executable and archive it in the zip.
An example CloudFormation template exists in the cf directory. This will create everything needed for the lambda function to function as well as a mock s3 bucket and kinesis stream that can be used for testing.
A note on GCP configuration
If you want to send Cloudtrail records to GCP's PubSub or GCP's Stackdriver (or both), you will need to additionally
provide a JSON file in the Lambda code bundle named
gcp_credentials.json that holds the Service Account credentials
that will be used to publish to the configured topic or write to the configured log stream.
We support (and recommend) using sops to encrypt this JSON blob.
CT_STACKDRIVER_NAME (required if neither CT_TOPIC_ID or CT_KINESIS_STREAM are set)
The name of the Stackdriver logger that Cloudtrail records will be sent to.
CT_TOPIC_ID (required if neither CT_STACKDRIVER_NAME or CT_KINESIS_STREAM are set)
The topic id of the GCP PubSub topic that Cloudtrail records will be pushed to.
CT_PROJECT_ID (required if CT_STACKDRIVER_NAME or CT_TOPIC_ID are set)
The id of the GCP project that holds the PubSub topic that Cloudtrail records will be pushed to.
CT_KINESIS_STREAM (required if neither CT_STACKDRIVER_NAME or CT_TOPIC_ID are set)
The name of the Kinesis stream that Cloudtrail records will be pushed to.
CT_KINESIS_REGION (required if CT_KINESIS_STREAM is set)
The region that the Kinesis stream lives in.
Role to assume for use by the s3 client.
Useful when this Lambda function and the S3 bucket with CloudTrail logs are in different AWS accounts.
The type of event that will be sent to the Lambda function. Default is
To use the SNS event handler, set
CT_DEBUG_LOGGING=1 will enable debug logging within the handler.
The number of records in a batched put to the Kinesis stream.
CT_KINESIS_BATCH_SIZE is set to
500 (which is the max allowed).
Comma-separated list of
eventSource:eventName that will be filtered out.
The structure of this project is based off of this AWS tutorial: https://docs.aws.amazon.com/lambda/latest/dg/with-cloudtrail.html