CTGrazer is code you can use to create an AWS Lambda Function that will collect all of your AWS CloudTrail logs and efficiently send them to your Splunk HEC (HTTP Event Collector) server.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
img
src
DCO
LICENSE
README.md

README.md

CTGrazer

CTGrazer

CTGrazer is code you can use to create an AWS Lambda Function that will collect all of your AWS CloudTrail logs and efficiently send them to your Splunk HEC (HTTP Event Collector) server.

Why?

Using CTGrazer to port your AWS CloudTrail logs into Splunk has many advantages

  • Speed CloudTrail logs are processed as soon as they become available
  • Security All data is encrypted in transit and it does not rely on AWS IAM Access Keys
  • Scalable CTGrazer will automatically scale up and down according to your needs
  • Reliable If CTGrazer can't get your logs to their destination, it will automatically retry until it can
  • Cost Effectiveness Pulling in 400K objects a month will cost you about the same as a cheeseburger!

Pre-requisites

AWS Splunk Python
Execution Permissions:
Put Trigger for S3 Bucket
HTTP Event Collector (HEC) :
Authentication Token
Using Pip - pip install requests
If pip not allowed, direct download of all modules -

Splunk Cloud | HEC VPC Configurations

If you are a Splunk Cloud Customer and plan to use splunk indexers as HEC endpoint, you should NOT set any VPC settings for Lambda to be able to stream data to your indexers. Only setting required here is to open inbound SG's on Splunk Cloud to allow this traffic. If no VPC Settings are enabled, Lambda uses it's own endpoint which has internet access by default to connect to resources outside of your own VPC.

Splunk Internal | HEC VPC Configurations

For Internal HEC endpoints, VPC , subnets and SG rules need to be applied.

Configure, build and install

1. Project Structure

CTGrazer Project Structure

2. Download Requests Module

CTGrazer uses Requests Module to perform HTTP calls. Download python Requests module and copy them under /lib folder.

3. Config.ini

[REQUIRED PARAMETERS]

Parameter Description
aws_s3_bucket_name Specify an AWS S3 bucket name to collect the CloudTrail logs
aws_s3_bucket_prefix Specify the S3 subfolder location where the CloudTrail file object(s) are location
splunk_hec_endpoint Specify splunk HTTP Event Collector endpoint of format https://[HEC_HOST]:[PORT]/services/collector/event
splunk_hec_key Specify splunk HTTP Event Collector Key
splunk_source_type Specify a value for the splunk sourcetype

[OPTIONAL PARAMETERS]

Parameter Description
batch_thread_size Specify the number threads to be used for event batch
retry_sleep_time Time (in seconds) to sleep. Used when event is triggered but the object isn't there yet. Sleeps for the time specified and retries to get the object from S3
minutes_to_process Time (in minutes) to process events that are older than the below specified minutes
log_destination SPLUNK OR CLOUDWATCH (Default: CLOUDWATCH)
log_message_prefix Sets the format for Lambda log processing events.
debug Turn ON or OFF debugging
splunk_debug_sourcetype Splunk sourcetype to use when logging debug messages
4. Deployment Package
  • Create a zip file to be uploaded as a AWS Lambda Function.
  • Use Automation process of your choice for deployment.
  • Configure S3 Put Trigger for the Cloudtrail Bucket - Event Trigger
  • Configure CloudWatch Event Rule (Eg: 30 min) - Scheduled Trigger
  • Configure applicable VPC, Security Group, Role Settings
  • Set Memory and Timeout limits (256MB , 5 mins)