Splunk for AWS S3 Server Access Logs
This an app I built using Splunk Lab to extract data from AWS S3 Server Access Logs and use that data for graphs and analysis.
Getting your AWS S3 Server Access Logs
You'll need to configure Server Access Logging in AWS S3. Once that's done, you can either pull down many small logfiles directly, or instead perform daily rollup on them with an AWS S3 Rollup app which I built specifically for this purpose.
Either way, you can use the
aws CLI app to download all of your logs into
directory and then concatenate the contents of each directory into a single file for that bucket with something like this:
aws s3 sync s3://my-accesslogs/rollup-day/ logs
for DIR in $(find . -type d); do cat $DIR/* > $DIR.txt; done
for DIR in $(find . -type d); do rm -rfv $DIR/* > $DIR.txt; done
Naturally, this is highly dependent on how you're storing logs.
Starting up Splunk Lab
Next, start up Splunk Lab with this command:
bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-aws-s3-server-accesslogs/master/go.sh)
The script will guide you through various settings you can send to Splunk Lab.
From there, you can go to https://localhost:8000, log into Splunk with the credentials you specified when starting it, and you should be able to search for data or view reports in dashbaords.
Q: I see an error about exceeding "the configured depth_limit"?
A: You'll need to increase that value in
app/limits.conf. You can read more about that here.
Here's how to get in touch with me: