Skip to content
Generate Splunk Dashboards for your AWS S3 Server Access Logs
Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app
bin
img
.dockerignore
.gitignore
CODE_OF_CONDUCT.md
Dockerfile-splunk
LICENSE
README.md
go.sh
user-prefs.conf

README.md

Splunk for AWS S3 Server Access Logs

This an app I built using Splunk Lab to extract data from AWS S3 Server Access Logs and use that data for graphs and analysis.

Getting your AWS S3 Server Access Logs

You'll need to configure Server Access Logging in AWS S3. Once that's done, you can either pull down many small logfiles directly, or instead perform daily rollup on them with an AWS S3 Rollup app which I built specifically for this purpose.

Either way, you can use the aws CLI app to download all of your logs into logs/ directory and then concatenate the contents of each directory into a single file for that bucket with something like this:

  • aws s3 sync s3://my-accesslogs/rollup-day/ logs
  • cd logs/
  • for DIR in $(find . -type d); do cat $DIR/* > $DIR.txt; done
  • for DIR in $(find . -type d); do rm -rfv $DIR/* > $DIR.txt; done

Naturally, this is highly dependent on how you're storing logs.

Starting up Splunk Lab

Next, start up Splunk Lab with this command:

  • bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-aws-s3-server-accesslogs/master/go.sh)

The script will guide you through various settings you can send to Splunk Lab.

From there, you can go to https://localhost:8000, log into Splunk with the credentials you specified when starting it, and you should be able to search for data or view reports in dashbaords.

Known Issues

Q: I see an error about exceeding "the configured depth_limit"?

A: You'll need to increase that value in app/limits.conf. You can read more about that here.

Development

  • ./bin/devel.sh splunk
  • ./bin/build.sh
  • ./bin/push.sh

Additional Resources

Credits

Bugs/Contact

Here's how to get in touch with me:

You can’t perform that action at this time.