aws-elk-billing is a combination of configuration snippets and tools to assist with indexing AWS programatic billing access files(CSV's) and visualizing the data using Kibana.
Currently it supports
AWS Cost and Usage Report type, although it might work for other AWS Billing Report Types which contains some extra columns along with all the columns from
AWS Cost and Usage Report.
You can create
AWS Cost and Usage Report at https://console.aws.amazon.com/billing/home#/reports
Make sure that it contains the following dimensions only (Don't include Resource IDs)
- Account Identifiers
- Invoice and Bill Information
- Usage Amount and Unit
- Rates and Costs
- Product Attributes
- Pricing Attributes
- Cost Allocation Tags
There are Four Docker containers.
- priceboard/docker-alpine:kibana (https://github.com/PriceBoardIn/docker-alpine/tree/master/alpine-kibana)
- aws-elk-billing (Refer: Dockerfile of this repository)
Integration among the 4 containers is done with
|Elasticsearch index mapping||
|Indexing Kibana dashboard||
|Indexing Kibana visualisation||
|Indexing Kibana default index (This file is just for reference purpose, we will automate this part eventually)||
|Parsing the aws-billing CSV's and sending to logstash||
|Connecting the dots:
|Integrating all 4 containers||
Clone the Repository and make sure that no process is listening to the ports used by all these dockers.
Set S3 credentials and AWS Billing bucket and directory name
Rename prod.sample.env to
prod.env and provide values for the following keys
S3_BUCKET_NAME = S3 bucket name (Refer the image above)
S3_REPORT_PATH = Report path (Refer the image above)
S3_REPORT_NAME = Report name (Refer the image above)
prod.env is added in
.gitignore so that you don't push your credentials upstream accidentally.
The entire process is automated through scripts and docker. All the components would be downloaded automatically inside your docker
sudo docker-compose up -d
2.1 Use the index pattern as
aws-billing-*and select the time field as
Kibana AWS Billing Dashboardhttp://localhost:5601/app/kibana#/dashboard/AWS-Billing-DashBoard
2.3 For MAC replace localhost with the ip of docker-machine To find IP of docker-machine
docker-machine ip default
sudo docker-compose stop to shutdown all the docker containers.
sudo docker-compose down to shutdown and remove all the files from docker. Note: Next time you do a
docekr-compose up every thing will start from scratch. Use this if you see some problems in your data or ES is timing out.
aws-elk-billingcontainer will take time while running the following two process
- Downloading and extracting AWS Billing report from AWS S3.
- Depending on the size of AWS Billing CSV report
main.gowill take time to index all the data to Elasticsearch via Logstash.
- You can view the dashboard in kibana, even while
main.gois still indexing the data.
- In order to index new data, you'll have to run
docker-compose up -dagain.
We'll love to hear feedback and ideas on how we can make it more useful. Just create an issue.