Components to load vpc flow logs into Elastic Search
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs
extractor
ingestor
LICENSE
README.md
cloudformation.yaml
elk.json
extractor.zip

README.md

aws-vpcflowlogloader

Components to load vpc flow logs into Elastic Search based on this blog post

Series

This repository is part of a series which aims to create security monitoring in an AWS account with AWS 'native' solutions.

Setup

The cloudformation template in this repo will deploy a cloudwatch loggroup subscription, two lambda's, firehose stream and corresponding IAM entities. It will not create an elasticsearch domain. I assume you will be using a central one for monitoring and have already created it.

The structure (flow) of the application is still the same: infra

Lambdas

There are two lambda functions

  1. Ingesting the flowlog data into firehose
  2. Enriching the flowlog data with geo information and sg-ids.

Adding the code

cd ingestor && zip ../ingestor.zip index.js
aws s3api put-object --key lambdafunctions/ingestor.zip --body ../ingestor.zip --bucket mycodebucket

cd ../extractor && npm install
zip -r ../extractor.zip *
aws s3api put-object --key lambdafunctions/extractor.zip --body ../extractor.zip --bucket mycodebucket

Dashboard

Dashboard The dashboard will show a quick overview of the network data; Countries, a pie diagram and the rejected vs accepted traffic.

To get the visualizations and dashboard you can import the export I made in the management section of your elastic search domain.

There are a lot more visualization which you can use!