View AWS flow logs using clickhouse and grafana
Fast and not very complicated way to view network logs from AWS. The workflow is VPC Flow Logs -> CloudWatch -> Stream to Lambda -> Send to clickhouse server -> Decompress and ingest to DB -> View in Grafana
- Start by installing clickhouse, grafana, define schema, install clickhouse plugin for grafana and import dashboard.
- Clickhouse - https://clickhouse.yandex/#quick-start
- Schema - In clickhouse/schema
- Grafana plugin - https://grafana.com/plugins/vertamedia-clickhouse-datasource/installation
- Dashboard - grafana/dashboard.json
- Create new Lambda function as in lambda/forwarder.py Adjust IP and PORT
- Add server-side scripts:
- For receiving logs and saving extracted lines: ingest/input.sh
- For batch-inserting logs: ingest/output.sh Add cron job to do this every X minutes
- Enable flow logs in VPC dashboard (Select VPC -> Flow Logs -> Create Flow Log). Enable only logs with state ACCEPT
- In CloudWatch Logs select previously created Log Group and select Stream to Lambda. Select function from step 3.
Sample dashboard with data:
