Updates (mrl)

5/31/2018: Added support for ES6+ strict content types. (Ref: https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests)

CloudFront log viewer

Import CloudFront logs into a local Elasticsearch instance and visualize them using Kibana

Start a local Elasticsearch instance

Set environment variables for Elasticsearch

export ES_ROOT=~/elasticsearch
export ES_PORT=9201
export ES_VERSION=6.0.0-rc1

Download Elasticsearch and untar into $ES_ROOT - this needs to be done once

\rm -rf $ES_ROOT
mkdir -p $ES_ROOT
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ES_VERSION.tar.gz -O $ES_ROOT/elasticsearch.tar.gz
tar xzf $ES_ROOT/elasticsearch.tar.gz -C $ES_ROOT
mv $ES_ROOT/elasticsearch-$ES_VERSION $ES_ROOT/elasticsearch

Start instance

$ES_ROOT/elasticsearch/bin/elasticsearch -E http.port=$ES_PORT

Test connection via curl

curl http://$HOST:$ES_PORT

Import cloudfront logs into local Elasticsearch

python import_logs.py --log_file_pattern 'logs/*' --es_url 'http://$HOST:$ES_PORT' --index cloudfront --type prod --clean_index --verbosity INFO

Start a local Kibana instance

Set environment variables for Kibana

export ESKB_PORT=9211
export ESKB_UNAME=darwin # darwin | linux
export ESKB_VERSION=$ES_VERSION

Download Kibana and untar into $ES_ROOT.tar.gz - this needs to be done once

wget https://artifacts.elastic.co/downloads/kibana/kibana-$ESKB_VERSION-$ESKB_UNAME-x86_64.tar.gz -O $ES_ROOT/kibana.tar.gz
tar xzf $ES_ROOT/kibana.tar.gz -C $ES_ROOT
mv $ES_ROOT/kibana-$ESKB_VERSION-$ESKB_UNAME-x86_64 $ES_ROOT/kibana

Start instance

$ES_ROOT/kibana/bin/kibana --host=$HOST --port=$ESKB_PORT --elasticsearch=http://$HOST:$ES_PORT

License

The source code is available here under MIT licence. Feel free to use any part of the code. Please send any bugs, feedback, complaints, patches to me at varunkumar[dot]n[at]gmail[dot]com.

-- Varun