Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
README.md
ncedc-earthquakes-dashboards.json
ncedc-earthquakes-dataset.tar.gz Repo Restructure Jul 7, 2017
ncedc-earthquakes-filebeat.yml
ncedc-earthquakes-logstash.conf Repo Restructure Jul 7, 2017
ncedc-earthquakes-pipeline.json
ncedc-earthquakes-screenshot.png Repo Restructure Jul 7, 2017
ncedc-earthquakes-template.json Earthquake updated to 6.0 Nov 14, 2017

README.md

Earthquakes Demo

This example provides sample files to ingest, analyze and visualize earthquake data using the Elastic Stack. You may refer to Earthquake data with the Elastic Stack blog post to find out your own story behind the data.

Since its initial publishing, this example has been modified to use Filebeat+Ingest Node instead of Logstash for data ingestion. The data distributed with this example is for 2016. More recent data can be obtained from the links below.

Version Requirements

The example has been tested in the following versions:

  • Elasticsearch 6.0
  • Filebeat 6.0
  • Kibana 6.0

Datasets

The earthquake datasets are gathered from the Northern California Earthquake Data Center through the ANSS Composite Catalog Search.

Acknowledgement

"Waveform data, metadata, or data products for this study were accessed through the Northern California Earthquake Data Center (NCEDC), doi:10.7932/NCEDC."

Earthquakes

Filename: earthquakes.txt
Search parameters: catalog=ANSS, start_time=2016/01/01,00:00:00, end_time=2016/11/14,13:29:52, minimum_magnitude=1, maximum_magnitude=10, event_type=E
Size: 38095 lines (3031148 bytes)

Blasts (Quarry or Nuclear)

Filename: blasts.txt
Search parameters: catalog=ANSS, start_time=2016/01/01,00:00:00, end_time=2016/11/14,13:30:59, minimum_magnitude=1, maximum_magnitude=10, event_type=B
Size: 222 lines (17837 bytes)

Installation & Setup

  • Follow the Installation & Setup Guide to install and test the Elastic Stack (you can skip this step if you have a working installation of the Elastic Stack,)

  • Run Elasticsearch & Kibana

<path_to_elasticsearch_root_dir>/bin/elasticsearch
<path_to_kibana_root_dir>/bin/kibana
  • Check that Elasticsearch and Kibana are up and running.
    • Open localhost:9200 in web browser -- should return status code 200
    • Open localhost:5601 in web browser -- should display Kibana UI.

Note: By default, Elasticsearch runs on port 9200, and Kibana run on ports 5601. If you changed the default ports, change the above calls to use appropriate ports.

  • Download and install Filebeat as described here. Do not start Filebeat

Download Example Files

Download the following files in this repo to a local directory:

  • ncedc-earthquakes-dataset.tar.gz - sample data (in csv format)
  • ncedc-earthquakes-filebeat.yml - Filebeat config for ingesting data into Elasticsearch
  • ncedc-earthquakes-template.json - template for custom mapping of fields
  • ncedc-earthquakes-pipeline.json - ingest pipeline for processing documents produced by Filebeat
  • ncedc-earthquakes-dashboards.json - config file to load prebuilt creating Kibana dashboard

Unfortunately, Github does not provide a convenient one-click option to download entire contents of a subfolder in a repo. You can either (a) download or clone the entire examples repo and navigate to Exploring Public Datasets/nyc_traffic_accidents subfolder, or (b) individually download the above files. The code below makes option (b) a little easier:

wget https://raw.githubusercontent.com/elastic/examples/master/Exploring%20Public%20Datasets/earthquakes/ncedc-earthquakes-dataset.tar.gz
wget https://raw.githubusercontent.com/elastic/examples/master/Exploring%20Public%20Datasets/earthquakes/ncedc-earthquakes-template.json
wget https://raw.githubusercontent.com/elastic/examples/master/Exploring%20Public%20Datasets/earthquakes/ncedc-earthquakes-pipeline.json
wget https://raw.githubusercontent.com/elastic/examples/master/Exploring%20Public%20Datasets/earthquakes/ncedc-earthquakes-dashboards.json
wget https://raw.githubusercontent.com/elastic/examples/master/Exploring%20Public%20Datasets/earthquakes/ncedc-earthquakes-filebeat.yml

Ingest Data

  1. Extract the dataset archive with tar -zxf ncedc-earthquakes-dataset.tar.gz from the terminal.

  2. Install the earthquakes ingest pipeline i.e.

    curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_ingest/pipeline/ncedc-earthquakes' -d @ncedc-earthquakes-pipeline.json
  3. Install the index template i.e.

    curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_template/ncedc-earthquakes' -d @ncedc-earthquakes-template.json
  4. Modify the ncedc-earthquakes-filebeat.conf file as follows:

    • The parameter hosts: ["localhost:9200"] in case your are not running Elasticsearch node on your local host

    • The files path to each of the data files extracted above i.e.

        paths:
          - ./ncedc-earthquakes-dataset/earthquakes.txt

      and

        paths:
          - ./ncedc-earthquakes-dataset/blasts.txt
  5. Move the file ncedc-earthquakes-filebeat.yml to the Filebeat installation directory i.e.

    mv ncedc-earthquakes-filebeat.yml <filebeat_installation_dir>/ncedc-earthquakes-filebeat.yml
  6. Start Filebeat to begin ingesting data to Elasticsearch

    cd <filebeat_installation_dir>
    ./filebeat -e -c ncedc-earthquakes-filebeat.yml

    Note: Expect the following error to be repeated twice

    WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"number_format_exception","reason":"empty String"}}

    This simply represents the csv headers not conforming to the required field types. These documents are not required and the above can be safely ignored.

  7. After several minutes repeat the following commands until a count of 38315 is returned:

    curl http://localhost:9200/ncedc-earthquakes-*/_refresh
    curl http://localhost:9200/ncedc-earthquakes-*/_count

Importing Kibana Visuals and Dashboards

  • Access Kibana by going to http://localhost:5601 in a web browser
  • Connect Kibana to the ncedc-earthquakes-* indices in Elasticsearch (autocreated in step 1)
    • Click the Management tab >> Index Patterns tab >> Create New. Specify ncedc-earthquakes-* as the index pattern name and click Create to define the index pattern. (Leave the Use event times to create index names box unchecked and use @timestamp as the Time Field)
    • If this is the only index pattern declared, you will also need to select the star in the top upper right to ensure a default is defined.
  • Load sample dashboard into Kibana
    • Click the Management tab >> Saved Objects tab >> Import, and select ncedc-earthquakes-dashboard.json.
    • On import you will be asked to overwrite existing objects - select "Yes, overwrite all". Additionally, select the index pattern "ncedc-earthquakes-*" when asked to specify a index pattern for the dashboards.
  • Open dashboard
    • Click on Dashboard tab and open the Earthquake dashboard

Dashboard Screenshot