An example of a complete flow monitoring system
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ansible
backend
cep
frontend
middleend
shared
zeromq_to_elasticsearch
.gitignore
README.asciidoc
kibana-dashboards.json
pom.xml

README.asciidoc

An example of a complete monitoring system.

Detailed description of all components can be found here in French.

Vagrant installation

You can use vagrant and ansible scripts to create a virtual machine with all application configured.

Manual installation

Requirements

To run all the elements you will need:

Installation

After all the required elements are installed:

  • Create the index in ElasticSearch (see below)

  • Run mvn install in the root directory to build all the Java components

  • Run bundle install in the frontend directory to install all the frontend’s dependencies

  • Optionally import the kibana-dashboards.json data in the .kibana index in Elasticsearch to get a sample dashboard

Starting the applications

First start the external components:

  • Redis

  • Elasticsearch

  • Kibana

Check each application contains the information to start it:

Setup the index in ElasticSearch

Even if Elasticsearch can auto-create indexes, it’s best to setup it manually so fields are analyzed the right way:

curl -XPOST 'http://localhost:9200/monitoring' -d '{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "cep_to_elasticsearch": {
      "properties": {
        "correlation_id": {
          "type": "string",
          "index": "not_analyzed"
        },
        "module_type": {
          "type": "string",
          "index": "not_analyzed",
          "doc_values": true
        },
        "module_id": {
          "type": "string",
          "index": "not_analyzed"
        },
        "endpoint": {
          "type": "string",
          "index": "not_analyzed",
          "doc_values": true
        },
        "message_type": {
          "type": "string",
          "index": "not_analyzed",
          "doc_values": true
        },
        "timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "begin_timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "end_timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "elapsed_time": {
          "type": "double"
        },
        "service_params": {
          "type": "object"
        },
        "headers": {
          "type": "object"
        },
        "result": {
          "type": "object"
        }
      }
    },
    "throttling_violation": {
      "properties": {
        "alert_type": {
          "type": "string",
          "index": "not_analyzed"
        },
        "timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "avg_cnt": {
          "type": "double"

        },
        "correlation_id": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    },
    "unit_sla_violation": {
      "properties": {
        "alert_type": {
          "type": "string",
          "index": "not_analyzed"
        },
        "timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "module": {
          "type": "string",
          "index": "not_analyzed"
        },
        "time": {
          "type": "double"
        }
      }
    },
    "global_sla_violation": {
      "properties": {
        "alert_type": {
          "type": "string",
          "index": "not_analyzed"
        },
        "timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "correlation_id": {
          "type": "string",
          "index": "not_analyzed"
        },
        "count": {
          "type": "double"
        }
      }
    }
  }
}'

Monitoring message format

{
    "correlation_id": "octo.local_MonitoringBase_24389_2015-01-30 11:05:29 UTC_36cddd01-7bcd-4ced-8024-919ff1dbe6ca",  // correlation id
    "timestamp": "2015-01-30T12:05:29.230+01:00", // message timestamp

    "module_type": "FrontendApp", // module type sending the message
    "module_id": "FrontendApp_octo.local_001", // module identifier
    "endpoint": "GET /messages",
    "message_type": "Send message to backend", // message type

    "begin_timestamp": "2015-02-19T22:11:15.939+01:00", // optional: timestamp of the beginning of the current action
    "end_timestamp": "2015-02-19T22:11:15.959+01:00", // optional: timestamp of the end of the current action
    "elapsed_time": 0.020169, // optional: elapsed time of the current action in second

    "service_params": {
        // optional: service parameters
    },

    "headers": {
        // optional: current headers
    }

    "result": {
        // optional: result of current action
    }
}

Call service from the command line

If you want to watch the system running from some time you can schedule services call from the command line:

while sleep 5; do curl -X POST --data 'numberOfMessages=10&timeToSpend=1' 'http://localhost:9292/messages' ; done

License

Code © 2015 released under the MIT license.