No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Monitoring and Detection of Incidents in Big Data Applications

This project illustrates monitoring, detection and classification of incidents in Big Data applications on a simple example.


We introduce a generic architecture of an incident management system that can be adapted to the specific needs of Cloud-based Big Data analytics applications.

alt text

Following components are included in the prototype:

  • Generic classification database
  • Monitoring agent
  • Detection and classification rule


  • Hong-Linh Truong, Manfred Halper, Characterizing Incidents in Cloud-based IoT Data Analytics, Proceedings of The 42nd IEEE International Conference on Computers, Software and Applications (COMPSAC 2018), Author version, Pre-print long version, Slides


Required Software

The simulation of the Big Data application uses the following Software to simulate the incident used to illustrate the functionality:

Run the identification and classification rule

A short overview of the scenario.

alt text

  1. Install Docker and Docker Compose
  2. Load the project from This is used to simulate the IoT sensors producing data.
  3. This scenario starts different Docker containers for further information refer to the product documentation pipenv run python config.sample.yml
  4. Install hadoop and start the hdfs
  5. Start Apache Nifi and configure
    • Configure a MQTT processer consuming the MQTT stream from the Dockerised components
    • Configure a HDFS processor writing into the local hdfs
    • Configure a message queue between the two processors
  6. Install Elasticsearch
  7. Install Kibana
  8. Install Neo4j
    • The database has to be initialised with the classification graphdatabase.cypher
  9. Install logstash on the server running the Apache Nifi installation
  10. Configure logstash with nifiPipeline.conf
  11. Install python on the server running the rules
  12. Execute the

The log data from the test scenario is fetched and fed via logstash into elasticsearch. The rule then identifies, classifies and writes the incident into elasticsearch where it can be visualised with Kibana.

If an incident is added to the prototyp three changes have to be comitted:

  1. The classification has to be expanded to cover all the specifics of a new incident
  2. A monitoring agent has to be implemented that delivers the necessary instrumentation data
  3. A detecion and classification rule needs to be developed.