Merge pull request #13 from mafi-mcfly/monitoring

Monitoring

docs/source/monitoring/fault_detection.rst

@@ -0,0 +1,47 @@
+Fault Detection
+===============
+
+The Fault Detection components monitors every data stream that is available to the IoTCrawler framework. In the first layer, the component categorises faults as definite faults (faults that occur due to hardware issues) or as anomalies, which could occur because of a brief environmental factor, an unexpected behaviour detected through learned patterns and contextual information. These anomalies can be categorised as faults, if they persist for a longer period of time.
+
+As the detection of faults requires different approaches in different scenarios, not every data stream can have the best detector suited to its needs. To cater to the needs of most of the sensor streams, the fault detection component uses two approaches, ARIMA based fault detection with time series analysis and a Dirichlet Process Gaussian State Model, which determines the likelihood for a value to occur based on the previous observations of the sensor.
+
+Workflow
+--------
+
+The Fault Detection component subscribes to any new data stream that becomes available through the MDR. Through the metadata, the Fault Detection will determine which approach should be used. This is differentiated based on how much information is provided in metadata. For example, if seasonality/periodicity of a data stream is defined and the historical data is available for the desired duration, then ARIMA can be applied.
+
+For the actual detection, the component will make use of the trained models and the corresponding QoI values for each data stream, calculated for each stream by the Semantic Enrichment. It will then notify the MDR in case of faults.
+
+.. image:: ../images/fault_detection.png
+   :alt: Fault Detection component
+
+Stop a Fault Detection Instance
+-------------------------------
+
+The Fault Detection is an autonomous component and does not require human supervision. However, because it can not be used in every scenario, e.g. if it detects a lot of false positives, it can be manually stopped for an individual data stream. Stopping the Fault Detection is not performed autonomously in any case as the ground truth is not known in most scenario and hence, an accurate decision cannot be made. This method allows a manual intervention, e.g. by an administrator of an IoTCrawler instance.
+
+.. http:get:: /api/stopFaultDetection
+
+with payload
+
+::
+
+  {
+    "sensorID" : <The ID of the sensor to stop>
+  }
+
+
+Restart a Fault Detection Instance
+----------------------------------
+
+A need to restart a Fault Detection instance may arise when the information related to the stream is changed. This change in information may happen due to the change of sensor device, correction of human error or some added insights about the data from the provider. This will require the Fault Detection instance to be restarted to re-evaluate the best available approach, reset the learning parameters and to train on relevant historical data.
+
+.. http:get:: /api/restartFaultDetection
+
+with payload
+
+::
+
+  {
+    "sensorID" : <The ID of the sensor to restart>
+  }

docs/source/monitoring/fault_recovery.rst

@@ -0,0 +1,3 @@
+Fault Recovery
+==============
+

docs/source/monitoring/monitoring.rst

@@ -1,9 +1,22 @@
 Monitoring
 ==========
 
-Here is our documentation for the Monitoring component:
+The main task of the Monitoring component in IoTCrawler is to monitor the connected data streams in order to detect faulty/anomalous data samples.
+
+The subcomponent **Fault Detection** is responsible to detect 'unusual behaviour' in the data streams and determine if a fault is present. In this case, counter measures are triggered, including recovery mechanisms to provide a quick response by imputing single *StreamObservations* or deploying a Virtual Sensor. To detect faults in a single data stream the **Fault Detecttion** uses a modified Dirichlet Process Gaussian State Machine Model and a ARIMA-based approach. The **Fault Recovery** utilises Markov Chain Monte Carlo (MCMC) methods to generate sensor samples.
+
+In case a faulty data stream is detected, the subcomponent **Virtual Sensor Creator** is able to deploy a Virtual Sensor as a counter measure. It uses machine learning techniques to create sensor samples in relation to correlating, neighbouring sensors.
+
+The following figure shows an overview of the **Monitoring** component and the interactions with other IoTCrawler components. For more details see the respective subsections.
+
+.. image:: ../images/monitoring.png
+   :alt: Monitoring component
+
+The documentation for the **Monitoring's** subcomponents can be found here:
 
 .. toctree::
    :maxdepth: 1
 
-   vs_creator
+   fault_detection
+   fault_recovery
+   vs_creator

docs/source/monitoring/vs_creator.rst

@@ -58,6 +58,29 @@ The :red:`data` field in the response will contain the NGSI-LD description for t
 
 **Please note:**  although this API method is named *replace* BrokenSensor, the original sensor is not removed from the IoTCrawler system.
 
+Deploy a new Virtual Sensor
+---------------------------
+
+:red:`This method is not fully implemented!`
+
+Instead of replacing a broken sensor this component also allows to deploy a Virtual Sensor for locations, where no real sensor is present. For this the following API method can be used. It must include the parameter **location** where the new Virtual Sensor shall be deployed as well as the ID of an *ObservableProperty* (parameter **propertyID**) to indicate what physical phenomena shall be 'measured'. The rest of the optional parameters have the same meaning as for the replaceBrokenSensor method. The Virtual Sensor will use linear regression to predict new *StreamObservations*.
+
+.. http:get:: /api/deployVirtualSensor/
+
+with payload
+
+::
+
+    {
+      "location" : [<latitude>, <longitude>, <altitude>],
+      "propertyID": <The ID of an ObservableProperty the new sensor shall measure>
+      "maxDistance": <The maximum distance in meter to search for contributing sensors. Integer, default 20000.>,
+      "onlySameObservations": <Consider only sensors that measure the same observation as the broken sensor. Boolean, default true.>,
+      "limitSourceSensors": <Limit the number of contributing sensors. Interger, default 8.>,
+      "updateInterval": <Number of seconds, in which the Virtual Sensor shall predict a new value>
+    }
+
+
 Stop a Virtual Sensor
 ---------------------
 This method can be used to stop an previously deployed Virtual Sensor, identified by its ID as returned by the deployment methode. If no Virtual Sensor with the given ID (parameter **sensorID**) exists this method returns an error. Otherwise the Virtual Sensor is stopped and the Virtual Sensor Creator component will delete the related entries in MDR, such as *Sensor*, *Platform* or *StreamObservations*. The un-deployment is triggered by a HTTP request to: