[cygnus-ngsi][OrionStatsSink] Create

[frbattid]

What about a sink receiving (certain) Orion notifications and (online) computing some kind of simple statistics for each entity's (numerical) attribute, in order to update new entities at Orion? This could the OrionStatsSink :)

More details:

• This sink will only work with a subset of the notifications, not all. This is because the statistics computing may result in a very time consuming task, requiring a lot of memory resources (although we'll try to avoid that point). Which notifications? This is clearly a new configuration parameter. Or be part of a global filtering system deciding which sink will receive each notification.
• In order to reduce the computation time and the resources usage, the sink must be able to compute the statistics online:
  • Maximum: if notified_value > max then max = notified_value. Please observe max=null at initial time, which denotes no value has been notified yet and the first notified value is directly assigned to max.
  • The same for the minimum.
  • The average and variation/standard deviation can be computed online as explained here and here, respectively.
  • See this link for other statiscal measures.
  • If sampling, maybe the distribution could be figured out.
• This should apply only to numerical attributes. Orion data types are merely descriptive, thus figuring out which attributes are numerical or not is a difficult task. Nevertheless, as previousl said, only those entity's attributes specified by the user will be used... thus let's hope the user chooses well :)

Regarding output entities to be updated in Orion, certain naming conventions should be used, e.g. if the entity's name is some_entity_name then the output entity could be named some_entity_name_stats. Other possibility could be to add the statistic values as metadata of the original entity's attribute.

In general, this is an online way of obtaining a subset of all the statistics that the Short-Term historic will be able to generate from previously stored batches.

[frbattid]

Since this is an experimental sink, not required by FIWARE nor the IoT Platform, I'll develop it in parallel to real backlog, using my free time ;)

[fgalan]

Good idea!

Orion implements an "/statistics" operation (see https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Publish/Subscribe_Broker_-_Orion_Context_Broker_-_Installation_and_Administration_Guide#Statistics) but this OrionStatsSink would be a very useful complement.

[frbattid]

What do you think about feeding Orion? Is it better to update a new "<original_entity_id>_stats" sufixed entity containing the statistics? Or could we take advantage of the metadata and add the statistics to the already existing entity?

[fgalan]

I really love the idea of using attribute metadata for attach statistics to Orion entities.

[frbattid]

Some resources from a previous trial; the following ones can be directly reused:

• OrionBackend interface --> 9519732
• OrionBackendImpl --> 774a9c3
• OrionBackendImplTest --> 932ff64
• OrionStats --> ce451e2
• OrionStatsTest --> ac005bd

This can be used as an inspiration of what must be done regarding the sink, but must be adapted to the latests changes in hte core (batches, data_model, etc):

• OrionStatsSink --> 6877856