MDA Version 1.2

Updates

1. Implementation of a Queue Consumer task composed of two priority queues, one acting and running as a waiting queue and the last one as an execution queue:

The prior version of this component was leveraging the power of multithreading, that is, each metric (active or not) was associated with a specific thread. However, this implementation translates into high delays and unreasonable exhaustion of computational resources, especially when dealing with numerous monitoring specs, metrics, and resources to fetch data to be monitored and transformed. Due to this problem, the viability and efficiency of this flow were questioned. We concluded that it won't be feasible to continue to invest our time and therefore, we were required to revise this process.

We then reached the point of using these priority queues exploiting the Python Queue library. For each metric, it is associated a field designated as "next_run_at" which will be the feature bound to automatically order the items inside the queues. When a metric is shifted to the execution queue, there are a fixed number of threads to execute the processes at hand. The thread associated will then be responsible to fetch the data from OSM, transform the data retrieved, perform the hash operation and subsequently post the 1. encrypted hash data; 2. raw data to the DL Kafka topic. The final step before this task is concluded is to update the "next_run_at". The inclusion of the waiting queue was because of not running out of threads with metrics which their next run would take a while to reach.

2. Non-integration with Prometheus:

Prometheus revealed not to be a worthwhile tool for MDA, due to the config files needed to execute and his ability to differentiate the different data sources and data intervals. Prometheus will demand that our component will expose a unique endpoint "metrics" configurable with a specific step interval to fetch data and consume it into their system. Since our component will handle various metrics, each one with a different step, this won't be a valid solution.

3. Major revisions on the PostgreSQL database that this component handles. More specifically in the attributes of the tables (config and metric) and in the operations that it supports

MDA Flow

Current architecture of the MDA component