Improves Architecture Documentation (#10)

* Adds initial description of the architecture and its components

* Describe the multiple components presented on the architecture

* Translate architecture documentation to PT-BR

.gitignore

@@ -2,3 +2,4 @@ build
 _*
 
 .idea/
+*.mo

source/architecture.rst

@@ -15,13 +15,186 @@ refer to each component's own documentation.
 Components
 ----------
 
-This section should describe - in high level - the components that comprise the solution, giving
-the reader enough detail to understand the responsabilities of the components, the technologies
-involved in its design and implementation and its relation with its peers.
+With the idea of utilizing open-source and consolidated components, as a starting point for the *dojot* IoT middleware we
+adopted the european project Fiware (FIWARE, 2016). The solutions developed from this framework followed a micro-services
+architecture, having as the main component a context broker that is responsible for redistributing events among services
+that are part of the middleware.
+
+The first architecture proposal took into account a basic group of Fiware services together with new services, developed
+in the scope of this project, having as the main purpose, increasing the usability of the *dojot* platform. This initial
+architecture can be seen on :numref:`old_arch`.
+
+.. _old_arch:
+.. figure:: images/old_architecture.png
+    :width: 100%
+    :align: center
+    :alt: Initial Architecture based on the Fiware project
+
+    Initial Architecture
+
+In this proposal we utilize the following components from Fiware:
+
+    * **Orion**: a context broker used as the communication bus for all the internal services of the middleware
+    * **STH**: the history service used for storing IoT devices data in a MongoDB database
+    * **Perseo CEP**: the service that is responsible for treating events in real time
+    * **Iot-agent**: service used as an abstraction layer for integrating MQTT and HTTP devices
+
+To this services we added the Kong API Gateway to act as a centralized point-of-access removing the need of direct
+communication with each one of the services, an orchestration service to abstract the middleware configuration, an
+authentication service to validate user access credentials and also an user application with graphical interface with
+the purpose of managing the middleware (users, devices and data flows management).
+
+Considering this architecture the basic utilization flow is as follows: The user configures IoT devices through the
+GUI or directly using the REST APIs provided by the API Gateway, he also configures processing flows to deal with the
+data generated by the configured devices. As an example we can generate notifications when the data of a device has one
+of its values reaching a threshold or we can add an entry to a database when a device enters or leaves an specific
+geographic location. This user operations on the API result in configurations across the internal services of the
+middleware (Broker, CEP and iot-agent), being partially abstracted by the orchestration service.
+
+The user contexts are isolated and there is no data sharing, the access credentials are validated by the authorization
+service for each and every operation (API Request). Once the devices and the flows are configured, the iot-agent is
+capable of mapping the data received from devices, encapsulated on MQTT for example, and send then to the context broker
+for internal distribution, reaching, for instance, the history service so it can persist the data on a database and the
+CEP for processing it based on rules. If certain conditions are matched when rules are being processed, a new event is
+generated and sent to the broker service to be redistributed to the interested services.
+
+This architecture made possible the validation of ideas and limitations and possible improvements were identified,
+converging to a reviewed architectural proposition as described on :numref:`new_arch`. This new proposal is under
+development and should become part of the solution.
+
+.. _new_arch:
+.. figure:: images/new_architecture.png
+    :width: 100%
+    :align: center
+    :alt: Revised *dojot* Architecture
+
+    Revised Architecture
+
+More detailed and updated information can be found on the `dojot Github repository <https://github.com/dojot>`_.
+
+Each one of the components that are part of the architecture are briefly described on the sub-sections below.
+
+Kafka + Subscription Manager + NGSI
+***********************************
+
+Apache Kafka is a distributed messaging platform that can be used by applications which need to stream data or
+consume/produce data pipelines. In contrast to what Orion is, Kafka seems to be more appropriate to fulfil
+*dojot*'s architectural requirements (responsibility isolation, simplicity, and so on). And using it with a
+subscription manager and a NGSI interface translator, we can compose an entity which is very close to
+the features offered by Orion, in addition to improved speed and easier scalability.
+
+In Kafka, a specialized topics structure is used to insure isolation between different users and applications data,
+enabling a multi-tenant infrastructure.
+
+The subscription manager service makes use of an in-memory database for efficiency. It adds context to Apache Kafka,
+making it possible that internal or even external services are able to subscribe or query data based on context. The
+subscription manager is also a distributed service to avoid it being a single point of failure or even a bottleneck for
+the architecture.
+
+To keep a certain level of compatibility with Fiware services, meaning, for using Fiware services and components in the
+*dojot* platform with the minimal amount of adaptations, we added a NGSI interface translation service.
+
+Device Manager
+**************
+
+The IoT Device manager is a core entity, responsible for maintaining the devices data models and its abstractions, it is
+also responsible for propagating this models to services that are interested in this kind of information, for example,
+the iot-agent.
+
+This service is stateless, having its data persisted to a database, with data isolation for users and applications,
+making possible a multi-tenant architecture for the middleware.
+
+Iot-agent
+*********
+
+The iot-agent is an adaptation service between the data model of the context broker and the devices data models. The
+*dojot* platform can have multiple iot-agents, each one of them being specialized in a specific protocol like, for
+instance, MQTT/JSON, CoAP/LWM2M and HTTP/JSON.
+
+Security measures like the management of the secure channel used for the communication of the platform with the device
+is also treated by this service.
+
+User Authorization Service
+**************************
+
+This service is responsible for managing user profiles and access control. Basically any API call that reaches the
+platform via the API Gateway is validated by this service.
+
+To be able to deal with a high volume of authorization calls, it uses caching, it is stateless and it is scalable
+horizontally. Its data is stored on a database.
+
+Service Orchestrator
+********************
+
+This service provides a high level API for configuring the *dojot* with the objective of reducing the need of knowing
+how to handle each one of the services that are part of the platform. More specifically, it is responsible for modeling
+different services, exposing a simplified configuration interface and propagating this configuration to the various
+services when requested. It acts as a centralized configuration manager for multiple services.
+
+Fiware Perseo
+**************
+
+The CEP service is responsible for analysing in real time the data processing flows for selected events and triggering
+actions when specific conditions or thresholds are reached. This component is used for creating notification events from
+the pure data that is incoming from the IoT devices. It is integrated with the platform through the context broker and
+its configuration is abstracted by the service orchestrator.
+
+History (Logstash)
+******************
+
+The Logstash connects to the context broker and works as a pipeline for data and events that must be persisted on a
+database. The data is converted into an storage structure and is sent to the corresponding database.
+
+For internal storage, the MongoDB non-relational database is being used, it allows a Sharded Cluster configuration that
+may be required according to the use case.
+
+The data may also be directed to databases that are external do the *dojot* platform, requiring only a proper
+configuration of Logstash and the data model to be used.
+
+Logging and Auditing Service
+****************************
+
+All the services that are part of the *dojot* platform generate usage metrics of its resources and make then available
+to the logging and auditing service, which process this registers and summarize then based on users and applications.
+
+The consolidated data is presented back to the services, allowing then, for example, to expose this data to the user via
+a graphical interface, to limit the usage of the system based on resource consumption and quotas associated with users
+or even to be used by billing services to charge users for the utilization of the platform.
+
+Kong API Gateway
+****************
+
+The Kong API Gateways is used as the entry point for applications and external services to reach the services that are
+internal to the dojot platform, resulting in multiple advantages like, for instance, single access point and ease when
+applying rules over the API calls like traffic rate limitation and access control.
+
+Management Application
+**********************
+
+Web Application responsible for providing responsive interfaces to manage the *dojot* platform, including
+functionalities like:
+
+* **User Profile Management**: define profiles and the API permission associated to those profiles
+* **User Management**: Creation, Visualization, Edition and Deletion Operations
+* **Applications Management**: Creation, Visualization, Edition and Deletion Operations
+* **Device Models Management**: Creation, Visualization, Edition and Deletion Operations
+* **Devices Management**: Creation, Visualization (real time data), Edition and Deletion Operations
+* **Processing Flows Management**: Creation, Visualization, Edition and Deletion Operations
+
+Elastic Service Controller
+**************************
+
+This is a service specialized for cloud environments, that is capable of monitoring the utilization of the platform,
+being able to increase or decrease its storage and processing capacity in an dynamic and automatic fashion to adapt to
+the variability on the demand.
+
+This controller depends that the dojot platform services are horizontally scalable, as well as the databases must
+be clusterizable, which match with the adopted architecture.
 
 Infrastructure
 --------------
 
+**TODO**:
 This section should describe the components that are used as ready-made pieces of working software
 that compose the solution, but have no implementation specific to the project. Relevant topics that
 might be discussed here are:
@@ -34,13 +207,15 @@ might be discussed here are:
 Communications
 --------------
 
+**TODO**:
 This section should provide the reader with the communication strategy used to bind together the
 components that comprise the solution, as well as the interfaces (protocols, serialization formats)
 available to the applications and devices developers.
 
 Deployment strategies
 ---------------------
 
+**TODO**:
 This section should list the deployment requirements and implementation decisions made to satisfy
 those requirements. "Why orchestrator platform 'x'?", "How can this be deployed on commercial cloud
 environments?", "How can this be deployed on stand-alone environments?" are all questions that
@@ -49,6 +224,7 @@ should be answered here.
 Comparative analysis
 --------------------
 
+**TODO**:
 This section should detail the features that differenciate the platform from a "stock" deployment
 of fiware, as well as a feature summary comparing the proposed solution with a reduced set of
 third-party implementations of IoT platforms available.

source/conf.py

@@ -180,3 +180,5 @@
 
 #
 locale_dirs = ['locale/'] # path is example but recommended
+
+numfig = True