add Kafka2ftp service and node (#86)

README.md

@@ -35,7 +35,7 @@ pipenv --python 3.8
 Install the dependencies via `pipenv`:
 
 ```shell
-pipenv install --all
+pipenv install
 ```
 
 Enter in the virtual environment:

readthedocs.yml

@@ -7,6 +7,6 @@ formats:
 requirements_file: ./requirements/requirements.txt
 
 python:
-  version: 3.6
+  version: 3.8
   use_system_site_packages: false
   setup_py_install: false

requirements/requirements.txt

@@ -1,6 +1,36 @@
-Sphinx==1.8.5
+alabaster==0.7.12
+argh==0.26.2
+Babel==2.8.0
+certifi==2020.6.20
+chardet==3.0.4
+click==7.1.2
+docutils==0.16
+idna==2.10
+imagesize==1.2.0
+Jinja2==2.11.2
+livereload==2.6.3
+MarkupSafe==1.1.1
+packaging==20.4
+pathtools==0.1.2
+port-for==0.3.1
+Pygments==2.6.1
+pyparsing==2.4.7
+pytz==2020.1
+PyYAML==5.3.1
+requests==2.24.0
+six==1.15.0
+snowballstemmer==2.0.0
+Sphinx==3.2.1
 sphinx-autobuild==0.7.1
-sphinx-intl==1.0.0
+sphinx-intl==2.0.1
 sphinx-rtd-theme==0.5.0
+sphinxcontrib-applehelp==1.0.2
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-htmlhelp==1.0.3
+sphinxcontrib-jsmath==1.0.1
 sphinxcontrib-plantuml==0.18
-sphinxcontrib-websupport==1.1.2
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.4
+tornado==6.0.4
+urllib3==1.25.10
+watchdog==0.10.3

source/architecture.rst

@@ -128,12 +128,29 @@ set of actions. These flows can be extended using external processing blocks
 
 Data Manager
 ************
-This service manages the dojot's data configuration, making possible to import and export configuration.
+
+This service manages the dojot's data **configuration**,
+making possible to import and export configuration
+like templates, devices and flows.
 
 Cron
-**********
-Cron is a dojot's microservice that allows you to schedule events (or requests) to
-be emitted (done) to other microservices inside dojot platform..
+****
+
+Cron is a dojot's microservice that  allows you to schedule events to be emitted - or requests to be sent - to other microservices inside dojot platform.
+
+
+Kafka2Ftp
+*********
+
+The kafka2ftp service provides a connector solution for
+forwarding messages from Apache Kafka to FTP servers.
+It subscribes to the topic ``tenant.dojot.ftp``
+(*tenant* is defined in the environment variable).
+In these topics there are messages with information about the file name,
+encoding format and file content,
+these messages are published by some service or
+the `Publish in FTP topic` node from Flowbroker.
+
 
 Persister/History
 *****************

source/components-and-apis.rst

@@ -84,6 +84,10 @@ Components
     - `GitHub - x509-identity-mgmt`_
     -
     - `API - x509-identity-mgmt`_
+  * - Kafka2Ftp
+    - `GitHub - Kafka2Ftp`_
+    -
+    -
 
 
 Exposed APIs (API Gateway)
@@ -311,41 +315,43 @@ dojot.
 .. _Kafka doc.: http://kafka.apache.org/documentation/
 
 
-.. _GitHub - auth: https://github.com/dojot/auth
-.. _API - auth: https://dojot.github.io/auth/apiary_v0.4.3.html
+.. _GitHub - auth: https://github.com/dojot/auth/tree/v0.5.0
+.. _API - auth: https://dojot.github.io/auth/apiary_v0.5.0.html
 .. _Auth  doc.: http://dojotdocs.readthedocs.io/projects/auth/en/latest/
 .. _Messages - auth: https://dojotdocs.readthedocs.io/projects/auth/en/latest/kafka-messages.html
 
-.. _GitHub - history: https://github.com/dojot/history
-.. _API - history: https://dojot.github.io/history/apiary_v0.4.3.html
+.. _GitHub - history: https://github.com/dojot/history/tree/v0.5.0
+.. _API - history: https://dojot.github.io/history/apiary_v0.5.0.html
 
 
-.. _GitHub - DeviceManager: https://github.com/dojot/device-manager
-.. _API - DeviceManager: https://dojot.github.io/device-manager/apiary_v0.4.3.html
+.. _GitHub - DeviceManager: https://github.com/dojot/device-manager/tree/v0.5.0
+.. _API - DeviceManager: https://dojot.github.io/device-manager/apiary_v0.5.0.html
 .. _DeviceManager doc.: http://dojotdocs.readthedocs.io/projects/DeviceManager/en/latest/
 .. _Messages - DeviceManager: http://dojotdocs.readthedocs.io/projects/DeviceManager/en/latest/kafka-messages.html
 
-.. _GitHub - image-manager: https://github.com/dojot/image-manager
-.. _API - image-manager: https://dojot.github.io/image-manager/apiary_v0.4.3.html
+.. _GitHub - image-manager: https://github.com/dojot/image-manager/tree/v0.5.0
+.. _API - image-manager: https://dojot.github.io/image-manager/apiary_v0.5.0.html
 
 
-.. _GitHub - GUI: https://github.com/dojot/gui
+.. _GitHub - GUI: https://github.com/dojot/gui/tree/v0.5.0
 
 
-.. _GitHub - flowbroker: https://github.com/dojot/flowbroker
-.. _API - flowbroker: https://dojot.github.io/flowbroker/apiary_v0.4.3.html
+.. _GitHub - flowbroker: https://github.com/dojot/flowbroker/tree/v0.5.0
+.. _API - flowbroker: https://dojot.github.io/flowbroker/apiary_v0.5.0.html
 
-.. _GitHub - data-broker: https://github.com/dojot/data-broker
-.. _API - data-broker: https://dojot.github.io/data-broker/apiary_v0.4.3.html
+.. _GitHub - data-broker: https://github.com/dojot/data-broker/tree/v0.5.0
+.. _API - data-broker: https://dojot.github.io/data-broker/apiary_v0.5.0.html
 
 .. _Messages - iotagent-mosca: http://dojotdocs.readthedocs.io/projects/iotagent-mosca/en/latest/operation.html#sending-messages-to-other-components-via-kafka
-.. _GitHub - iotagent-mosca: https://github.com/dojot/iotagent-mosca
+.. _GitHub - iotagent-mosca: https://github.com/dojot/iotagent-mosca/tree/v0.5.0
+
+.. _GitHub - Data Manager: https://github.com/dojot/data-manager/tree/v0.5.0
+.. _API - Data Manager: https://dojot.github.io/data-manager/apiary_v0.5.0.html
 
-.. _GitHub - Data Manager: https://github.com/dojot/data-manager
-.. _API - Data Manager: https://dojot.github.io/data-manager/apiary_v0.4.3.html
+.. _GitHub - Cron: https://github.com/dojot/cron/tree/v0.5.0
+.. _API - Cron: https://dojot.github.io/cron/apiary_v0.5.0.html
 
-.. _GitHub - Cron: https://github.com/dojot/cron
-.. _API - Cron: https://dojot.github.io/cron/apiary_v0.4.3.html
+.. _GitHub - x509-identity-mgmt: https://github.com/dojot/dojot/tree/v0.5.0/x509-identity-mgmt
+.. _API - x509-identity-mgmt: https://dojot.github.io/dojot/x509-identity-mgmt/apiary_v0.5.0.html
 
-.. _GitHub - x509-identity-mgmt: https://github.com/dojot/dojot/tree/master/x509-identity-mgmt
-.. _API - x509-identity-mgmt: https://dojot.github.io/dojot/x509-identity-mgmt/apiary_v0.5.0.html
+.. _GitHub - Kafka2Ftp: https://github.com/dojot/dojot/tree/v0.5.0/connector/kafka2ftp

source/flow.rst

@@ -233,10 +233,9 @@ Ftp request
     :alt: http_node
 
 This node sends a file to a FTP server. When uploading a
-file, its name can be set by setting the "Filename" field in the same way as
-other output variables (it should refer to a variable set in the flow). The
-file encoding will set the file enconding, which could be, for instance,
-"base64" or "utf-8".
+file, its name can be set by the "Filename" field in the same way as
+other output variables (it should refer to a variable set in the flow).
+The file encoding can also be set to, for example, "base64" or "utf-8".
 
 
 .. _ftp-request-panel:
@@ -599,6 +598,67 @@ Fields:
 * **Name** *(optional)*: Name of the node
 
 
+Publish in FTP topic
+********************
+
+.. _kafka2ftp_node:
+.. image:: images/nodes/kafka2ftp_node.png
+    :width: 20%
+    :align: left
+    :alt: kafka2ftp_node
+
+Node to forward messages to Apache Kafka FTP topic.
+
+It publishes to the ``tenant.dojot.ftp`` topic (*tenant* is defined by which tenant the flow belongs to)
+in which the messages are produced with informations about the file name,
+encoding format and file content.
+
+.. _kafka2ftp_node_cfg:
+.. figure:: images/nodes/kafka2ftp_node_cfg.png
+    :width: 50%
+    :align: center
+    :alt: kafka2ftp_node_cfg
+
+    : *Publish in FTP topic* configuration
+
+Fields:
+
+* **Encoding** *(required)*: The encoding that the file to be sent uses. Valid values are: ascii, base64, hex, utf16le, utf8 and binary.
+* **Filename** *(required)*: Variable with the name of the file to be sent.
+* **Content** *(required)*: Variable with the file contents to be sent.
+* **Name** *(optional)*: Name of the node
+
+
+Example of a message sent by this node:
+
+.. code-block:: json
+
+    {
+        "metadata": {
+            "msgId": "33846252-659f-42cc-8831-e2ccb923a702",
+            "ts": 1571858674,
+            "service": "flowbroker",
+            "contentType": "application/vnd.dojot.ftp+json"
+        },
+        "data": {
+            "filename": "filename.jpg",
+            "encoding": "base64",
+            "content": "..."
+        }
+    }
+
+Where the keys above are:
+
+* msgId:  Value of type uuidv4 used to uniquely identify the message in dojot's context.
+* ts: Timestamp in Unix Timestamp (ms) format from the moment the message was produced.
+* service: Name of the service that generated the message.
+* contentType: Type of encoding used by the file.
+* filename: Name of the file to be sent to the FTP server.
+* encoding: Encoding the contents of the file. Valid values are: ascii, base64, hex, utf16le, utf8 and binary.
+* content: File contents.
+
+This can be used with the kafka2ftp component. See more in :doc:`components-and-apis`.
+
 Deprecated nodes
 ****************