Upgrade to version 1.9.0

The main issues included in this release are:

PubSub Server / API REST:

  1. Setting limiting on server level

    It is now possible to limit the hourly rate of incoming HTTP requests,
    as well maximum body length. If activated, the server responds to each
    request with headers indicating the remaining quota.

  2. Rate limiting on entity level

    In addition to global rate limits, rate limiting was also added on entity
    level, i.e. provider and application. Every entity can have its particular
    hourly rates of incoming and outgoing (subscription) HTTP requests. If
    activated, headers indicating the remaining quotas are added in the server
    responses and requests.

  3. Monitor API

    In order to provide a more transparent and accessible overview of the
    system state, a new independent "monitor" API was created. The API is
    accessible at a standalone port. It provides a variety of potentially
    useful information on the Sentilo instance, such as memory, cpu, disk
    usage for every used module, state of the agents, number of events
    processed by every component etc.

Catalog:

  4. Background map with WMS

    As an alternative to Google Maps, OGC WMS support via Leaflet has been
    added. Any WMS layer can be used in catalog maps, if available in the EPSG
    3857 projected coordinate system.

  5. Metrics section

    New section that displays dashboards on the status of the instance. This
    page consumes the new "monitor" API.

Agents:

  6. The new Metrics Monitor agent stores data from the new "monitor" API
     in Elasticsearch.

Clients:

  7. Node-RED library upgrades

    The Node-RED library gained a new node for programmed subscription. Also,
    all nodes provide a new output for status code. This enables more
    fine-grained control of the flow, especially in case of errors. Node-RED
    1.0+ compatibility.

  8. Clients and examples reviewed

    Java, RaspberryPi and HMAC examples have been reviewed and updated. Also,
    the NodeJS client is now compatible with newer versions of NodeJS.

docs/_static/css/styles.css

@@ -2,10 +2,6 @@ table {
 	border-collapse: collapse;
 }
 
-table, th, td {
-	border: 1px solid black;
-}
-
 th {
 	text-align: center;
 	background-color: #F3F3F3;

docs/_static/js/alert_test.js

@@ -0,0 +1 @@
+$("header").html('<div style="width:100%;height:50px;font-size:2em;color:red;margin-top:20px">THIS IS A TEST VERSION. PLEASE VISIT <a href="https://sentilo.readthedocs.io/">https://sentilo.readthedocs.io/</a></div>')

docs/_themes/sentilo_sphinx_rtd_theme/layout.html

@@ -86,7 +86,7 @@
 {% block extrabody %}
 <div class="met_header_wrap met_header_1 met_fixed_header">
 <header class="met_content clearfix">
-  <a href="http://www.sentilo.io/wordpress/" class="met_logo"><img src="http://www.sentilo.io/wordpress/wp-content/uploads/2013/11/sentilo_logo.png" data-retina="http://www.sentilo.io/wordpress/wp-content/uploads/2013/11/sentilo_logo_2x.png" alt="Sentilo" height="34px"></a>
+  <a href="http://www.sentilo.io/wordpress/" class="met_logo"><img src="https://www.sentilo.io/wordpress/wp-content/uploads/2013/11/sentilo_logo.png" data-retina="http://www.sentilo.io/wordpress/wp-content/uploads/2013/11/sentilo_logo_2x.png" alt="Sentilo" height="34px"></a>
 
   <nav class="met_vcenter">
     <ul class="met_clean_list met_vcenter sf-js-enabled sf-arrows">

docs/api_docs.rst

@@ -8,6 +8,8 @@ Contents:
 
    api_docs/general_model
    api_docs/security
+   api_docs/rate_limiting
+   api_docs/monitor_api
    api_docs/services
 
 The Application Programming Interface (API) of Sentilo defines a set of commands,

docs/api_docs/general_model.rst

@@ -78,25 +78,12 @@ and consists of the following parts:
    (e.g. 8081).
 -  **service:** catalog, data, order, etc..
 
-Every service has a custom URL format as especified for each services.
+Every service has a custom URL format as specified for each services.
 
 Representations
 ---------------
 
-Data formats that will support the platform are: **JSON** (initial
-version) and **XML** (in a future releases).
-
-The default format used by the platform is **JSON**.
-
-To specify a different format, you must add the parameter **“format”**
-in the request, as shown in the following example:
-
-::
-
-   http://<your_api_server.com>/service/<id_provider>?format=XML
-
-JSON format
-~~~~~~~~~~~
+Data formats that will supports the platform is currently only  **JSON**.
 
 Example data in JSON format:
 
@@ -106,18 +93,6 @@ Example data in JSON format:
        {"value":"12.3","timestamp":"17/09/2012T12:34:45"}
    ]}
 
-XML format
-~~~~~~~~~~
-
-Example data in XML format:
-
-.. code:: xml
-
-   <observation>
-       <value>12.3</value>
-       <timestamp>17/09/2012T12:34:45</timestamp>
-   </observation>
-
 Operators
 ---------
 
@@ -135,7 +110,7 @@ are:
 used and by the service, provider or sensor specified in the URL
 invoked.**
 
-Reply
+Response
 -----
 
 The response to a request to the platform is managed through the
@@ -159,6 +134,12 @@ response **HTTP status codes.**
 | 403                   | Forbidden             | Not authorized for    |
 |                       |                       | the requested action  |
 +-----------------------+-----------------------+-----------------------+
+| 404                   | Not Found             | The requested entity  |
+|                       |                       | does not exist        |
++-----------------------+-----------------------+-----------------------+
+| 429                   | Too Many Requests     | Global quota or       |
+|                       |                       | entity quota exceed   |
++-----------------------+-----------------------+-----------------------+
 | 5xx                   | Server Error          | Error processing the  |
 |                       |                       | request               |
 +-----------------------+-----------------------+-----------------------+

docs/api_docs/monitor_api.rst

@@ -0,0 +1,41 @@
+Monitor API (Beta)
+==================
+
+Since v1.9, Sentilo offers an internal Monitor API that enables direct access to
+select real-time information of the API server and other components.
+With this internal API you can also remotely restart API server, for example in case the main API is too busy.
+
+The Monitor API runs on a different port then public API. The default port number is :literal:`7081` and is configured in
+:literal:`/sentilo/sentilo-platform/sentilo-platform-server/src/main/resources/properties/config.properties`
+by property :literal:`monitor.port`:
+
+..
+
+    monitor.port=7081
+
+This API currently works with the Catalog Application token. You can find the catalog token in the Application
+section of the Catalog.
+
+.. note::
+
+   The catalog application token is very sensitive information, since it is used for admin operations. Make sure
+   you do store it in a private repository. Also make sure calls to the monitoring API are encrypted. See more in the
+   `Security section <./services/security.html>`__
+
+
+List of API resources in the monitoring API:
+
++--------------------------+--------+-----------------------------------------------------------------------------------+
+|         Resource         | Method |                                    Description                                    |
++==========================+========+===================================================================================+
+| /monitor/ping            | GET    | Simple service that responds with http code 200  if the server is up and running. |
++--------------------------+--------+-----------------------------------------------------------------------------------+
+| /monitor/rl_input_status | GET    | Returns values of global rate limiting (incoming requests), for the last hour.    |
++--------------------------+--------+-----------------------------------------------------------------------------------+
+| /monitor/metrics         | GET    | Returns a list of components with their metrics                                   |
++--------------------------+--------+-----------------------------------------------------------------------------------+
+| /monitor/restart         | POST   | Gracefully restarts the API server. No body needed.                               |
++--------------------------+--------+-----------------------------------------------------------------------------------+
+| /monitor/force-restart   | POST   | Forces a restart of the API server. No body needed.                               |
++--------------------------+--------+-----------------------------------------------------------------------------------+
+

docs/api_docs/rate_limiting.rst

@@ -0,0 +1,102 @@
+Rate Limiting
+=============
+
+Since v1.9, Sentilo offers rate limiting features for the REST API.
+
+Rate limiting prevents the Sentilo instance from request flooding. Also, protects
+ external systems that are subscribed to Sentilo by limiting outcoming requests.
+
+Setting global limits
+---------------------
+
+You can set global incoming quota :literal:`api.global_rate_limit.quota` in file
+:literal:`sentilo-platform/sentilo-platform-server/src/main/resources/properties/config.properties`.
+If left unset or set to 0, Sentilo won't apply any global limit.
+
+The maximum body length can also be set globally with :literal:`api.body.max_length` property.
+
+::
+
+   # number of requests per hour
+   api.global_rate_limit.quota=100
+
+   # Maximum body length in bytes
+   api.body.max_length=4096
+
+
+If global rate limiting is set, every response of the API contains following headers:
+
++--------------------------------------+-----------------------------------------------------+
+|                Header                |                     Description                     |
++======================================+=====================================================+
+| X-RateLimit-Global-Inbound-Limit     | Value of the api.global_rate_limit.quota property   |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Global-Inbound-Remaining | Number of remaining requests until the current      |
+|                                      | limit per hour is exceed                            |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Global-Inbound-Reset     | If the API limit is exceeded, this value indicates  |
+|                                      | the number of minutes until the server will accept  |
+|                                      | requests again. Otherwise is 0.                     |
++--------------------------------------+-----------------------------------------------------+
+
+In case the global rate limit is exceed, the server will reject the request and return a HTTP error code 429.
+
+In case the maximum payload size is exceed, the server responses with HTTP error code 400.
+
+
+
+
+Limit requests by entity
+------------------------
+
+A provider or an application can have particular limits, which can be configured in catalogue:
+
+|rl_entity_190.png|
+
+If input quota for a entity is different from 0, every response of the API contains following headers:
+
++--------------------------------------+-----------------------------------------------------+
+|                Header                |                     Description                     |
++======================================+=====================================================+
+| X-RateLimit-Inbound-Limit            | Value of the input quota                            |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Inbound-Remaining        | Number of remaining requests until the current      |
+|                                      | limit per hour is exceed                            |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Inbound-Reset            | If the entity quota is exceeded, this value         |
+|                                      | indicates the number of minutes until entity will   |
+|                                      | be permitted again. Otherwise is 0.                 |
++--------------------------------------+-----------------------------------------------------+
+
+If output quota for a entity is different from 0, every subscription request that Sentilo sends to the
+subscription endpoint will contain following headers:
+
++--------------------------------------+-----------------------------------------------------+
+|                Header                |                     Description                     |
++======================================+=====================================================+
+| X-RateLimit-Outbound-Limit           | Value of the output quota                           |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Outbound-Remaining       | Number of remaining requests until the current      |
+|                                      | limit per hour is exceed                            |
++--------------------------------------+-----------------------------------------------------+
+| X-RateLimit-Outbound-Reset           | This value indicates the number of minutes until    |
+|                                      | the counter of remaining outgoing requests of the   |
+|                                      | entity will be reset. Otherwise is 0.               |
++--------------------------------------+-----------------------------------------------------+
+
+For example, consider that the subscribed system receives a message, together with these headers:
+
+-  X-RateLimit-Outbound-Limit: 5
+-  X-RateLimit-Outbound-Remaining: 1
+-  X-RateLimit-Outbound-Reset: 58
+
+We can see that this entity that is subscribed to Sentilo data has a output quota set to  5 notifications/hour,
+that it can still receive 1 more notification and that the current window is still valid during next 58 minutes.
+
+As with the global limit,if the entity's input quota is exceed, the server will reject the request and return
+a HTTP error code 429. Global rate limit is prevalent to entity's quota, so the request is rejected when the lowest of
+the two is met.
+
+
+
+.. |rl_entity_190.png| image:: /_static/images/api_docs/rl_entity_190.png