Merge 1f65c60 into 1b525f1

.travis.yml

@@ -11,6 +11,7 @@ install:
 
 before_script:
   - pipenv install
+  - sudo service postgresql stop
 
 script:
   - source setup_dev_env.sh

Pipfile

@@ -24,6 +24,7 @@ pyyaml = ">=4.2"
 redis = "~=2.10"
 requests = ">=2.20"
 rethinkdb = "==2.3"
+pickle-mixin = "==1.0.2"
 
 [dev-packages]
 

README.md

@@ -13,8 +13,12 @@
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4394/badge)](https://bestpractices.coreinfrastructure.org/projects/4394)
 
 QuantumLeap is the first implementation of [an API](https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb)
-that supports the storage of [FIWARE NGSIv2](http://docs.orioncontextbroker.apiary.io/#)
+that supports the storage of [FIWARE NGSIv2](https://fiware.github.io/specifications/ngsiv2/stable/)
 data into a [time series database](https://en.wikipedia.org/wiki/Time_series_database).
+It currently also experimentally supports the injection of
+[NGSI-LD](https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.01.01_60/gs_CIM009v010101p.pdf) in 
+a backward compatible way with NGSI-v2 API. I.e. you can retrieve NGSI-LD stored data via NGSI v2
+API and retrieve data will be describe following NGSI v2 format.
 
 Want to know more? Refer to the [docs](https://quantumleap.readthedocs.io/en/latest/)
 or checkout the Extra Resources below.
@@ -42,9 +46,12 @@ QuantumLeap supports both Crate DB and Timescale as time-series DB
 backends but please bear in mind that at the moment we only support
 the following versions:
 
-* Crate backend: Crate DB version `3.3.*` (will be deprecated from QL `0.8` version) and `4.*`
+* Crate backend: Crate DB version `3.3.*` (will be deprecated from QL `0.9` version) and `4.*`
 * Timescale backend: Postgres version `10.*` or `11.*` +
   Timescale extension `1.3.*` + Postgis extension `2.5.*`.
+
+PR #373 introduced basic support for NGSI-LD. In short this means that using
+the current endpoint you are able to store NGSI-LD payloads with few caveats (see #398)
 
 ## Usage
 
@@ -78,4 +85,4 @@ consistency.
 
 QuantumLeap is licensed under the [MIT](LICENSE) License
 
-© 2017-2019 Martel Innovate
+© 2017-2020 Martel Innovate

docker/docker-compose-dev.yml

@@ -48,6 +48,14 @@ services:
     volumes:
       - redisdata:/data
 
+  redis-commander:
+    image: rediscommander/redis-commander:latest
+    restart: always
+    environment:
+    - REDIS_HOSTS=local:redis:6379:1
+    ports:
+    - "8081:8081"
+
   timescale:
     image: timescale/timescaledb-postgis:${TIMESCALE_VERSION}
     ports:
@@ -58,6 +66,18 @@ services:
     environment:
       - POSTGRES_PASSWORD=*
 
+  pgadmin:
+    image: dpage/pgadmin4:4.26
+    restart: always
+    environment:
+      - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
+      - PGADMIN_CONFIG_SERVER_MODE=False
+      - PGADMIN_DEFAULT_EMAIL=admin@admin.org
+      - PGADMIN_DEFAULT_PASSWORD=admin
+      - PGADMIN_LISTEN_PORT=80
+    ports:
+      - "8080:80"
+
   quantumleap-db-setup:
     build: ../timescale-container/
     image: quantumleap-db-setup

docs/manuals/admin/benchmarks.md

@@ -0,0 +1,53 @@
+# Basic benchmarks
+
+QuantumLeap heavily relies on crate (or timescale) to query and store data model configuration,
+this implies during insert an high number of queries on the backend, despite the change
+of models between an insert and another of the same entity type should not be that frequent.
+
+As tested, caching reduce such number of queries, and thus increase throughput.
+Previous test used in-memory cache, but that's not ideal in a concurrent environment,
+thus we developed experimental support for REDIS based caching.
+
+To measure basic insertion performances we developed a simple load script based on [k6](https://k6.io/)
+that you can find [here](https://github.com/smartsdk/ngsi-timeseries-api/blob/master/src/tests/run_load_tests.sh)
+
+Example:
+```
+docker run -i --rm loadimpact/k6 run --vus 30 --duration 60s - < notify-load-test.js
+
+    checks.....................: 100.00% ✓ 6000 ✗ 0   
+    data_received..............: 1.6 MB  27 kB/s
+    data_sent..................: 1.6 MB  27 kB/s
+    http_req_blocked...........: avg=302.26µs min=1.62µs      med=6.21µs   max=1.05s   p(90)=10.21µs  p(95)=18µs    
+    http_req_connecting........: avg=262.78µs min=0s          med=0s       max=1.05s   p(90)=0s       p(95)=0s      
+    http_req_duration..........: avg=208.12ms min=16.92ms     med=155.56ms max=1.06s   p(90)=409.38ms p(95)=514.53ms
+    http_req_receiving.........: avg=1.2ms    min=-8.817564ms med=158.96µs max=133.5ms p(90)=2.69ms   p(95)=5.99ms  
+    http_req_sending...........: avg=89.36µs  min=12.59µs     med=38.08µs  max=20.85ms p(90)=111.97µs p(95)=218.72µs
+    http_req_tls_handshaking...: avg=0s       min=0s          med=0s       max=0s      p(90)=0s       p(95)=0s      
+    http_req_waiting...........: avg=206.82ms min=16.78ms     med=153.98ms max=1.06s   p(90)=406.54ms p(95)=513.29ms
+    http_reqs..................: 6000    99.814509/s
+    iteration_duration.........: avg=30.02s   min=30.02s      med=30.02s   max=30.03s  p(90)=30.02s   p(95)=30.02s  
+    iterations.................: 60      0.998145/s
+    vus........................: 30      min=30 max=30
+    vus_max....................: 30      min=30 max=30
+```
+
+Tests shows that:
+* caching metadata improves throughput of inserts , but usage of redis, has not the same impact as in memory
+* the number of queries linked to metadata is not the only element affecting insert performance,
+  due to the nature of http request, up to QL v0.7.6, we created a db connection for each insert,
+  opening the connection takes time. From v0.8 we reuse existing connections. This proved to increase throughput of 100%
+  compared just to just redis caching.
+
+System used for testing:
+Mac2016 with 3.1 GhZ i7 (i.e. 4 cores) and 16GB Ram
+
+* Baseline (v0.7.6): 44 req/s - 600 ms avg response time -
+  (crate queries peak: select 111 q/sec, insert 54 q/sec)
+* Redis caching: 60 req/s -  (no data collected on response time)
+* In memory caching:  95 req/s - (no data collected on response time)
+* Redis caching + connection re-usage: 100 req/s - 200 ms avg response time
+  (crate queries peak: select 7 q/sec, insert 142 q/sec)
+* Connection re-usage (without any cache): 55 req/s  - 400 ms avg response time
+  (crate queries peak: select 177 q/sec, insert 86 q/sec)
+

docs/manuals/admin/configuration.md

@@ -19,8 +19,11 @@ To configure QuantumLeap you can use the following environment variables:
 | `REDIS_HOST`       | Redis Host              |
 | `REDIS_PORT`       | Redis Port              |
 | `USE_GEOCODING`    | `True` or `False` enable or disable geocoding |
+| `DEFAULT_CACHE_TTL`| Time to live of metadata cache, default: 60 (seconds) |                              |
 | `QL_CONFIG`        | Pathname for tenant  configuration  |
-| `LOGLEVEL`         | define the log level for all services (`DEBUG`, `INFO`, `WARNING` , `ERROR`)      |
+| `QL_DEFAULT_DB`    | Default backend: `timescale` or `crate`  |
+| `USE_FLASK`        | `True` or `False` to use flask server (only for Dev) or gunicorn. Default to `False`  |
+| `LOGLEVEL`         | Define the log level for all services (`DEBUG`, `INFO`, `WARNING` , `ERROR`)      |
 
 **NOTE**
 * `DEFAULT_LIMIT`. This variable specifies the upper limit L of rows a query

docs/manuals/admin/index.md

@@ -105,7 +105,7 @@ you can leverage the Helm Charts in [this repository](https://smartsdk-recipes.r
 
 In particular you will need to deploy:
 * [CrateDB](https://github.com/orchestracities/charts/tree/master/charts/crate)
-* [Optional] Timescale - for which you can refer to [Patroni Helm Chart](https://github.com/helm/charts/tree/master/incubator/patroni).
+* [Optional/Alternative] Timescale - for which you can refer to [Patroni Helm Chart](https://github.com/helm/charts/tree/master/incubator/patroni).
 * [QuantumLeap](https://github.com/orchestracities/charts/tree/master/charts/quantumleap)
 
 ## FIWARE Releases Compatibility