Skip to content

Commit

Permalink
add benchmark info in the docs
Browse files Browse the repository at this point in the history
  • Loading branch information
chicco785 committed Nov 18, 2020
1 parent 69d5db6 commit b3c46d6
Show file tree
Hide file tree
Showing 2 changed files with 54 additions and 0 deletions.
53 changes: 53 additions & 0 deletions docs/manuals/admin/benchmarks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Basic benchmarks

QuantumLeap heavily relies on crate (or timescale) to query and store data model configuration,
this implies during insert an high number of queries on the backend, despite the change
of models between an insert and another of the same entity type should not be that frequent.

As tested, caching reduce such number of queries, and thus increase throughput.
Previous test used in-memory cache, but that's not ideal in a concurrent environment,
thus we developed experimental support for REDIS based caching.

To measure basic insertion performances we developed a simple load script based on [k6](https://k6.io/)
that you can find [here](https://github.com/smartsdk/ngsi-timeseries-api/blob/master/src/tests/run_load_tests.sh)

Example:
```
docker run -i --rm loadimpact/k6 run --vus 30 --duration 60s - < notify-load-test.js
checks.....................: 100.00% ✓ 6000 ✗ 0
data_received..............: 1.6 MB 27 kB/s
data_sent..................: 1.6 MB 27 kB/s
http_req_blocked...........: avg=302.26µs min=1.62µs med=6.21µs max=1.05s p(90)=10.21µs p(95)=18µs
http_req_connecting........: avg=262.78µs min=0s med=0s max=1.05s p(90)=0s p(95)=0s
http_req_duration..........: avg=208.12ms min=16.92ms med=155.56ms max=1.06s p(90)=409.38ms p(95)=514.53ms
http_req_receiving.........: avg=1.2ms min=-8.817564ms med=158.96µs max=133.5ms p(90)=2.69ms p(95)=5.99ms
http_req_sending...........: avg=89.36µs min=12.59µs med=38.08µs max=20.85ms p(90)=111.97µs p(95)=218.72µs
http_req_tls_handshaking...: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...........: avg=206.82ms min=16.78ms med=153.98ms max=1.06s p(90)=406.54ms p(95)=513.29ms
http_reqs..................: 6000 99.814509/s
iteration_duration.........: avg=30.02s min=30.02s med=30.02s max=30.03s p(90)=30.02s p(95)=30.02s
iterations.................: 60 0.998145/s
vus........................: 30 min=30 max=30
vus_max....................: 30 min=30 max=30
```

Tests shows that:
* caching metadata improves throughput of inserts , but usage of redis, has not the same impact as in memory
* the number of queries linked to metadata is not the only element affecting insert performance,
due to the nature of http request, up to QL v0.7.6, we created a db connection for each insert,
opening the connection takes time. From v0.8 we reuse existing connections. This proved to increase throughput of 100%
compared just to just redis caching.

System used for testing:
Mac2016 with 3.1 GhZ i7 (i.e. 4 cores) and 16GB Ram

* Baseline (v0.7.6): 44 req/s - 600 ms avg response time -
(crate queries peak: select 111 q/sec, insert 54 q/sec)
* Redis caching: 60 req/s - (no data collected on response time)
* In memory caching: 95 req/s - (no data collected on response time)
* Redis caching + connection re-usage: 100 req/s - 200 ms avg response time
(crate queries peak: select 7 q/sec, insert 142 q/sec)
* Connection re-usage (without any cache): 55 req/s - 400 ms avg response time
(crate queries peak: select 177 q/sec, insert 86 q/sec)

1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,4 @@ nav:
- 'Timescale': 'admin/timescale.md'
- 'Grafana': 'admin/grafana.md'
- 'Data-Migration': 'admin/dataMigration.md'
- 'Bechmarks': 'admin/benchmarks.md'

0 comments on commit b3c46d6

Please sign in to comment.