Benchmark TANNER

[glaslos]

How many requests can we serve per second? Which APIs are slow? Which handlers are slow? Does https://github.com/MagicStack/uvloop speed up things?

[afeena]

I made some checks and improvements, the results in the table below. Tested with wrk. It works through SNARE, so it have SNARE latency in the measurements.
Testing string: `wrk -t12 -c400 -d30s ``

request	asyncio	uvloop&improvements
index	Requests/sec: 17.14	Requests/sec: 49.26
rfi	Requests/sec: 32.20	Requests/sec: 60.65
lfi	Requests/sec: 46.18	Requests/sec: 51.34
xss	Requests/sec: 24.59	Requests/sec: 50.54
sqli	Requests/sec: 49.13	Requests/sec: 45.27
api	Requests/sec: 126.38	Requests/sec: 396.19

In this case, api testing is the most meaningful: we send requests directly on TANNER. So, first measurement without any changes:

root@afeena-ubuntu16:~/wrk# wrk -t12 -c400 -d30s http://127.0.0.1:8080/api/stats
Running 30s test @ http://127.0.0.1:8080/api/stats
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.24s   477.97ms   2.00s    54.30%
    Req/Sec    13.99     13.01   140.00     88.41%
  3802 requests in 30.08s, 623.91KB read
  Socket errors: connect 0, read 3614, write 0, timeout 1463
  Non-2xx or 3xx responses: 3802
Requests/sec:    126.38
Transfer/sec:     20.74KB

After adding asyncio_redis, Requests/sec became about 290, and after adding uvloop it became faster:

root@afeena-ubuntu16:/home/afeena/tanner# wrk -t12 -c400 -d30s http://127.0.0.1:8090/api/stats
Running 30s test @ http://127.0.0.1:8090/api/stats
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   814.75ms  209.02ms   1.68s    77.34%
    Req/Sec    45.69     44.02   303.00     84.65%
  11925 requests in 30.10s, 2.56MB read
  Socket errors: connect 0, read 11911, write 0, timeout 0
Requests/sec:    396.19
Transfer/sec:     87.05KB

The slowest emulator is sqli emulator, I don't know what we can improve in it right now: sqlite3 not async and we haven't any solution for libinjection.

Also we have a lot of file operation in the lfi emulator, maybe it will be better, if we move the virtual file system to redis?

[glaslos]

Good work. Use runsnakerun for some analysis: http://www.vrplumber.com/programming/runsnakerun/

[afeena]

Benchmark with latest changes:

Without uvloop:

root@afeena-ubuntu16:/home/afeena/tanner# wrk -t12 -c400 -d30s http://127.0.0.1:8090/
Running 30s test @ http://127.0.0.1:8090/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    72.24ms   55.54ms   1.80s    99.51%
    Req/Sec   261.80    187.36   666.00     49.02%
  45449 requests in 30.10s, 8.11MB read
  Socket errors: connect 0, read 45543, write 0, timeout 9
Requests/sec:   1509.90
Transfer/sec:    275.73KB

With uvloop

root@afeena-ubuntu16:/home/afeena/tanner# wrk -t12 -c400 -d30s http://127.0.0.1:8090/
Running 30s test @ http://127.0.0.1:8090/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    99.31ms   14.37ms 285.36ms   86.92%
    Req/Sec   292.76     79.62     1.00k    82.12%
  103899 requests in 30.10s, 18.53MB read
  Socket errors: connect 0, read 103843, write 0, timeout 0
Requests/sec:   3452.12
Transfer/sec:    630.42KB

[glaslos]

The 9 timeouts without uvloop is pretty scary...
Did you manage to make a profile with runsnakerun to see where we spend most of the time?

[afeena]

Yes, I used snakeviz instead of runsnakerun (because of I haven't gui on linux):

Without uvloop:

With uvloop:

[glaslos]

Good work. I think we can close this issue for now. We might want to keep an eye on the average response time when we do some monitoring.