ASGI Python Frameworks load testing

https://github.com/mtag-dev/py-rest-stress-testing

Updated: 2022-12-23

Original benchmark: https://klen.github.io/py-frameworks-bench/

---

We aim to create a tool for technical decision-making.

This is a benchmark for modern ASGI micro-frameworks. Designed to be not one more "Hello world" benchmark, but cover some real-like scenarios instead. Doesn't provide composite scoring, drills down instead.

To be data source agnostic (API, DB, etc.) framework provides thread-safe async dummy pool with configurable size and data retrieving latency. Because of our target is benchmark web-frameworks itself, this feature by default configured with default pool size 10 and latency 1ms (emulates very fast data source, like Redis.)

Through polls, we have found out that AVG mature service contains ~30 endpoints. So during a time, routing performance can drop. We cover this by adding 30 more dynamic endpoints with different methods. These endpoints are just present for routing cycle load.

This benchmark doesn't provide load tests for different ASGI servers. All tests (except AioHTTP) runs on the Uvicorn.

Doesn't contain benchmarks for OAuth 2.0/OIDC/JWT features. This is because we pretty sure that these features should be a part of API-Gateway and not in end service itself.

Further development implies adding scenarios from different domains. For now, it is only issue tracker as example

Table of contents

• The Methodic
• The Results
• JSON response without schema
  • GET: User info
  • POST: Create task
  • PUT: Update task
  • GET: Sprint board
• JSON response using schema
  • GET: User info
  • POST: Create task
  • PUT: Update task
  • GET: Sprint board

The Methodic

The benchmark runs as a Github Action. According to the github documentation the hardware specification for the runs is:

• 2-core vCPU (Intel® Xeon® Platinum 8272CL (Cascade Lake), Intel® Xeon® 8171M 2.1GHz (Skylake))
• 7 GB of RAM memory
• 14 GB of SSD disk space
• OS Ubuntu 20.04

ASGI apps are running from docker using the gunicorn/uvicorn command:

gunicorn -k uvicorn.workers.UvicornWorker -b 0.0.0.0:8080 app:app

Applications' source code can be found here.

Results received with WRK utility using the params:

wrk -d15s -t4 -c32 [URL]

The "Issue tracker" suite has a four kind of tests:

• getting user information
• create task
• update task
• board with short information with 100 issues on it

You can find test data here.

The Results (2022-12-23)

JSON response without schema

User info (GET)

Get user information and return using direct serialisation from Python primitives to JSON.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-raw 0.3.0	3452	17.78	20.31	18.63
blacksheep-raw 1.2.8	3277	18.28	21.80	19.57
muffin-raw 0.87.3	3070	19.11	24.03	20.87
baize-raw 0.18.2	2874	21.06	25.28	22.26
emmett-raw 2.4.12	2790	21.23	26.39	22.98
falcon-raw 3.1.0	2646	22.80	27.78	24.22
starlette-raw 0.21.0	2507	24.34	29.87	25.49
sanic-raw 22.9.1	2184	28.23	33.62	29.33
aiohttp-raw 3.8.3	1626	39.41	41.93	39.47
fastapi-raw 0.88.0	1444	43.17	50.23	44.27
quart-raw 0.18.3	1265	48.17	55.04	50.49

Sprint issues (GET)

Get sprint tickets overview and return using direct serialisation from Python primitives to JSON.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-raw 0.3.0	2015	29.81	34.82	31.76
muffin-raw 0.87.3	1613	37.35	44.84	39.61
emmett-raw 2.4.12	1408	41.61	51.62	45.38
blacksheep-raw 1.2.8	1329	44.49	55.41	48.11
baize-raw 0.18.2	1213	50.95	58.30	52.66
starlette-raw 0.21.0	1208	49.17	61.78	53.01
falcon-raw 3.1.0	1144	52.44	63.58	55.79
sanic-raw 22.9.1	1122	52.34	64.31	57.11
aiohttp-raw 3.8.3	974	65.66	68.36	65.56
quart-raw 0.18.3	836	74.46	83.64	76.36
fastapi-raw 0.88.0	163	380.03	419.94	388.69

Create task (POST)

Create task object using default payload and return created object using direct serialisation from Python primitives to JSON.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
muffin-raw 0.87.3	2619	23.15	28.44	24.38
falcon-raw 3.1.0	2420	25.18	30.72	26.40
squall-raw 0.3.0	2383	25.45	30.80	26.87
blacksheep-raw 1.2.8	2360	25.75	31.33	27.10
starlette-raw 0.21.0	2266	27.73	32.36	28.17
baize-raw 0.18.2	2127	29.95	31.26	30.05
emmett-raw 2.4.12	1985	30.44	36.84	32.48
sanic-raw 22.9.1	1952	31.61	37.43	32.77
quart-raw 0.18.3	1218	51.15	56.30	52.43
aiohttp-raw 3.8.3	1170	54.28	56.19	54.57
fastapi-raw 0.88.0	1125	56.18	64.09	56.81

Update task (PUT)

Update task object using default payload.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-raw 0.3.0	2932	20.52	24.58	21.82
falcon-raw 3.1.0	2882	20.87	24.86	22.16
blacksheep-raw 1.2.8	2723	21.64	27.25	23.48
muffin-raw 0.87.3	2717	21.93	27.17	23.51
baize-raw 0.18.2	2455	25.78	27.16	26.03
emmett-raw 2.4.12	2312	25.86	31.99	27.81
starlette-raw 0.21.0	2139	28.98	34.71	29.87
sanic-raw 22.9.1	1926	31.64	38.10	33.17
quart-raw 0.18.3	1338	46.29	53.68	47.74
aiohttp-raw 3.8.3	1276	50.06	52.35	50.12
fastapi-raw 0.88.0	1240	49.52	59.69	51.55

JSON response using schema

User info (GET)

Get user information and return using dataclasses, no extra validation.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-dataclasses 0.3.0	2847	20.85	26.05	22.49
blacksheep-dataclasses 1.2.8	2167	28.67	33.80	29.46
fastapi-dataclasses 0.88.0	1208	51.41	60.98	52.88

Sprint issues (GET)

Get sprint tickets overview and return using dataclasses, no extra validation.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-dataclasses 0.3.0	992	60.53	74.10	64.40
blacksheep-dataclasses 1.2.8	106	558.72	700.65	593.48
fastapi-dataclasses 0.88.0	73	788.09	917.51	844.82

Create task (POST)

Create task object using default payload and return created object using dataclasses, no extra validation.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-dataclasses 0.3.0	2365	26.29	31.16	27.00
blacksheep-dataclasses 1.2.8	914	70.06	79.26	69.85
fastapi-dataclasses 0.88.0	629	102.62	115.72	101.41

Update task (PUT)

Update task object using default payload, no extra validation.

Sorted by max req/s

Framework	Requests/sec	Latency 50% (ms)	Latency 75% (ms)	Latency Avg (ms)
squall-dataclasses 0.3.0	2766	22.07	26.44	23.09
blacksheep-dataclasses 1.2.8	1934	31.92	37.81	33.04
fastapi-dataclasses 0.88.0	1418	43.69	51.94	45.04

Conclusion

Nothing here, just some measures for you.

License

Licensed under a MIT license (See LICENSE file)