Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Frank Vieira
Frank Vieira Update README.md
Fixed a typo.
Latest commit 0a60e17 Jun 22, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
core Aiohttp is now using uvloop and byte responses. Jun 22, 2018
frameworks
wrk_scripts
.gitignore
Dockerfile Initialized Repo Jun 16, 2018
README.md
config.json Updated config. Jun 22, 2018
requirements.txt
run.py

README.md

Last update: 22/06/2018. TLDR: Aiohttp is now using uvloop, Sanic slighly improved .

Before bashing this please open an issue and help us improve it.

Python Web Benchmarks

A simple project to benchmark Python web frameworks against Vibora in a few use cases (more cases will be added soon).

Running Instructions

  1. Install Docker (sudo apt-get install docker.io)

  2. Clone the project (git clone https://github.com/vibora-io/benchmarks)

  3. Build the image (sudo docker build . -t vibora_benchmarks)

  4. Run the benchmark (sudo docker run vibora_benchmarks)

You can configure the benchmark by editing config.json.

Disclaimer

  • Benchmarks are more of a sport than science, don't make assumptions, do your own benchmarks.

  • There are one zillion more important things to consider when choosing a framework. Be wise.

  • I tried to "optimize" each framework but I'm not an expert in every single one.

  • This benchmark mixes sync/async frameworks so to make up a "fair" fight there is no latency between database requests. Async frameworks under high concurrency and high latency databases would perform much better than sync ones thanks to the architecture.

  • Talking about concurrency: WRK is configured to 100 concurrent users which is pretty low number, again, to make it fair against sync frameworks.

  • Users of WSGI are configured to use Gunicorn + Meinheld which is the fastest option although not the most stable one.

Results (AWS c5.2xlarge - 8 CPU cores - Dedicated Tenancy)


Infamous Hello World

Testing how fast they can answer a static response. Not much useful but it sets the high bar for each framework.

Sanic does not send the "Date" http header which is a violation of the HTTP protocol (actually you could argue that this header is not mandatory but since it's used every where by cache engines (and the RFC says unless you don't have a clock you must send) I wouldn't even lose my time arguing).


Frameworks Requests/Sec Version
Tornado 14,197 5.02
Django 22,823 2.0.6
Flask 37,487 1.0.2
Aiohttp 61,252 3.3.2
Sanic 119,764 0.7.0
Vibora 368,456 0.0.6

Validate JSON

Testing how fast they receive a JSON, parse and validate it. Frameworks that do not have a validation engine (or it's too slow) are using Marshmallow, at least for now, unless a fast and tested library appears on the wild.

Vibora/Aiohttp have "streaming enabled" by default which means the user can receive the response in chunks (or even reject it) before sending the response... So beware that Sanic, for example, has a streaming mode which makes the route slower than normal and could affect performance.


Frameworks Requests/Sec Version
Tornado 12,126 5.02
Flask 18,326 1.0.2
Django 18,317 2.0.6
Aiohttp 36,299 3.3.2
Sanic 69,921 0.7.0
Vibora 130,197 0.0.6

Forms Parsing

Testing how fast they receive (and parse) a POST with a multipart-form inside it containing a few parameters and one single file.

Most frameworks with the exception of Vibora and Aiohttp do not have a streaming multipart parser which means they must load the entire form in-memory (big file uploads are impracticable) before letting the user interact with it (unless of course you read the stream yourself and parse it).


Frameworks Requests/Sec Version
Aiohttp 2,315 3.3.2
Django 7,311 2.0.6
Tornado 10,292 5.02
Flask 11,923 1.0.2
Sanic 47,520 0.7.0
Vibora 76,612 0.0.6

Redis API

Testing how fast they receive a POST with a JSON, validate it and do a simple GET in Redis.


Frameworks Requests/Sec Version
Tornado 12,126 5.02
Django 18,317 2.0.6
Flask 18,326 1.0.2
Aiohttp 28,521 3.3.2
Sanic 70,132 0.7.0
Vibora 131,614 0.0.6
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.