Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Benchmark of Python WSGI Servers
Python
branch: master

README.md

Benchmark of Python WSGI Servers

We were interested in comparing Gevent with Tornado to see which would better suit our needs. The starting point was a benchmark done by Nicholas Piël in 2010 showing that Tornado had excellent performance followed closely by Gevent.

We did a very quick and non-rigorous benchmark in 2013-01-05. Nevertheless we are making the code and results available in case this can be useful elsewhere.

Scenarios

First we tested a linux client generating HTTP traffic towards a MacOSX running the server backend. This test was run over the corporate wi-fi network. Then, we recreated the same test scenario in two different sets of Amazon instances to cross-validate the performance measured.

In the client side, we used wrk, weighttp and locust to generate traffic.

The test cases where basically:

  • the HTTP server just responds a "hello world" to a GET request.
  • the HTTP server queries a backend Virtuoso triple-store database
  • the HTTP server queries a backend REDIS database

Infra-structure details

In the tests we used the following hardware configuration:

  • (corporate) Linux running Fedora 16, Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz 8Gb RAM, cache size 4096 KB
  • (corporate) MacOSX lion 10.7.5 - 2Ghz Intel Vore i5 8Gb RAM, L2 Cache (per Core) 256 KB
  • (Amazon) Linux Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-virtual x86_64), Intel(R) Xeon(R) CPU E5645 @ 2.40GHz, cache size 12288 KB

Results

The test cases were:

  • tornado_lan_flask_hw - Tornado returning a simple "Hello World" for GET / with Flask in corporate network
  • tornado_lan_sock_hw - Tornado returning a simple "Hello World" for GET / with pure sockets in corporate network
  • tornado_virt_lan - Tornado doing a query in Virtuoso (all servers in the same machine) in corporate network
  • tornado_virt_s3 - Tornado doing a query in Virtuoso (all servers in the same machine) in Amazon Srv3
  • tornado_virt_s1s2_pycurl - Tornado doing a query in Virtuoso (servers in different machines) in Amazon Srv1-Srv2
  • tornado_virt_s1s2_native - Tornado doing a query in Virtuoso (servers in different machines) in Amazon Srv1-Srv2 (without PyCurl)
  • tornado_virt_pypy - Tornado doing a query in Virtuoso (servers in different machines) in Amazon Srv1-Srv2 (over Pypy)
  • tornado_inc_load - Tornado incremental load

  • gevent_lan_flask_hw - Gevent returning a simple "Hello World" for GET / with Flask in corporate network

  • gevent_lan_sock_hw - Gevent returning a simple "Hello World" for GET / with pure sockets in corporate network
  • gevent_pure_wsgi - Gevent returning a simple "Hello World" for GET / with pure WSGI in corporate network
  • gevent_virt_lan - Gevent doing a query in Virtuoso (all servers in the same machine) in corporate network
  • gevent_virt_s3 - Gevent doing a query in Virtuoso (all servers in the same machine) in Amazon Srv3
  • gevent_virt_s3_alt - Gevent (alternate implementaion) doing a query in Virtuoso (all servers in the same machine) in Amazon Srv3
  • gevent_virt_s1s2 - Gevent doing a query in Virtuoso (servers in different machines) in Amazon Srv1-Srv2
  • gevent_virt_s1s2_alt - Gevent doing a query in Virtuoso (servers in different machines) in Amazon Srv1-Srv2
  • gevent_inc_load - Gevent incremental load
  • gevent_inc_load_alt - Gevent incremental load (alternate implementaion)

The table below depicts the average results for comparison:

Test Case Runs Avg. Requests/s Max Requests/s Min Requests/s
tornado_lan_flask_hw 10 1929.68 2186.74 1395.3
gevent_lan_flask_hw 5 883.21 941.02 791.95
tornado_lan_sock_hw 2 5723.74 5731.39 5716.09
gevent_lan_sock_hw 2 6597.43 6617.0 6577.87
gevent_pure_wsgi 3 3652.08 3906.9 3404.44
tornado_virt_lan 4 643.21 740.14 377.55
gevent_virt_lan 6 140.12 162.46 130.91
tornado_virt_s3 5 900.48 920.0 881.46
gevent_virt_s3 4 436.85 449 428.79
gevent_virt_s3_alt 4 522.32 525 516
tornado_virt_s1s2_pycurl 2 986 991 981
tornado_virt_s1s2_native 2 624 629 620
gevent_virt_s1s2 2 414 417 411
gevent_virt_s1s2_alt 2 167 171 164
tornado_virt_pypy 2 822 855 789

The following table presents how "request/s" evolves as the client load increases. The columns represent the number of parallel connections in the client that generates the load.

Test Case 10 20 30 40 50
tornado_inc_load 896 899 879 862 852
gevent_inc_load 462 446 437 308 256
gevent_inc_load_alt 518 508 413 310 311

The attached Benchmarks.md file contains dumps of the raw tests executed.

Conclusions

Tornado shows consistently a better performance than Gevent considering just "requests/s", and responds better when we increase the number of requests, thus, it is a good hint that will scale better. However, the code produced with Gevent was cleaner and more readable than the code produced with Tornado.

When using Tornado, the adoption of the pycurl client (instead of the native client) provides an enhancement in the performance.

The use of Flask significantly reduces the performance of both Tornado or Gevent.

Comparing tornado_virt_s3 with tornado_virt_pypy showed that PyPy adoption did not contributed to any performance enhancement. We were unable to run PyPy with Gevent.

We did not have consistent results of which is the best http client for Gevent: either the patched "requests" lib or "geventhttpclient". When the servers are separate (more realistic use case), patched "requests" performed better than "geventhttpclient".

We did some other tests accessing a backend Redis server, instead of Virtuoso. The source code is also published in this project, but we did not reported here the detailed results of those tests. When querying the Redis server using a synchronous driver the results of Tornado and Gevent had equivalent performances, however neither was stable.

DISCLAIMER: This is a quick-and-dirty benchmark for us to have a glimpse of how to code in Tornado in comparison with how to code in Gevent, and to have a coarse grain idea of performance. A serious benchmark would need much more repetitions under more controlled conditions.

Team members

The team members that took part in the testing efforts were:

References

Something went wrong with that request. Please try again.