Example with aiohttp

[nhumrich]

In your uvloop docs you talk about running aiohttp with this parser instead of the built in one. Do you have any documentation on how that would work?

[1st1]

In your uvloop docs you talk about running aiohttp with this parser instead of the built in one. Do you have any documentation on how that would work?

What docs are you referring to?

Aiohttp guys tried to port it to use httptools, but IIRC that work wasn't merged.

[nhumrich]

Sorry, not exactly docs, the blog post for uvloop http://magic.io/blog/uvloop-blazing-fast-python-networking/
The benchmarks show using aiohttp with this parser

[1st1]

Sorry, not exactly docs, the blog post for uvloop http://magic.io/blog/uvloop-blazing-fast-python-networking/
The benchmarks show using aiohttp with this parser

Got it. Sorry for the confusion -- in that blog post we benchmarked asyncio and uvloop on vanilla aiohttp and simple httptools protocol. There is no way to run aiohttp with httptools for now ;(

[asvetlov]

I believe the issue should be closed

[kespindler]

@nhumrich Not sure about aiohttp, but I just incorporated httptools into the web framework I'm writing, https://github.com/kespindler/albatross . Feel free to poke around my implementation if you want an example of using httptools.

[1st1]

Since there are no plans about incorporating httptools into aiohttp, I'll just close this issue.

[asvetlov]

Honestly I have such plan but aiohttp need some internal transformations on this way.

[alexjc]

After benchmarking, I found that aiohttp is generally slow everywhere else: 5x-7x slower than bottle for example. It matches with these benchmark results:
https://github.com/klen/py-frameworks-bench

This seems to be caused by the choice of abstraction levels in aiohttp that simply adds up fast. httptools integration does not help improve performance much because of this—as reported here too: aio-libs/aiohttp#858

[asvetlov]

Would you publish your benchmarking code, please?
aiohttp 1.1 is about 4x slower than bottle by my rough measures based on https://github.com/KeepSafe/aiohttp/blob/master/benchmark/prof.py run.
Still not perfect but I'm working on it.

You are right: current abstraction layers are tricky and slow, there are many spikes.
I have strong intention to get rid of them but it's not easy task.

P.S.
Access log disabling makes a little boost.

[alexjc]

@asvetlov Thanks for the reply. I'm using pretty much the same code as here:
https://github.com/klen/py-frameworks-bench

I found the logging bottleneck very quickly using PyPI profiling module, also removed it. I need to try the latest code! Glad to hear you're on this, I think the choice of abstraction and how to implement them could be fixed incrementally. aiohttp is by far the most polished asyncio framework, it'd be an easy choice if it was performance comparable!

[alexjc]

@asvetlov Also, I found this project which seems to be at 10x aiohttp according to the author's benchmarks (subjective): https://github.com/kimjoseph95/sanic-python-web-server

Could be interesting to compare execution of a single request in-depth and see what can be ported back.

[vazir]

difference is there... In my tests aiohttp gives ~7300 rps with uvloop and 6500 rps with regular ioloop, and sanic gives ~30000 rps.
But for those who needs performance, in my test using "httptools" - there is 70000 RPS - so ~ 10x difference to it's all...

Clearly aiohttp is drastically inefficient.

Below are "Hello World" tests on Dell XPS 15 (core i7, 4x cores)

Python uvloop+httptools (1x core) = 70653 RPS
Python uvloop+aiohttp (1x core) = 7315 RPS
Python Sanic (1 core) = 30716 RPS
Java Vert.x, utilizing 2x cores rough = 114000 RPS
Go fasthttp (4x cores) = 320000 RPS !!!

---- uvloop+httptools = 70653 RPS

$ ./aiot.py --type=uvloop+httptools
using uvloop loop: <uvloop.Loop running=False closed=False debug=False>
using httptools HTTP server
serving on: ('127.0.0.1', 8080)

$ wrk -t2 -c50 -d10s http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
2 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 704.06us 219.55us 6.50ms 76.07%
Req/Sec 35.51k 1.43k 37.40k 86.00%
706625 requests in 10.00s, 125.34MB read
Requests/sec: 70653.09
Transfer/sec: 12.53MB

--- uvloop+aiohttp = 7315 RPS
$ ./aiot.py --type=uvloop+aiohttp
using uvloop loop: <uvloop.Loop running=False closed=False debug=False>
using aiohttp HTTP server
serving on: ('127.0.0.1', 8080)

$ wrk -t2 -c50 -d10s http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
2 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.82ms 1.45ms 16.20ms 71.98%
Req/Sec 3.67k 105.51 3.90k 72.50%
73228 requests in 10.01s, 11.38MB read
Requests/sec: 7315.24
Transfer/sec: 1.14MB

--- Sanic = 30716 RPS
017-05-09 15:19:26 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8080
2017-05-09 15:19:26 - (sanic)[INFO]: Starting worker [9274]

$ wrk -t2 -c50 -d10s http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
2 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.62ms 281.53us 4.86ms 85.70%
Req/Sec 15.43k 622.32 16.12k 82.50%
307180 requests in 10.00s, 37.50MB read
Requests/sec: 30716.65
Transfer/sec: 3.75MB

--- Java Vert.x = 114713 RPS
$ wrk -t2 -c50 -d10s http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
2 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 792.79us 3.99ms 87.29ms 98.76%
Req/Sec 57.66k 14.35k 68.40k 90.00%
1147214 requests in 10.00s, 67.83MB read
Requests/sec: 114713.89
Transfer/sec: 6.78MB

--- GoLang fasthttp = 321061 RPS
$ wrk -t2 -c50 -d10s http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
2 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 231.32us 1.09ms 28.14ms 97.80%
Req/Sec 161.45k 17.29k 200.42k 76.00%
3215733 requests in 10.02s, 456.95MB read
Requests/sec: 321061.94
Transfer/sec: 45.62MB

[RobertoPrevato]

I am preparing Docker images and testing different setups in a more realistic scenario: not on my local network, but on instances of Standard S1 machines in Azure West Europe data center (deploying services using ARM templates), running benchmark from Warsaw, Poland, using Apache Benchmark tool. I am not ready to share my code, yet; however I saw the following:

• Sanic with uvloop requires CPython 3.6.2 (it cannot be used with pypy3, yet) and it offers pretty much the same performance of pypy3 + Gunicorn + Gevent + Flask setup - results are surprisingly similar. However, I expect real applications to perform faster when benefiting of pypy3
• uvloop with httptools is much faster (using CPython 3.6.2), but of course httptools is really barebone, so it's a bit unfair to compare it with user friendly frameworks like Sanic or Flask

I also got decent performance with pypy3 + Gunicorn + SyncWorker class + Flask (multiple workers, of course).

I will prepare tests to compare the performance of ASP.NET Core applications in Docker and outside of Docker, using Azure S1 machines, I am really curious how these compare. Also, I will get to know wrk and try to use it instead of Apache Benchmark.