Daphne performance stuck to 10rq/s

[senechalloic]

Setup

I have configured Django Channels inside a docker container on server Ubuntu 20.04 OVH VPS (8vCore, 16Go RAM). To run django-channels I used daphne. Versions:

Django==3.1.4
channels==3.0.2
daphne==3.0.1

I used Docker-compose with an image for Daphne-Apache, an image for Mysql, an image for Redis.

Issue

I used Locust to check the load performance of the application. When I have more than 10rq/s, the server HTTP response time becomes huge (> 10 sec) and the application becomes unavailable.

Where is the bottleneck? What could improve the performance?

Extra Info

• Note 1: I have reduced the SQL Querries to very low amount, about 5 per page.
• Note 2: On the VPS info page, CPU, Mem, bandwidth is not fully used at all.
• Note 3: I serve static files with Apache Alias, but when the application is overloaded, even static files takes time to load.
• Note 4: The error I get when there is too much load ConnectionError(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
• Note 5: I have followed all guidelines/configuration proposed in the read-the-doc.

I've searched and found out using multiples process with supervisord or Kubernetes could help. But I want to be sure everything is normal before evolving to that.

I can share project files docker-compose.yml, routing.py, settings.py but it doesn't seems helpful to me for now.

[carltongibson]

Hi @senechalloic wondering if this is django/channels#1587. See also django/channels#1582.

I'd be inclined to serve Django views using a WSGI server, using Daphne/ASGI just for the Channels specific stuff for now.

[senechalloic]

Hi @carltongibson

Thanks a lot for your response that helps me a lot. This looks really a lot like other issues you mentioned, I will follow these issues.
Thanks for the tip, I will try to split ASGI and WSGI, even if it's not the most convenient.

I have no idea if we should close this issue or not, but I now have a workaround for my issue.

[carltongibson]

@senechalloic -- I'll leave it open for now. It's on my list to address for the new year, but Django 3.2a1 out the way first. 😄

[senechalloic]

New tests

Hello, I have tested:

• Serve Django views using a WSGI server, using Daphne/ASGI just for the Channels.
• Reverting back to Channels 2.X / Daphne 2.X (keeping the latest Django 3.X)

I have had the exact same issue, which is :

• If I perform more than 10 req/s the server goes down.

So I'm not sure the problem is coming from daphne in my case.

Next steps

I will try:

• Check profile of execution
• Look into asynchronous views if it can help
• Try to setup multiple instance of the server with Kubernetes

[senechalloic]

Final solution

For anyone having the same issue. I found out using Uvicorn:

uvicorn myApp.asgi:application --workers=8 --port 8443 --ssl-keyfile=key.pem --ssl-certfile=crt.pem

Instead of daphne:

daphne -e ssl:8443:privateKey=key.pem:certKey=crt.pem -b 0.0.0.0 myApp.asgi:application

This helped me to have much better performance. About five time better.

[Archmonger]

@senechalloic That's not really a solution but a workaround. Each individual worker will still be blocking while serving each client request, but you now just have 8 workers that will be rotated through.

[senechalloic]

Yes it's a workaround. And this didn't really help, as you mentionned.
Requests were treated sequentially, not concuratly, like mentioned in issue #1587

What saved my live, in this project, is two things:
1. Run ASGI(daphne) and WSGI(gunicorn) separatly like mentioned by cartlongibson
2. Using NGINX instead of Apache

As a result the performances were much much much better allowing more than 100rq/s

[carltongibson]

This was resolved by the per-request async context added in Django 4.0 — It's not really a Daphne issue.

[epsawan]

I am still getting the same issue (using 4.1.13 django version)
When I hit 2 API first API which normally take 20 seconds 2nd API which take normally 3 sec.
But when I hit first API than 2nd API, main thread blocked by the first API and I got response of second API only after getting response from first API