WARNING - server - Application instance took too long to shut down and was killed #1119
Comments
|
Here is my log: |
|
This means something in your consumer is not exiting after the disconnect (you don't need to send If you can trace it down to a clear problem in channels with steps to reproduce from a fresh project, please re-open the bug - you can see more about where to ask questions at http://channels.readthedocs.io/en/latest/support.html |
|
In my case, the problem was with check if there are tcp connections in In my case I had around 50+ in such state. Had to update redis. |
|
I'm gonna say this here as it may help others. In my case the problem was just a silly mistake. In the example above the same pattern exists. However I am not sure if that was the case back then in 2018 or this is a new style in Channels' code. |
|
I have a similar situation, but not the same
I am using Redis as backend with the code: Using Daphne for Http and WebSocket My asgi file: Here is the error message: It is not the same case as mentioned above, because:
Can you explain why I have this issue? I thought the reason is django's version incompatibility |
I'm facing the same issue in my project. |
|
In my particular case it was asgiref==3.3.1 the culprit, i did a rollback to 3.2.10 and it works well ! I have daphne 2.5.0/channels 2.4.0/django 3.0.10/djangorestframework 3.11.1 |
I could not solve the problem as I wanted. Instead, I run two containers:
So, I used Nginx as the reverse proxy to navigate all requests with "ws/" path to the second container, and the rest to the first container. Here is my conf.template code: |
This solved the issue for us. The app started to time out requests and emitting those warnings. Pip freezing seems to be a must. |
|
I have the same issue, I tried the solution from @MrVhek and unfortunately it didn't work. Any other hint ? Thanks |
|
@andrewgodwin I can reproduce this by changing the chat example in document: and then
It seems that the server push code (here the chat_message) preventing the call to disconnect(), so the self.disconnected flag have no chance to be set to True. And here is my log output: |
|
@tinylambda thanks for the extra info. Not sure yet if this is a usage issue we should document or something else, but I will reopen to investigate |
@carltongibson Thanks for the reply. In my use case, I want to use django-channels as an access layer, and keep sending new game state to user either by time interval (every 1 second) or states (state changed) from redis server. I think it should be a typical use case when it comes to websockets. |
If the coroutine (here chat_message) handle the message takes too long time to complete (to broadcast messages for example), it will block every message with the same type process until previous coroutine completes (maybe never). As a potential solution, how about dispatching every handler coroutine (chat_message) in tasks (loop.create_task), and trace every task instance in a central place (for a consumer instance), and check the tasks at some point (when and how?) to clear completed ones, and when websocket disconnected, just cancel all tasks remaining active. We can add a MAX_ACTIVE_TASKS to limit the max tasks can be created to avoid creating too many slow tasks. If set MAX_ACTIVE_TASKS=1, we can use django-channels server as a broadcast service. It's comparable: |
I implemented a Consumer to dispatch message handler as tasks (only affect user defined tasks): In this implementation, you can send message such as "g1" to start the long running handler(chat_message), and still you can send "g2" to start the chat_message2 handler, if you send other data not endswith 1 or 2, you will receive a "invalid message" reply. Client intput and output: Server input and output: It's a full duplex websocket consumer now, we can send data to websocket server at any time and get response, and websocket server can send data at any time too. The long running consumer handler will not block the consumer in the new implementation. So, are you interested to implement it at the channels framework level ? @carltongibson @andrewgodwin |
You Saved me from a lot of troubles, May I ask why asgiref is the issue and how you found it ? |
Thanks!! This worked for me. |
#205, same issue I met, maybe there's something about djangorestframework+channels+django+asgiref(new version) |
|
We're having the same problem using GraphQL so I'd remove djangorestframework from the picture 🤔 |
Experiencing the same with GraphQL 😭 |
|
Has this been solved, because i have the same issue and i haven't been able to fix it |
|
in my case it was that the disk space was full and i just freed up some space and it was solved ! |
|
For anyone else with this error: my solution was to downgrade aioredis to 1.3.0. The newest version (2.0) is a complete rewrite and requires restructure of code, which I believe channels-redis has not taken into account when requiring aiosredis. |
|
I have a Django project.
Please help me out over here, |
@ReadMost I am facing for some weeks the same problem. How did you solve it? |
|
@alvaro-alonso I have written my solution on the top or here is link |
|
I managed to resolve this issue in local dev without needing to use both wsgi and asgi servers together (
const options = {
WebSocket: WebSocket,
connectionTimeout: 10000,
maxRetries: 3,
minUptime: 2000,
maxEnqueuedMessages: 1000000,
};
const newSocket = new ReconnectingWebSocket(
'ws://' +
window.location.host +
'/ws/somepath/' +
roomName +
'/' +
'?' +
queryString,
[],
options
);
Initially, this worked fine for connection timeouts. However, in my particular case, my disconnect method requires a relatively lengthy ORM operation involving celery workers. Therefore, after switching to a remote database, the disconnect method became an issue once again. Fortunately, there is a simple workaround. Instead of using the development server — i.e. Simply run: daphne -t 60 --application-close-timeout 60 your_project.asgi:applicationThe Of course, with this approach, you will need to serve any local static files through `STATIC_ROOT'. Whitenoise can be used to serve static files. It will work with an ASGI server (within Django anyway. Outside of Django I don't think it supports ASGI). If you don't want file caching, replace their suggested STATICFILES_STORAGE with: STATICFILES_STORAGE = 'whitenoise.storage.CompressedStaticFilesStorage'Hope this helps someone. Thanks to everyone for such an awesome project🙏 💯 |
@benedictkioko I have the same problem using graphene_django + channels, i solved it use the command "daphne -b 0.0.0.0 -p 8000 --application-close-timeout 60 --proxy-headers core.asgi:application", hope this can help someone. daphne help doc: |
I changed from asgiref==3.2.7 to asgiref==3.2.10 worked for me on django 2.2 |
|
I am not sure if my issue is the same, but it results in a similar error. In my case, I am running my Django Channels ASGI app using Daphne, and I have a couple API calls that call an external API. I found that when that external API errored out and I did not have a timeout set on the requests call, it would hang the Daphne server. While I did have a flaw in my code, I do not think that such a flaw should render the entire server unresponsive and not able to be restarted through systemctl. Note that when I switched to uvicorn, I no longer had an issue even with the misbehaving code. In my real application, I have of course fixed my external API calls to have an explicit timeout. Here is a simplified repro case based on my real application: Here is my exact error: |
|
@jessamynsmith Thank you so much for posting this!! I thought I was losing my mind 😆 I'm in a similar situation -- my API calls are taking too long, so if a user navigates away from the page where the websocket was instantiated, there's no chance for it to cleanly exit, leading to the "took too long to shut down and was killed" error message. |
Looking back on this thread, in the case of blocking I/O or CPU intensive operations during a disconnect method, it might be worth offloading those tasks to a worker, with something like celery or RQ. |
That was my thought too. I was considering using Celery, but Celery isn't meant to be a real-time queue, and more importantly, if you offload the task to Celery you would have a hard(er) time pushing the result back to the channels consumer. I guess you could submit the task and then continually poll it, waiting for it to be complete, but I didn't like that approach. Basically, what I did was create a Using this setup, It's probably not the most "elegant" way to solve the problem, but it keeps the solution entirely inside the Edit: If my logic/thought process about not using Celery here is incorrect I would love to know. |
|
Update I refactored the |
|
@jrosebr1 Nice! It depends on the use case I guess, but if you need to send a result to the client from outside the consumer class, you could send it via a channel layer. You could also use something like python-socketio instead of channels to emit a message from an external process. |
|
Thanks @Saran33, I'll take a look at those. |
|
@jessamynsmith saved my life |
and others
first message is succeed but further messages throw this exception:
My consumer class
requirements.txt
How to solve my issue?
Thanks in advance.
The text was updated successfully, but these errors were encountered: