Using asgiref.local.Local with gevent leads to memory leaks #144
Comments
reupen
changed the title
|
Yeah, Are you letting |
Yes, Gunicorn does that when using gevent workers: https://github.com/benoitc/gunicorn/blob/7d8c92f48a6d9cee6b15fbdade6981721182d073/gunicorn/workers/ggevent.py#L38 |
|
That'll be the problem then. Hard to reason about threads when they're... not threads. I'll need to think about the best approach to this one. I really don't want to have asgiref depend on gevent at all, but detecting that threads aren't threads is going to be tricky otherwise. |
|
@jamadden – any input or thoughts about this? |
We've been suffering from a memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. This adds a workaround which is inactive by default. This is so it can be we can check that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak was causing worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak is causing Gunicorn worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak is causing Gunicorn worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak is causing Gunicorn worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak is causing Gunicorn worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
We've been suffering from a problematic memory leak since upgrading to Django 3 which is believed to be caused by django/asgiref#144. The memory leak is causing Gunicorn worker processes to exhaust available memory and crash and restart. This adds a workaround which is inactive by default. This is so it can be we can verify that it fixes the problem in e.g. the dev environment.
reupen
changed the title
|
OK, this might well be solved by the new patch for Local that removes directly inspecting the class type of threads (https://github.com/django/asgiref/tree/new_local) If someone wants to try installing the |
|
I've given it a quick test locally – it does indeed seem to have fixed the problem. Thanks! |
|
Alright, I'll get it merged in then, as it seems to solve at least two problems and not cause any issues! |
This one stores on threads/tasks directly, much like the implementation in the standard library. This removes the need for garbage-collection of local storage, and should be more compatible with libraries that override threading, like gevent. Fixes #144.
We use Django with Gunicorn and gevent workers. Since updating to Django 3, we started seeing some reasonably severe memory leaks in our web processes.
I bisected one of the leaks I could reproduce locally to django/django@a415ce7.
In particular, it seems to be caused by the switch from
threading.localtoasgiref.local.Local.I investigated a fair bit. There seem to be a few things going on:
this
isinstance(key, threading.Thread)test evaluating toFalse(Note:
else:is not handled there at all.)key.is_alive()also evaluating toFalse(this is after working around the previous problem)(Also, I'm not sure how gevent-friendly
CLEANUP_INTERVAL = 60is.)It could be argued that gevent-related bugs are a problem with gevent. However, I think this problem is quite severe and also not easily noticed as it's mostly a silent problem.
While any fixes (here or in gevent) would be appreciated, it would be good to document the incompatibility too.
(I tested gevent 1.4.0 and 1.5a3.)
Thanks
The text was updated successfully, but these errors were encountered: