AssertionError: AbstractLinkable._notifier is None in _notify_links method

[ronaldnwilliams]

• gevent version: "~21.12.0" pypi
• Python version: python:3.9 from slim-buster docker image
• Operating System: python:3.9-slim-buster docker image

Description:

Running django 3.2 using gunicorn 20.1. I keep seeing this traceback in my logs from gevent/_abstract_linkable.py assert self._notifier is notifier, (self._notifier, notifier).

Traceback (most recent call last):
File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
AssertionError: (None, <callback at 0x7f3af9646600 args=([],)>)

No other logs surrounding these in the given context.

Is this something to be concerned about or anything I can fix?

What I've run:

ddtrace-run gunicorn --workers 5 --worker-class gevent project_name.wsgi:application

[dannygoldstein]

I am also running into this issue

[dannygoldstein]

Digging into this a bit more, I've found that when I start my application with

ddtrace-run gunicorn -k gevent main:app

gevent version: "~21.12.0" pypi
Python version: python:3.9 from slim-buster docker image
Operating System: python:3.9-slim-buster docker image

I see these errors emitted at the beginning

Traceback (most recent call last):
File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
AssertionError: (None, <callback at 0x7f3af9646600 args=([],)>)

but the server actually does startup and handles http requests in an asynchronous way. I tried load testing this by sending ~1000 requests to an endpoint that time.sleep()s for 5 seconds and another that makes an http request to a server that sleeps for five seconds and in both cases all 1000 requests returned in 5 seconds. So it seems like these errors are not actually affecting the performance of the server. This is by no means comprehensive and it may be that certain parts of the library are broken but the ones I tested work. But based on my limited experience it seems that things are still working. It would be great to understand what is causing this error and understand its broader scope

[jd]

I don't know why it does this, but I found a workaround for gunicorn which is:

diff --git gunicorn/arbiter.py gunicorn/arbiter.py
index 0e705075..aacf883d 100644
--- gunicorn/arbiter.py
+++ gunicorn/arbiter.py
@@ -612,6 +612,7 @@ class Arbiter(object):
         of the master process.
         """

+        time.sleep(0)
         for _ in range(self.num_workers - len(self.WORKERS)):
             self.spawn_worker()
             time.sleep(0.1 * random.random())

It does not make much sense to me, but it seems that yielding back the control to the hub prevents this issue from happening.

[SuperH-0630]

I am also running into this issue.

[balanikaran]

Hey there!

I'm also running into the same issue.
The setup is basically to use gunicorn with gevent as the worker class with ddtrace. This is a Django Application.
I'm using Python 3.10.6.

Any update on the issue?

[arfn1366]

Maybe this help! I had same problem when I tried to to use monkey patch in gunicorn server hook: when_ready.

[JackLiuCrypto]

I don't know why it does this, but I found a workaround for gunicorn which is:

diff --git gunicorn/arbiter.py gunicorn/arbiter.py
index 0e705075..aacf883d 100644
--- gunicorn/arbiter.py
+++ gunicorn/arbiter.py
@@ -612,6 +612,7 @@ class Arbiter(object):
         of the master process.
         """

+        time.sleep(0)
         for _ in range(self.num_workers - len(self.WORKERS)):
             self.spawn_worker()
             time.sleep(0.1 * random.random())

It does not make much sense to me, but it seems that yielding back the control to the hub prevents this issue from happening.

Thanks for the workaround. But after using that, the original http RESTful server function is not working.

[twall]

I'm seeing this from multiprocessing.Process spawned from a gunicorn process. The subprocess sometimes dies almost immediately and other times after a little bit of processing (I/O w/redis and/or elasticsearch, mostly).

[arcivanov]

multiprocessing requires https://github.com/karellen/geventmp to function.

[Pajinek]

I have started to receive this error when I upgrade the package gevent from 23.9.1 to 24.2.1 (Python image 3.11.8)

2024-02-15T17:21:44Z <callback at 0x7f4f63cd7800 args=([],)> failed with AssertionError
AssertionError: (None, <callback at 0x7f4f63cd7800 args=([],)>)
File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links