python3.11 + Celery + Redis + Gevent -> "returned NULL without setting an exception" #1985
Comments
|
Having the same issue on Huey + redis + gevent since upgrading to py3.11. Someone else (here) had the same error and is not using redis. That seems to point to gevent as culprit. |
|
What greenlet version does everyone have installed? |
|
|
Because the gevent
We can rule out the first one by running gevent without its C extensions: set the We could also test with greenlet 3 and see if that makes a difference. It's a drop-in replacement for greenlet 2, so you can just That's about all I can suggest until I have a reproducible test case. (I don't know when I will have time to dive into the greenlet code and go speculatively hunt bugs.) |
|
We tried it already with Greenlet 3, but the problem still occurs. So just switching to Greenlet 3 unfortunately will not work. |
|
Another note: there was a CPython issue of similar sort, and it was closed as miscompilation by GCC4 was blamed - python/cpython#94825. |
|
I started getting this error after upgrading to Python 3.11 ( Exception ignored in:
<function AsyncResult.__del__ at 0x7f07761cb2e0>
Traceback (most recent call last):
File "/venv/lib/python3.11/site-packages/celery/result.py", line 392, in __del__
self.backend.remove_pending_result(self)
File "/venv/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 208, in remove_pending_result
self.on_result_fulfilled(result)
File "/venv/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 216, in on_result_fulfilled
self.result_consumer.cancel_for(result.id)
File "/venv/lib/python3.11/site-packages/celery/backends/redis.py", line 176, in cancel_for
self._pubsub.unsubscribe(key)
File "/venv/lib/python3.11/site-packages/redis/client.py", line 1659, in unsubscribe
return self.execute_command("UNSUBSCRIBE", *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/redis/client.py", line 1469, in execute_command
self.connection = self.connection_pool.get_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/redis/connection.py", line 1457, in get_connection
connection.connect()
File "/venv/lib/python3.11/site-packages/redis/connection.py", line 699, in connect
sock = self.retry.call_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
^^^^
File "/venv/lib/python3.11/site-packages/redis/connection.py", line 700, in <lambda>
lambda: self._connect(), lambda error: self.disconnect(error)
^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/redis/connection.py", line 970, in _connect
for res in socket.getaddrinfo(
^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo
addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/gevent/resolver/thread.py", line 63, in getaddrinfo
return self.pool.apply(_socket.getaddrinfo, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/gevent/pool.py", line 161, in apply
return self.spawn(func, *args, **kwds).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 356, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
File "src/gevent/_abstract_linkable.py", line 455, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
gevent.exceptions.InvalidSwitchError:
Invalid switch into AsyncResult.wait(): None
Traceback (most recent call last):
File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch
SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7f076ee52340> returned NULL without setting an exception
<built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7f076ee52340> failed with SystemError
Unrecoverable error: SystemError('<built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7f076ee52340> returned NULL without setting an exception')
Traceback (most recent call last):
File "/venv/lib/python3.11/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/venv/lib/python3.11/site-packages/celery/worker/loops.py", line 130, in synloop
connection.drain_events(timeout=2.0)
File "/venv/lib/python3.11/site-packages/kombu/connection.py", line 316, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/kombu/transport/virtual/base.py", line 971, in drain_events
get(self._deliver, timeout=timeout)
File "/venv/lib/python3.11/site-packages/kombu/transport/redis.py", line 581, in get
events = self.poller.poll(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/kombu/utils/eventio.py", line 279, in poll
read, write, error = _selectf(
^^^^^^^^^
File "/venv/lib/python3.11/site-packages/gevent/select.py", line 199, in select
return result.select(rlist, wlist, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/gevent/select.py", line 136, in select
event.wait(timeout=timeout)
File "src/gevent/event.py", line 163, in gevent._gevent_cevent.Event.wait
File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch
SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7f076ee52340> returned NULL without setting an exception
Error on stopping Pool: This operation would block forever
Hub: <Hub '' at 0x7f07738710d0 epoll default pending=0 ref=0 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f076f7a02d0 pool=<ThreadPool at 0x7f076f6bbae0 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f07738710d0 thread_ident=0x7f0778df5b48>>> threadpool=<ThreadPool at 0x7f076f6bbae0 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f07738710d0 thread_ident=0x7f0778df5b48>> thread_ident=0x7f0778df5b48>
Handles:
[]
Traceback (most recent call last):
File "/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 148, in send_all
fun(parent, *args)
File "/venv/lib/python3.11/site-packages/celery/bootsteps.py", line 369, in stop
return self.obj.stop()
^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/celery/concurrency/base.py", line 120, in stop
self.on_stop()
File "/venv/lib/python3.11/site-packages/celery/concurrency/gevent.py", line 101, in on_stop
self._pool.join()
File "/venv/lib/python3.11/site-packages/gevent/pool.py", line 430, in join
result = self._empty_event.wait(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/gevent/event.py", line 163, in gevent._gevent_cevent.Event.wait
File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever
Hub: <Hub '' at 0x7f07738710d0 epoll default pending=0 ref=0 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f076f7a02d0 pool=<ThreadPool at 0x7f076f6bbae0 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f07738710d0 thread_ident=0x7f0778df5b48>>> threadpool=<ThreadPool at 0x7f076f6bbae0 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f07738710d0 thread_ident=0x7f0778df5b48>> thread_ident=0x7f0778df5b48>
Handles:
[] |
|
I'll try to come up with a repro if nobody beats me to it |
|
here is my using library version:
when I use Python3.11 running in Docker, it will be hit this SystemError in random time. and This error cannot be occurred stably, it confused me a long time. Because I want to upgrade my python version from 3.7 to 3.11, and I must upgrade the greenlet and gevent version, but they can not run stably in 3.11. here is the stacktrace: Traceback (most recent call last):
File "./start.py", line 44, in <module>
File "./base/server.py", line 144, in start
File "src/gevent/greenlet.py", line 1065, in gevent._gevent_cgreenlet.joinall
File "src/gevent/greenlet.py", line 1075, in gevent._gevent_cgreenlet.joinall
File "src/gevent/_hub_primitives.py", line 250, in gevent._gevent_c_hub_primitives.wait_on_objects
File "src/gevent/_hub_primitives.py", line 287, in gevent._gevent_c_hub_primitives.wait_on_objects
File "src/gevent/_hub_primitives.py", line 185, in gevent._gevent_c_hub_primitives._WaitIterator.__next__
File "src/gevent/_hub_primitives.py", line 176, in gevent._gevent_c_hub_primitives._WaitIterator.__next__
File "src/gevent/_waiter.py", line 195, in gevent._gevent_c_waiter.MultipleWaiter.get
File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch
SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7f05aeb12520> returned NULL without setting an exceptionhop someone can help to solve this or provide some ideas I can try, thanks! :( |
|
I found an stable reoccurred exception, When the requests library raise this exception in one coroutine, the program will be shutdown abnormally and occur the SystemError above.
Here is the stacktrace: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 454, in getresponse
httplib_response = super().getresponse()
^^^^^^^
RuntimeError: super(): __class__ cell not found |
|
Same issue with python 3.11 gevent 23.7.0 and redis 5.0.0 |
|
Same issue, but on Flask with uwsgi and unrelated to redis. gevent version: |
|
Hey guys, Readding my comment from #1987 as that one was closed, but may be helpful for some people
Thanks! |
|
Still happening to me after downgrading |
|
I haven't been able to come up with a reproducer for this, despite my best efforts. But I've found and fixed some other issues that might be related. gevent 23.8.0 and greenlet 3.0rc1 have those changes. They are building and releasing now. When possible, please give those versions a try. Pay close attention to the standard error output. There were some places in the C code that could silently swallow exceptions --- they have to swallow exceptions by design, because it's not safe to raise them (much the same way that exceptions from And of course, if anyone has a standalone reproducer, please share it. |
|
I think the gevent version with related fixes is 23.9.0? |
|
Still seeing this on macOS with these versions: greenlet==3.0.0rc1 The exact exception I'm seeing is For testing purposes I patched greenlet.cpp to raise an exception in green_switch when it detects a NULL return value with !PyErr_Ocurred(): This at least allows the main thread to keep running so I could see what the last thing that ran was. With some greenlet tracing I added enabled I see this as the last greenlet to run: stack trace for above greenlet: When it happens to me, the last greenlet always seems to be in urllib3, trying to connect. |
|
@dmbaggett Since you have the ability to compile and run greenlet with a scenario that crashes, can you please compile and run greenlet (and ideally gevent) with assertions enabled? e.g., something like This enables numerous assertions that may result in a crash closer to the source of the problem. The |
|
Seeing this: Assertion failed: (rhs), function operator<<=, file greenlet.cpp, line 867 But I can't seem to get a core dump. I have % ulimit -c But maybe there is some SIP issue with core dumps in recent macOS versions? No clue. Anyway, I'm going to try running under lldb so I can at least give you a C stack trace. |
|
|
I have a core file. How should I get it to you? |
|
@dmbaggett That is so helpful, thank you!
That...is...wow. Totally unexpected. I have no idea how that could happen (hence the assert!) but obviously it is. That gives me a good clue to start chasing down.
I appreciate that! Presumably it's very large? My macOS core files come out around 1.1GB gzip compressed, which makes them annoying to move around. Because you were able to get the C stack trace and the assertion failure, the core file is a bit less important, and I'm not sure what exactly I would look for in it right now. Unless you already have a way that's very convenient and easy for you to share large files, let me go ahead and see what I can do with the information you've already provided before we worry about moving cores around. Thank you again! |
|
@jamadden Yes, the core file is huge. I can put it up via a link but I'd rather share it privately. Does github support direct messages so I can send you a URL? Just LMK if you want it. Thanks for investigating; I have to admit that I find the gevent code very hard to understand and debug. |
Not to my knowledge, but you could DM me at twitter (@ossmkitty) or send me an email. |
|
Could always use gpg or something to encrypt the link with jamadden's public key |
|
Trying to get my web server to actually serve this 1.4GB file... I could FedEx you a CD [Edit: DVD]? (I kid.) |
|
Also very weird; Python printed this: XXX lineno: 350, opcode: 150 ...which seems to come from this section of ceval.c: Unfortunately I cannot tell when or from where that was printed -- i.e., what code had the unknown opcode. Possibly related to python/cpython#97002 which was closed with the comment "I finally found the root issue, which is that a poorly-timed GC collection just before tracing the resumption of a generator can cause two duplicate frame objects to be created, which corrupts the VM's internal state in all sorts of fun ways." |
|
My working theory also involves tracing and/or garbage collection happening at inopportune times 😄 I have been able to contrive scenarios that result in triggering either the assertion you saw or different assertions. I don't know if they're close enough to the actual scenario you're hitting to do any good, but I have been able to fix the scenarios I've found so far. This greenlet build has a few rounds of fixes in it. When it finishes, GHA will show the artifacts for it, which will include installable greenlet wheels for macOS and Linux systems. If you're able to test those in your situation and see what happens, that would be awesome. (The macOS wheels won't be built with debug assertions enabled, unfortunately, so it would be even better if you could build the same commit with debugging enabled.) I'm continuing to trace through the code looking for other ways unexpected situations can arise. |
As far as I can tell, that should be unrelated --- or at the most, there's been a lurking bug in the WSGI application that this happened to expose (this time). That's something that uWSGI checks internally when you call an API. Because uWSGI works on an "implicit" request basis, each API call checks a thread-local (I think greenlet local when running uWSGI with gevent) variable to see if there is a current request being handled. If there isn't, uWSGI raises this error. So this code is getting run from outside the scope of a uWSGI request (hence there's nothing to close). The changes in rc2 boil down to two simple classes of changes:
If that second case, garbage collection happening slightly differently, is what's affecting the WSGI application, I think I would consider that a bug in the WSGI application. Just as in normal Python, garbage collection can occur at arbitrary times and in arbitrary threads or arbitrary greenlets. So trying to close a request inside GC isn't safe, with or without gevent. If that's what's happening, the uWSGI log ought to show "[BUG] current_wsgi_req NOT FOUND !!!". |
|
Wow @jamadden, nice work! I will test with the new code. What git branch should I use now to build from source? |
|
There's a 3.0rc2 tag, or you can use the main branch. (The changes in the main branch since rc2 are just mechanical refactoring, moving stuff around. No semantic changes.) |
Sorry about that and yes, you're right. I reverted greenlet back to 2.0.2 and the problem still occurs. It must be that some other library got updated when I rebuilt my container. Serves me right for not freezing the versions in requirements. 😒 |
|
Reporting back: I've been testing with 3.0rc2 for a few days now and have not seen any issues. |
|
Thank you, that's very good to hear. While running stress tests, I did find one more crasher that I fixed in greenlet 3.0rc3. It occurred at process termination time under very specific circumstances; if you were going to see it, you probably would have by now (assuming you've run one than one process over the last few days 😄 ). So I think we can consider this fixed now. The most recent release of gevent (23.9.1) bumps the required greenlet dep for CPython 3.11 and 3.12, so installing it will automatically Do The Right Thing to avoid the issue. (I know, I know, and I don't like releasing final versions that have a dependency on an RC either, but I'm still hoping to be able to figure out the tracing thing under Python 3.12 before Python 3.12 is officially released. Either way, greenlet 3.0 will go final when Python 3.12 does, and I'll release a new version of gevent that depends on it ASAP.) Thank you, everyone, for your patience and assistance with this issue. |
|
Amazing work -- thanks @jamadden! |
|
Thanks a lot @jamadden for the tireless research! Congratulations! |
|
@jamadden I think we still have an issue. :( When I run my load test for a long time my code is getting into a mode where the server is accepting queries but seems to not be able actually do anything. I'll need to to a lot more investigating to see where it's getting stuck, but one very suspicious aspect here is that I run a garbage collection greenlet that does a gc.collect() every 60 seconds, and the hangs correlate to gc.collect having just run. I'm running greenlet with -O0 and -UNDEBUG but I'm not seeing any assertions or really anything unusual getting printed. Unfortunately it takes a long time for this to randomly happen so it might be more productive to try to force this to happen with one of your test harneses. Try running a separate greenlet that collects and maybe we'll get lucky and you'll reproduce it locally. I'll keep researching here. |
|
Update on this: I have been running for around 20 hours with PURE_PYTHON=1 and it hasn't locked up. I've also been able to reproduce it a few more times with gevent running without PURE_PYTHON=1 and I no longer believe it's necessarily correlated with the GC greenlet running a collection. |
|
I finally managed to get a stack trace showing where the greenlet get stuck: 2023-09-17 23:00:15.531974z P:7821 G:0x38f65ab60 [analysis-server] [GREENLET] ^-- gevent/thread.py:112, ldclient/rwlock.py:20, ldclient/init.py:109, addinserver/feature_flags.py:95, addinserver/analysis.py:4372, addinserver/analysis.py:3794, addinserver/inky_analysis_server.py:3865, addinserver/inky_analysis_server.py:3096, addinserver/inky_analysis_server.py:2701, web/application.py:495, web/application.py:517, web/application.py:271, web/application.py:280, web/application.py:278, web/application.py:703, web/application.py:278, web/application.py:278, web/application.py:686, web/application.py:278, web/application.py:278, web/application.py:686, web/application.py:278, web/application.py:278, web/application.py:703, web/application.py:278, web/application.py:278, web/application.py:686, web/application.py:278, web/application.py:290, web/application.py:319, gevent/pywsgi.py:1053, gevent/pywsgi.py:1107, gevent/pywsgi.py:804, gevent/pywsgi.py:574, gevent/pywsgi.py:1700, gevent/server.py:210, gevent/baseserver.py:34 It's getting stuck waiting for a lock in the launch darkly client. Here is the relevant code: def get(): I'm using a rather old version of launchdarkly-server-sdk (6.10.2) but it's a pure python package so it still seems problematic that its locking code can hang a greenlet. The underlying implementation is: But on second thought I guess this code implies there is no timeout, so the Condition will block forever if the lock is never released -- so if some other greenlet dies without releasing the lock this would explain the behavior we're seeing here. (Why is the finally clause not running, though?) @jamadden I would say ignore this until I can debug further and determine it's actually a greenlet/gevent issue. |
|
Hmm, I noticed this, which seems very odd: redis.exceptions.TimeoutError: Timeout reading from socket During handling of the above exception, another exception occurred: Traceback (most recent call last): greenlet_switch geting a TimeoutError? |
|
That's normal. Uncaught exceptions are thrown to parent greenlets as a result of |
|
Further update: same result with the latest launch darkly SDK. However, when I force the use of _AtomicSemaphore in gevent/lock.py the problem seems to go away. Of course, this could be luck and timing. I will keep my tests running. |
|
I'm starting to believe this is a real gevent issue. I've had two tests running since yesterday morning on two different machines and have not seen any hangs. The only change I've made is +if PURE_PYTHON or True: in src/gevent/lock.py I'll keep these load tests running for the rest of the week. |
|
I have not seen any hangs since applying the one-line patch above so I think this a real gevent issue. I wonder if something has changed with Cython since a comment in lock.py says that it's relying on Cython's not releasing the GIL. |
|
So what's up with this issue now? |
|
FWIW I'm seeing different behavior under Python 3.12 than I did under Python 3.11 so I'm really not sure what to think at this point. Seems like we need to do some more testing. |
|
To elaborate: I've upgraded our stack to 3.12 and haven't seen any problems under gevent 23.9.1 and greenlet 3.0.0rc3 under load testing. Seems like a bad interaction between gevent and something in 3.11. (I have not tried 3.10 or 3.9, to be fair with the latest greenlet/gevent.) |
|
So, interestingly we are also affected by segfaults due to garbage collector invalidating objects between threads in Python 3.11 in Sentry Profiler: getsentry/sentry-python#2386 . I wonder if this stuff has the same nature as the problems that we suffered with Gevent - it would great if this is all fixed in Python 3.12. |
|
@zyv After all of @jamadden 's fixes I did not actually see crashes even under 3.11 -- the remaining symptom I saw over long-term testing was very specific I/O-bound greenlets just getting stuck forever. Not seeing anything of the sort under 3.12 and I've been wanting to move to 3.12 anyway given that it supports Python stack traces in linux perf. |
|
I suppose the nogil release of Python (3.13? 3.14? 3.15?) will blow everything up again for us all... |
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 2.0.2 to 3.0.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.0 (2023-10-02)</h1> <ul> <li>No changes from 3.0rc3 aside from the version number.</li> </ul> <h1>3.0.0rc3 (2023-09-12)</h1> <ul> <li>Fix an intermittent error during process termination on some platforms (GCC/Linux/libstdc++).</li> </ul> <h1>3.0.0rc2 (2023-09-09)</h1> <ul> <li>Fix some potential bugs (assertion failures and memory leaks) in previously-untested error handling code. In some cases, this means that the process will execute a controlled <code>abort()</code> after severe trouble when previously the process might have continued for some time with a corrupt state. It is unlikely those errors occurred in practice.</li> <li>Fix some assertion errors and potential bugs with re-entrant switches.</li> <li>Fix a potential crash when certain compilers compile greenlet with high levels of optimization. The symptom would be that switching to a greenlet for the first time immediately crashes.</li> <li>Fix a potential crash when the callable object passed to the greenlet constructor (or set as the <code>greenlet.run</code> attribute) has a destructor attached to it that switches. Typically, triggering this issue would require an unlikely subclass of <code>greenlet.greenlet</code>.</li> <li>Python 3.11+: Fix rare switching errors that could occur when a garbage collection was triggered during the middle of a switch, and Python-level code in <code>__del__</code> or weakref callbacks switched to a different greenlet and ultimately switched back to the original greenlet. This often manifested as a <code>SystemError</code>: "switch returned NULL without an exception set."</li> </ul> <p>For context on the fixes, see <code>gevent issue [#1985](https://github.com/python-greenlet/greenlet/issues/1985) <https://github.com/gevent/gevent/issues/1985></code>_.</p> <h1>3.0.0rc1 (2023-09-01)</h1> <ul> <li>Windows wheels are linked statically to the C runtime in an effort to prevent import errors on systems without the correct C runtime installed. It's not clear if this will make the situation better or worse, so please share your experiences in <code>issue 346 <https://github.com/python-greenlet/greenlet/issues/346></code>_.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/ef510e4b1a862800e77bc8e290769d688f691b9e"><code>ef510e4</code></a> Preparing release 3.0.0</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/8b24b4d2122c101efbe98a1d08dc02ffd3899826"><code>8b24b4d</code></a> Updating change log.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/79f63518ae53488c0acc19e9a1bfd740fc47149e"><code>79f6351</code></a> Back to development: 3.0.0rc4</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/715128a4df7f93b247ce03a202e6d892a288e9b2"><code>715128a</code></a> Preparing release 3.0.0rc3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/f2ecf0c01a2f9524198be1f3220985de8eb5f0ff"><code>f2ecf0c</code></a> D'oh, tstate is used under Py 3.11+. UNUSED breaks compilation on windows.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/745181099a6d62bec05d443bae14c7b4dcd0a56a"><code>7451810</code></a> Finish refactoring to separate files.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/748a9e064f42a04e208c7c32e1316b44ced18db9"><code>748a9e0</code></a> Fix an intermittent error during process termination on GCC/Linux/libstdc++.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/cec93b106247e16ef6003175a15f113baf1d4285"><code>cec93b1</code></a> Move BrokenGreenlet to its own file.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/0b108b39224093ccfbad7f948532167afbd55ff8"><code>0b108b3</code></a> Move UserGreenlet code to its own file.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/215d0c98dcbefd68357c90e86c526b00026171c8"><code>215d0c9</code></a> Moving Greenlet, module globals and thread state destruction to their own files.</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/2.0.2...3.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 2.0.2 to 3.0.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.0 (2023-10-02)</h1> <ul> <li>No changes from 3.0rc3 aside from the version number.</li> </ul> <h1>3.0.0rc3 (2023-09-12)</h1> <ul> <li>Fix an intermittent error during process termination on some platforms (GCC/Linux/libstdc++).</li> </ul> <h1>3.0.0rc2 (2023-09-09)</h1> <ul> <li>Fix some potential bugs (assertion failures and memory leaks) in previously-untested error handling code. In some cases, this means that the process will execute a controlled <code>abort()</code> after severe trouble when previously the process might have continued for some time with a corrupt state. It is unlikely those errors occurred in practice.</li> <li>Fix some assertion errors and potential bugs with re-entrant switches.</li> <li>Fix a potential crash when certain compilers compile greenlet with high levels of optimization. The symptom would be that switching to a greenlet for the first time immediately crashes.</li> <li>Fix a potential crash when the callable object passed to the greenlet constructor (or set as the <code>greenlet.run</code> attribute) has a destructor attached to it that switches. Typically, triggering this issue would require an unlikely subclass of <code>greenlet.greenlet</code>.</li> <li>Python 3.11+: Fix rare switching errors that could occur when a garbage collection was triggered during the middle of a switch, and Python-level code in <code>__del__</code> or weakref callbacks switched to a different greenlet and ultimately switched back to the original greenlet. This often manifested as a <code>SystemError</code>: "switch returned NULL without an exception set."</li> </ul> <p>For context on the fixes, see <code>gevent issue [#1985](https://github.com/python-greenlet/greenlet/issues/1985) <https://github.com/gevent/gevent/issues/1985></code>_.</p> <h1>3.0.0rc1 (2023-09-01)</h1> <ul> <li>Windows wheels are linked statically to the C runtime in an effort to prevent import errors on systems without the correct C runtime installed. It's not clear if this will make the situation better or worse, so please share your experiences in <code>issue 346 <https://github.com/python-greenlet/greenlet/issues/346></code>_.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/ef510e4b1a862800e77bc8e290769d688f691b9e"><code>ef510e4</code></a> Preparing release 3.0.0</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/8b24b4d2122c101efbe98a1d08dc02ffd3899826"><code>8b24b4d</code></a> Updating change log.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/79f63518ae53488c0acc19e9a1bfd740fc47149e"><code>79f6351</code></a> Back to development: 3.0.0rc4</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/715128a4df7f93b247ce03a202e6d892a288e9b2"><code>715128a</code></a> Preparing release 3.0.0rc3</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/f2ecf0c01a2f9524198be1f3220985de8eb5f0ff"><code>f2ecf0c</code></a> D'oh, tstate is used under Py 3.11+. UNUSED breaks compilation on windows.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/745181099a6d62bec05d443bae14c7b4dcd0a56a"><code>7451810</code></a> Finish refactoring to separate files.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/748a9e064f42a04e208c7c32e1316b44ced18db9"><code>748a9e0</code></a> Fix an intermittent error during process termination on GCC/Linux/libstdc++.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/cec93b106247e16ef6003175a15f113baf1d4285"><code>cec93b1</code></a> Move BrokenGreenlet to its own file.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/0b108b39224093ccfbad7f948532167afbd55ff8"><code>0b108b3</code></a> Move UserGreenlet code to its own file.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/215d0c98dcbefd68357c90e86c526b00026171c8"><code>215d0c9</code></a> Moving Greenlet, module globals and thread state destruction to their own files.</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/2.0.2...3.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 2.0.2 to 3.0.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst">greenlet's changelog</a>.</em></p> <blockquote> <h1>3.0.1 (2023-10-25)</h1> <ul> <li>Fix a potential crash on Python 3.8 at interpreter shutdown time. This was a regression from earlier 3.0.x releases. Reported by Matt Wozniski in <code>issue 376 <https://github.com/python-greenlet/greenlet/issues/376></code>_.</li> </ul> <h1>3.0.0 (2023-10-02)</h1> <ul> <li>No changes from 3.0rc3 aside from the version number.</li> </ul> <h1>3.0.0rc3 (2023-09-12)</h1> <ul> <li>Fix an intermittent error during process termination on some platforms (GCC/Linux/libstdc++).</li> </ul> <h1>3.0.0rc2 (2023-09-09)</h1> <ul> <li>Fix some potential bugs (assertion failures and memory leaks) in previously-untested error handling code. In some cases, this means that the process will execute a controlled <code>abort()</code> after severe trouble when previously the process might have continued for some time with a corrupt state. It is unlikely those errors occurred in practice.</li> <li>Fix some assertion errors and potential bugs with re-entrant switches.</li> <li>Fix a potential crash when certain compilers compile greenlet with high levels of optimization. The symptom would be that switching to a greenlet for the first time immediately crashes.</li> <li>Fix a potential crash when the callable object passed to the greenlet constructor (or set as the <code>greenlet.run</code> attribute) has a destructor attached to it that switches. Typically, triggering this issue would require an unlikely subclass of <code>greenlet.greenlet</code>.</li> <li>Python 3.11+: Fix rare switching errors that could occur when a garbage collection was triggered during the middle of a switch, and Python-level code in <code>__del__</code> or weakref callbacks switched to a different greenlet and ultimately switched back to the original greenlet. This often manifested as a <code>SystemError</code>: "switch returned NULL without an exception set."</li> </ul> <p>For context on the fixes, see <code>gevent issue [#1985](https://github.com/python-greenlet/greenlet/issues/1985) <https://github.com/gevent/gevent/issues/1985></code>_.</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-greenlet/greenlet/commit/77a9a2952b166b83011f5c3ebeb6ad2d6370bbfa"><code>77a9a29</code></a> Preparing release 3.0.1</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/11b1b758f5b3a43eb2e0a5499047ddef2517e3ec"><code>11b1b75</code></a> CI: Use 3.12 final instead of -dev.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/76b4ac08969f24c3a87a389f8bd37807547be6ce"><code>76b4ac0</code></a> Wheels: We're forcing the platform tag for universal2, not the api tag (which...</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/4751fbe89d247a7082ebc76bd7b88b649c9e8f0a"><code>4751fbe</code></a> Fix <a href="https://redirect.github.com/python-greenlet/greenlet/issues/376">#376</a></li> <li><a href="https://github.com/python-greenlet/greenlet/commit/309575a62b9853bc3b6367225e388a65796e9c7c"><code>309575a</code></a> Centralize use of psutil.Process.uss; deal gracefully when it is not available.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/313d06b0eac0541ad6bde3cb0e3effc0a9e6b5ce"><code>313d06b</code></a> Remove test status badge from README. It was confusing.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/660a66f339c61b2572ea6237cc042bde356419b1"><code>660a66f</code></a> Back to development: 3.0.1</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/ef510e4b1a862800e77bc8e290769d688f691b9e"><code>ef510e4</code></a> Preparing release 3.0.0</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/8b24b4d2122c101efbe98a1d08dc02ffd3899826"><code>8b24b4d</code></a> Updating change log.</li> <li><a href="https://github.com/python-greenlet/greenlet/commit/79f63518ae53488c0acc19e9a1bfd740fc47149e"><code>79f6351</code></a> Back to development: 3.0.0rc4</li> <li>Additional commits viewable in <a href="https://github.com/python-greenlet/greenlet/compare/2.0.2...3.0.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Switch to wheel.mk. 3.0.1 (2023-10-25) ================== - Fix a potential crash on Python 3.8 at interpreter shutdown time. This was a regression from earlier 3.0.x releases. Reported by Matt Wozniski in `issue 376 <https://github.com/python-greenlet/greenlet/issues/376>`_. 3.0.0 (2023-10-02) ================== - No changes from 3.0rc3 aside from the version number. 3.0.0rc3 (2023-09-12) ===================== - Fix an intermittent error during process termination on some platforms (GCC/Linux/libstdc++). 3.0.0rc2 (2023-09-09) ===================== - Fix some potential bugs (assertion failures and memory leaks) in previously-untested error handling code. In some cases, this means that the process will execute a controlled ``abort()`` after severe trouble when previously the process might have continued for some time with a corrupt state. It is unlikely those errors occurred in practice. - Fix some assertion errors and potential bugs with re-entrant switches. - Fix a potential crash when certain compilers compile greenlet with high levels of optimization. The symptom would be that switching to a greenlet for the first time immediately crashes. - Fix a potential crash when the callable object passed to the greenlet constructor (or set as the ``greenlet.run`` attribute) has a destructor attached to it that switches. Typically, triggering this issue would require an unlikely subclass of ``greenlet.greenlet``. - Python 3.11+: Fix rare switching errors that could occur when a garbage collection was triggered during the middle of a switch, and Python-level code in ``__del__`` or weakref callbacks switched to a different greenlet and ultimately switched back to the original greenlet. This often manifested as a ``SystemError``: "switch returned NULL without an exception set." For context on the fixes, see `gevent issue #1985 <https://github.com/gevent/gevent/issues/1985>`_. 3.0.0rc1 (2023-09-01) ===================== - Windows wheels are linked statically to the C runtime in an effort to prevent import errors on systems without the correct C runtime installed. It's not clear if this will make the situation better or worse, so please share your experiences in `issue 346 <https://github.com/python-greenlet/greenlet/issues/346>`_. Note that this only applies to the binary wheels found on PyPI. Building greenlet from source defaults to the shared library. Set the environment variable ``GREENLET_STATIC_RUNTIME=1`` at build time to change that. - Build binary wheels for Python 3.12 on macOS. - Fix compiling greenlet on a debug build of CPython 3.12. There is `one known issue <https://github.com/python-greenlet/greenlet/issues/368>`_ that leads to an interpreter crash on debug builds. - Python 3.12: Fix walking the frame stack of suspended greenlets. Previously accessing ``glet.gr_frame.f_back`` would crash due to `changes in CPython's undocumented internal frame handling <https://github.com/python/cpython/commit/1e197e63e21f77b102ff2601a549dda4b6439455>`_. Platforms --------- - Now, greenlet *may* compile and work on Windows ARM64 using llvm-mingw, but this is untested and unsupported. See `PR <https://github.com/python-greenlet/greenlet/pull/224>`_ by Adrian Vladu. - Now, greenlet *may* compile and work on LoongArch64 Linux systems, but this is untested and unsupported. See `PR 257 <https://github.com/python-greenlet/greenlet/pull/257/files>`_ by merore. Known Issues ------------ - There may be (very) subtle issues with tracing on Python 3.12, which has redesigned the entire tracing infrastructure. 3.0.0a1 (2023-06-21) ==================== - Build binary wheels for S390x Linux. See `PR 358 <https://github.com/python-greenlet/greenlet/pull/358>`_ from Steven Silvester. - Fix a rare crash on shutdown seen in uWSGI deployments. See `issue 330 <https://github.com/python-greenlet/greenlet/issues/330>`_ and `PR 356 <https://github.com/python-greenlet/greenlet/pull/356>`_ from Andrew Wason. - Make the platform-specific low-level C/assembly snippets stop using the ``register`` storage class. Newer versions of standards remove this storage class, and it has been generally ignored by many compilers for some time. See `PR 347 <https://github.com/python-greenlet/greenlet/pull/347>`_ from Khem Raj. - Add initial support for Python 3.12. See `issue <https://github.com/python-greenlet/greenlet/issues/323>`_ and `PR <https://github.com/python-greenlet/greenlet/pull/327>`_; thanks go to (at least) Michael Droettboom, Andreas Motl, Thomas A Caswell, raphaelauv, Hugo van Kemenade, Mark Shannon, and Petr Viktorin. - Remove support for end-of-life Python versions, including Python 2.7, Python 3.5 and Python 3.6. - Require a compiler that supports ``noinline`` directives. See `issue 271 <https://github.com/python-greenlet/greenlet/issues/266>`_. - Require a compiler that supports C++11.
Description:
What are you trying to get done, what has happened, what went wrong, and what did you expect?
Working with celery worker on a container, with redis broker, a failure is happening (see stacktrace below).
Others have experienced the same issue on different circumstances. (See more examples here ). There is also an issue opened on celery.
[2023-08-04 11:10:28,806] [ERROR] [MainThread] [consumer.py:248 - perform_pending_operations()] Pending callback raised: SystemError('<built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> returned NULL without setting an exception') Traceback (most recent call last): File "/home/.local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 246, in perform_pending_operations self._pending_operations.pop()() File "/home/.local/lib/python3.11/site-packages/vine/promises.py", line 160, in __call__ return self.throw() ^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/vine/promises.py", line 157, in __call__ retval = fun(*final_args, **final_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/kombu/message.py", line 131, in ack_log_error self.ack(multiple=multiple) File "/home/.local/lib/python3.11/site-packages/kombu/message.py", line 126, in ack self.channel.basic_ack(self.delivery_tag, multiple=multiple) File "/home/.local/lib/python3.11/site-packages/kombu/transport/virtual/base.py", line 670, in basic_ack self.qos.ack(delivery_tag) File "/home/.local/lib/python3.11/site-packages/kombu/transport/redis.py", line 379, in ack self._remove_from_indices(delivery_tag).execute() File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2120, in execute return conn.retry.call_with_retry( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/retry.py", line 46, in call_with_retry return do() ^^^^ File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2121, in <lambda> lambda: execute(conn, stack, raise_on_error), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 1975, in _execute_transaction self.parse_response(connection, "_") File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 2060, in parse_response result = Redis.parse_response(self, connection, command_name, **options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/client.py", line 1286, in parse_response response = connection.read_response() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 882, in read_response response = self._parser.read_response(disable_decoding=disable_decoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 349, in read_response result = self._read_response(disable_decoding=disable_decoding) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 359, in _read_response raw = self._buffer.readline() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 262, in readline self._read_from_socket() File "/home/.local/lib/python3.11/site-packages/redis/connection.py", line 212, in _read_from_socket data = self._sock.recv(socket_read_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/.local/lib/python3.11/site-packages/gevent/_socketcommon.py", line 666, in recv self._wait(self._read_event) File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch File "src/gevent/_waiter.py", line 122, in gevent._gevent_c_waiter.Waiter.switch SystemError: <built-in method switch of gevent._gevent_cgreenlet.Greenlet object at 0x7fcdb37de700> returned NULL without setting an exception```The text was updated successfully, but these errors were encountered: