eventlet 0.36.0 incompatible with flask-socketio running under gunicorn

[dhdaines]

Haven't figured out a minimal test case for this yet, but our CI pipelines for https://github.com/roedoejet/g2p/ have all stopped working because of the 0.36.0 release. We are running a background flask-socketio server with gunicorn and eventlet, and trying to access it via Playwright, which hangs forever when trying to connect to the socket.

The documentation seems to indicate that we should not use eventlet anymore, so we'll do that :) but just to note that the most recent eventlet seems to be particularly ill-behaved.

[4383]

Hello @dhdaines,

Thanks for reporting this problem.
We recently modified how the eventlet wsgi server behaves with sockets (1f3db00), so, indeed, it could have some side effects on your tests.

Please can you share links where we can observes that broken jobs?

If you are able to migrate off of eventlet then fire! That's the solution we encourgage.

Else, having some details about your recent problem can help us troubleshooting what happens with the last version, and hence, helps far less lucky other users that can't migrate off easily.

[dhdaines]

Sure! Not sure how much it will help but here is before reverting eventlet:

https://github.com/roedoejet/g2p/actions/runs/8422888910/job/23064740602

And here is after:

https://github.com/roedoejet/g2p/actions/runs/8423476069/job/23065192040

We'll try using gevent, since it seems like that might be the only alternative Flask supports...

[4383]

Rather than moving to gevent, what do you think of using quart as a replacement for flask?

Quart is the official response of the flask team to asyncio. I'd encourage that solution rather than the gevent solution, which at some point, will surely lead you to the same maintenance problems that are currently seen in eventlet, see #824.

Quart offer the same API as Flask.

For other candidates for replacement, you can take a look to this curated list:
https://github.com/timofurrer/awesome-asyncio

I just proposed a pull request to update our migration doc with awesome-asyncio #947

[4383]

I don't know how is designed your code base, but, either you could switch to the Eventlet asyncio hub and then migrate to quart or, directly move to quart if you do not use eventlet for something else in your code base.

That will be a win-win accomplishment.

Forgot to thank you for your previous links :)

[dhdaines]

Thank you! At one point I already recreated our API + web app using FastAPI, but we didn't end up switching because Flask is fairly embedded in our codebase - so if Quart has the same API it seems like the migration would be simpler.

We don't particularly care about asyncio (though I agree that it is better), we just need something that supports SocketIO, but it seems that to do so with Flask in production requires eventlet, gevent, or something else, which from what I understand are all somewhat of a compromise.

To reproduce the error, it's not all browsers that seem to trigger it - Safari definitely does, while Chrome on Linux seems to work okay, yet somehow Playwright hangs in testing. Full sequence to get the error is:

git clone https://github.com/roedoejet/g2p.git
pip install g2p
gunicorn --worker-class eventlet -w 1 g2p.app:APP --no-sendfile --bind 0.0.0.0:5000
python g2p/g2p/tests/test_studio.py  # in another window obiously

Actually it seems that the error may be triggered by the websocket upgrade request, as I get this exception:

[2024-03-26 08:45:40 -0400] [16229] [ERROR] Error handling request /socket.io/?EIO=4&transport=websocket&sid=62vsmkRvQIZ13WscAAAU
Traceback (most recent call last):
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/flask/app.py", line 2552, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/flask_socketio/__init__.py", line 43, in __call__
    return super(_SocketIOMiddleware, self).__call__(environ,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/engineio/middleware.py", line 63, in __call__
    return self.engineio_app.handle_request(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/socketio/server.py", line 430, in handle_request
    return self.eio.handle_request(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/engineio/server.py", line 274, in handle_request
    packets = socket.handle_get_request(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/engineio/socket.py", line 90, in handle_get_request
    return getattr(self, '_upgrade_' + transport)(environ,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/engineio/socket.py", line 146, in _upgrade_websocket
    return ws(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/engineio/async_drivers/eventlet.py", line 44, in __call__
    return super().__call__(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dhd/work/ReadAlongs/venv/lib/python3.11/site-packages/eventlet/websocket.py", line 137, in __call__
    environ['eventlet.set_idle']()
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^
KeyError: 'eventlet.set_idle'

[dhdaines]

Oh. So, in fact, it is definitely related to that change you mentioned above (1f3db00)

[dhdaines]

So, this will fix it, but I don't know enough about the internals of eventlet to know if it's the right fix:

diff --git a/eventlet/websocket.py b/eventlet/websocket.py
index 9d6becc2..a2eba800 100644
--- a/eventlet/websocket.py
+++ b/eventlet/websocket.py
@@ -134,8 +134,8 @@ class WebSocketWSGI:
 
         # We're ready to switch protocols; flag the connection
         # as idle to play well with a graceful stop
-        environ['eventlet.set_idle']()
-
+        if 'eventlet.set_idle' in environ:
+            environ['eventlet.set_idle']()
         try:
             self.handler(ws)
         except OSError as e:

[dhdaines]

Ah, I understand, the problem is that WebSocketWSGI is being used from a server that is not eventlet.wsgi.Server and so the magical environment function is not set. There are examples in code below where we simply check if keys are set in there, so it should be the right fix I believe. I'll dig a bit more as this is interesting to understand, then make a PR.

[4383]

@dhdaines: thanks for all these details.

Do not hesitate to ping me when you propose your PR, I'll be happy to review it.

@tipabu: FYI, this issue may interest you.

[dhdaines]

@4383 okay, PR submitted! as noted ... mocking a Gunicorn worker is not something I have the time or talent to do at the moment so I couldn't add a test for it, but I can confirm that it fixes the problem

[flyingclimber]

Ran into this as well. I ended up rolling back to 0.35.2 which worked fine

[dhdaines]

Ran into this as well. I ended up rolling back to 0.35.2 which worked fine

Yes, you can simply version your dependency to "eventlet!=0.36.0" and that should work as I presume the bug will be fixed in the next release, being rather uncontroversial :)

[4383]

Thanks @dhdaines for your PR and for the advices.

I'm going to merge your fix. Once merged I'll start to prepare a new release (0.36.1).
@flyingclimber by pinning your requirements like David suggested, you will be able to pull this new version.

[4383]

Fix released!

Let us know if it fixed your problem and feel free to reopen this issue if you observe unexpected behaviors.

[dhdaines]

Thank you so much! Just reran our test suite, it picked up 0.36.1, and everything works: https://github.com/roedoejet/g2p/actions/runs/8452592573/job/23244684343?pr=345

[4383]

You are welcome, that have been our pleasure. Thanks.