self.call doesn't work in Thread?

[chulman444]

I'm using aiohttp for the server.

Tried googling and got nothing useful. Tried searching for '.call' or 'self.call' on issues and discussions. I'm getting zero result. And honestly, I think this using self.call inside a thread should work anyways.

Not sure if this is a bug or not, so posting on discussion. I've copy-pasted the reproduction code below.

I have 3 cases that I tested. The first case is using self.call, and that's what the reproduction code will run when you just run the server and client. And it just works.

The second case is using Thread and emit with callback which works.

The third case uses self.call in Thread. The client will send the response, but the server just hangs forever. When I try to quit with ctrl+c it prints:

Call request 2022-11-18 10:10:48.580518
Traceback (most recent call last):
  File "D:\Users\dev\test\2022-11-18 python-socketio bug\server.py", line 48, in <module>
    AsyncServer().start()
  File "D:\Users\dev\test\2022-11-18 python-socketio bug\server.py", line 32, in start
    web.run_app(self.app, port=8000)
  File "C:\Python39\lib\site-packages\aiohttp\web.py", line 518, in run_app
    _cancel_tasks({main_task}, loop)
  File "C:\Python39\lib\site-packages\aiohttp\web.py", line 445, in _cancel_tasks
    loop.run_until_complete(asyncio.gather(*to_cancel, return_exceptions=True))
  File "C:\Python39\lib\asyncio\base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "C:\Python39\lib\asyncio\windows_events.py", line 316, in run_forever
    super().run_forever()
  File "C:\Python39\lib\asyncio\base_events.py", line 596, in run_forever
    self._run_once()
  File "C:\Python39\lib\asyncio\base_events.py", line 1854, in _run_once
    event_list = self._selector.select(timeout)
  File "C:\Python39\lib\asyncio\windows_events.py", line 434, in select
    self._poll(timeout)
  File "C:\Python39\lib\asyncio\windows_events.py", line 783, in _poll
    status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
KeyboardInterrupt

but the stack is all about aiohttp and asynctio...

I would love to use self.call. I don't know what I'm doing wrong, so please help.

And as a side note, for case 2 and 3, when the server emits or calls, the client receives the message after 20 seconds of delay in my case. Do you know why this is the case and how to fix this?

---

Reproduction code:

# server.py ==============

# Python libs
import asyncio, datetime, signal
from threading import Thread

# Pip libs
import socketio
from aiohttp import web
from bson import json_util

# Without this, when I ctrl+c in the server, I get `KeyboardInterrupt` with
# `status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)` and it won't exit
signal.signal(signal.SIGINT, signal.SIG_DFL)

class AsyncServer(socketio.AsyncNamespace):
  def __init__(self):
    super().__init__()
    self.app = web.Application()
    self.sio = socketio.AsyncServer()

    self.sid = None
  
  async def on_connect(self, sid, environ):
    print("on connect", sid)
    self.sid = sid
    # CASE 2 -------------------------------
    # t = Thread(target=asyncio.run, args=(self.threadTaskEmit(),))
    # t.start()

    # CASE 3 -------------------------------
    # t = Thread(target=asyncio.run, args=(self.threadTaskCall(),))
    # t.start()
    return
    
  async def on_disconnect(self, sid):
    print("on disconnect", sid)
    return
  
  # REMOVE THIS AFTER CASE 1 -------------------------------
  async def on_connected(self, sid):
    print("On connected", datetime.datetime.now())
    response = await self.call("request", sid=self.sid)
    print("Got response:", type(response))
    print(response)
  
  def start(self):
    self.sio.register_namespace(self)
    self.sio.attach(self.app)
    web.run_app(self.app, port=8000)

  # For case 2
  async def threadTaskEmit(self):
    def cb(response):
      print("Got response:", type(response))
      print(response)
    print("Emit request", datetime.datetime.now())
    await self.emit("request", callback=cb)
  
  # For case 3
  async def threadTaskCall(self):
    print("Call request", datetime.datetime.now())
    response = await self.call("request", sid=self.sid)
    print("Got response:", type(response))
    print(response)

if __name__ == "__main__":
  AsyncServer().start()

The client code:

# Python libs
import asyncio, datetime

# Pip libs
import socketio

# 2022-11-13 12:35
# For readability of `ApiNodeBase`
class AsyncClient(socketio.AsyncClientNamespace):
  def __init__(self):
    super().__init__()
    self.sio = socketio.AsyncClient()
    return

  async def on_connect(self):
    print(f"connected")
    
    # REMOVE THIS AFTER CASE 1 -------------------------------
    await self.emit("connected")

  async def on_disconnect(self):
    print(f"disconnected!")
    await self.disconnect()
  
  def on_connect_error(self, data):
    print("The connection failed!")
  
  def start(self):
    self.sio.register_namespace(self)
    async def run():
      await self.sio.connect('http://localhost:8000')
      await self.sio.wait()
    asyncio.run(run())
    return

  def on_request(self):
    # response = json.dumps({ "foo": "bar" })
    response = { "foo": "bar" }
    print("on request:", datetime.datetime.now())
    print(response)
    return response

# For testing stuff
if __name__ == "__main__":
  AsyncClient().start()

---

Edit. Changing threadTaskCall to use timeout parameter response = await self.call("request", sid=self.sid, timeout=21), made it work.

The default timeout is 60, but it doesn't work when timeout is not given. It WORKS when I set timeeout=60. why does it only work when timeout is specifically given? And how do I reduce this 20 seconds delay?

[miguelgrinberg]

You seem to be missing the fact that asyncio is single threaded. You cannot mix threads with asyncio in this way, things are going to fail in strange ways as this isn't a supported usage.

[chulman444]

Oh... I was able to get it work by doing asyncio.create_task(self.threadTaskCall()) instead of using Thread. And this fixed the 20 seconds delay thing too.

I still don't understand why asyncio and Thread don't work this way. I guess I'll have to look that up.

[miguelgrinberg]

I'm not sure there is a detailed explanation of the why in the Python docs. This is the closest I could find: https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading.

Your solution is correct, a thread in multi-threading should be replaced with a task in asynchronous programming.