Error run fastapi-mqtt using gunicorn

[ruem2802]

I use fastapi-mqtt==0.3.0, python==3.9.7, fastapi==0.68.1
I run example application by command : uvicorn app:app --host 0.0.0.0 --port 7000 --reload
Everything is ok -> app run

But:
I run example application by command :
gunicorn app:app -k uvicorn.workers.UvicornWorker --workers=9 -b 0.0.0.0:7000

Error :
File "/home/hoanganh/smarthome-server/venv/lib/python3.9/site-packages/gmqtt/client.py", line 230, in connect
await self._connected.wait()
File "/usr/lib/python3.9/asyncio/locks.py", line 226, in wait
await fut
RuntimeError: Task <Task pending name='Task-3' coro=<LifespanOn.main() running at /home/hoanganh/smarthome-server/venv/lib/python3.9/site-packages/uvicorn/lifespan/on.py:84>> got Future attached to a different loop

[2021-09-21 20:42:13 +0700] [10419] [ERROR] Application startup failed. Exiting.
[2021-09-21 20:42:13 +0700] [10419] [INFO] Worker exiting (pid: 10419)
Task was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<Client._resend_qos_messages() running at /home/hoanganh/smarthome-server/venv/lib/python3.9/site-packages/gmqtt/client.py:176>>
sys:1: RuntimeWarning: coroutine 'Client._resend_qos_messages' was never awaited
[2021-09-21 20:42:14 +0700] [10418] [INFO] Shutting down: Master
[2021-09-21 20:42:14 +0700] [10418] [INFO] Reason: Worker failed to boot.

[sabuhish]

Hi, I gave a simple solution here can you check this pls and let me know.

#17

[ruem2802]

Hi, I gave a simple solution here can you check this pls and let me know.

#17

I try and error!

[DavidOsparks]

I had the same problem. Simple uvicorn running was working well, but

gunicorn --workers=2 -k uvicorn.workers.UvicornWorker my_api:app

or even use tiangolo/uvicorn-gunicorn-fastapi:python3.8 to run my program got the same error:

[2023-01-04 17:28:07 +0800] [9531] [ERROR] Traceback (most recent call last):
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 540, in lifespan
    async for item in self.lifespan_context(app):
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 481, in default_lifespan
    await self.startup()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 516, in startup
    await handler()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/fastapi_mqtt/fastmqtt.py", line 232, in startup
    await self.connection()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/fastapi_mqtt/fastmqtt.py", line 130, in connection
    await self.client.connect(
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/gmqtt/client.py", line 230, in connect
    await self._connected.wait()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
RuntimeError: Task <Task pending name='Task-3' coro=<LifespanOn.main() running at /opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/uvicorn/lifespan/on.py:86>> got Future <Future pending> attached to a different loop

[2023-01-04 17:28:07 +0800] [9531] [ERROR] Application startup failed. Exiting.
[2023-01-04 17:28:07 +0800] [9531] [INFO] Worker exiting (pid: 9531)
[2023-01-04 17:28:07 +0800] [9530] [ERROR] Traceback (most recent call last):
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 540, in lifespan
    async for item in self.lifespan_context(app):
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 481, in default_lifespan
    await self.startup()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/starlette/routing.py", line 516, in startup
    await handler()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/fastapi_mqtt/fastmqtt.py", line 232, in startup
    await self.connection()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/fastapi_mqtt/fastmqtt.py", line 130, in connection
    await self.client.connect(
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/gmqtt/client.py", line 230, in connect
    await self._connected.wait()
  File "/opt/anaconda3/envs/vpin_venv/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
RuntimeError: Task <Task pending name='Task-3' coro=<LifespanOn.main() running at /opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/uvicorn/lifespan/on.py:86>> got Future <Future pending> attached to a different loop

[2023-01-04 17:28:07 +0800] [9530] [ERROR] Application startup failed. Exiting.
[2023-01-04 17:28:07 +0800] [9530] [INFO] Worker exiting (pid: 9530)
Task was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<Client._resend_qos_messages() running at /opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/gmqtt/client.py:176>>
sys:1: RuntimeWarning: coroutine 'Client._resend_qos_messages' was never awaited
Task was destroyed but it is pending!
task: <Task pending name='Task-1' coro=<Client._resend_qos_messages() running at /opt/anaconda3/envs/vpin_venv/lib/python3.8/site-packages/gmqtt/client.py:176>>
sys:1: RuntimeWarning: coroutine 'Client._resend_qos_messages' was never awaited
[2023-01-04 17:28:07 +0800] [9528] [WARNING] Worker with pid 9530 was terminated due to signal 15
[2023-01-04 17:28:07 +0800] [9528] [INFO] Shutting down: Master
[2023-01-04 17:28:07 +0800] [9528] [INFO] Reason: Worker failed to boot.

[tomliptrot]

Was there any further thoughts on this? The error is mentioned here tiangolo/fastapi#3939.

This response suggest a solution tiangolo/fastapi#3939 (comment)

This is both gmqtt and fastapi-mqtt fault. It's the same issue we have with motor.

Step by step:

The example initializes FastMQTT.
On FastMQTT, they initialize the MQTTClient.
MQTTClient is a class that inherits some other classes on the gmqtt project, and one of those classes is EventCallback.
EventCallback runs asyncio.Event().
asyncio.Event() tries to get the current event loop internally and then the event is created on this event loop.
gunicorn runs and uvicorn creates its own event loop.
MQTTClient runs await self._connected.wait() at some point, and this operation runs on a different loop than the one uvicorn created.
In any case, possible solutions:

Add FastMQTT initialization on a startup event.
Ask for the fastapi-mqtt maintainer to change the place where 2 is done (it should be on their startup event).

[vvanglro]

Hope that solves this problem. 🥳

[azogue]

Closing this one as completed. In recent versions there is support for lifespan async context manager, and the init_app legacy method also works.

The example apps work with both uvicorn and gunicorn ✅