Unable to use with both Routers and MongoDB

[hamx0r]

Describe the bug

Details in this StackOverflow question. Essentially, fastapi-users relies on a MongoDB client instance, which could be set up globally in one's main.py. However, if I want to split up my app into Router files, each file would need to be able to get this MongoDB Client object (rather than making a new one) to ensure it's all working in the same thread. This would be done using a FastAPI startup trigger. Because such Router endpoints rely on fastapi-users for auth, then I need to somehow instantiate that in main.py too, which doesn't yet have access to a DB client object since it won't be made until the app has started up.

To Reproduce

Steps to reproduce the behavior:

1. Make app with Routers
2. Ensure each router relies on fastapi-users
3. Use startup trigger to make MongoDB client
4. Observe chaos
5. Scroll down to '....'
6. See error

Expected behavior

Allow FastAPIUsers() to take a DB-contention-generating function rather than a DB connection itself (ie a sort of promise)

Configuration

• Python version : 3.9
• FastAPI version : 0.63.0
• FastAPI Users version : 5.0.0

[frankie567]

Well, what you could do is just to have separate modules for DB initialization on one side, and FastAPI Users on the other side, and import them whenever you need it. I've created a working Repl.it example to show you:

https://repl.it/@frankie567/fastapi-users-modular-mongodb

[hamx0r]

I had tried a solution like yours and ran into error messages like this:

  File "/lib/python3.8/site-packages/fastapi_users/authentication/__init__.py", line 180, in _authenticate
    user = await backend(token, self.user_db)
  File "/lib/python3.8/site-packages/fastapi_users/authentication/cookie.py", line 87, in __call__
    return await user_db.get(user_uiid)
  File "/lib/python3.8/site-packages/fastapi_users/db/mongodb.py", line 45, in get
    user = await self.collection.find_one({"id": id})
RuntimeError: Task <Task pending name='Task-3' coro=<RequestResponseCycle.run_asgi() running at /lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py:396> cb=[set.discard()] created at /lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py:277> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at /lib/python3.8/asyncio/futures.py:360] created at /lib/python3.8/asyncio/base_events.py:422> attached to a different loop

However, your repl.it example worked. I'm curious why my less-trivial app throws the above "...attached to different loop" error. I'd normally solve this with (in db.py):

loop = asyncio.get_event_loop()
client = motor.motor_asyncio.AsyncIOMotorClient(
    DATABASE_URL, uuidRepresentation="standard", io_loop=loop
)

But that had no impact (Same error). I can try to make a minimal example to recreated the problem, but at a glance, can you think of why my app would fail with this error when it's using the same design as your example here? (ie, db (Mongo client) is created only once in db.py, and db is then imported into app.py, users.py and posts.py). )

[hamx0r]

I found out what was "wrong", though I still don't understand how the secret sauce works: In your example, you have a separate main.py which only spins up uvicorn to serve app:app. You then have your actual app code in app.py. In my application, my main.py also contained the FastAPI object (app), and at the bottom i had the same kind of:

if __name__ == "__main__":
    import uvicorn
    uvicorn.run("main:app", host="localhost", port=8000, log_level="debug", reload=True)

Splitting apart my code into a separate main.py and app.py allows me to use FastAPI-Users and other routers at the same time, using a global MongoDB motor client without the "attached to different loop" error. Success!

Thanks for your help.

[frankie567]

Curious!

I experienced the "attached to a different loop" error myself, but it was more linked to pytest. In this case, the solution is to define the event_loop fixture yourself to ensure it uses the same loop everywhere:

https://github.com/frankie567/fastapi-users/blob/393b0054cd705b4fe7abc2e89a360d36212ddeeb/tests/conftest.py#L50-L54

I'm surprised however that splitting the uvicorn initializer in its own file solves the issue. Actually, I did that so that it can run easily in Repl.it. Usually, I don't script the command in a file, I just call uvicorn directly in the command line:

uvicorn main:app --port 8000 --reload

If you have time, I would be curious if you try to do it like this and see if you still experience the issue. Event loop management is quite confusing sometimes in Python.

[hamx0r]

Interesting fix for pytest - thanks!

As to running with uvicorn directly (or with gunicorn as I do in production using tiangolo/uvicorn-gunicorn-fastapi:python3.8), I found that I had to hack things a bit. My main.py now looks like this to accommodate local development (where I use python to run main.py as a script) and deployed environments where it runs in Docker (using main.py as a module which includes an app object).

# main.py
if __name__ == "__main__":
    # This gets called when running in development (or rather, not in a docker container
    import uvicorn
    # uvicorn.run("main:app", host="localhost", port=8000, log_level="debug", reload=True, workers=4)
    uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)
else:
    # this gets used in docker container.  Base image expects /app/main.py:app
    # https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker#how-to-use
    from app import app

By adding the else block, it makes your uvicorn main:app --port 8000 --reload work.