Issue at using Beanie adapter for Fastapi-users for handling the entire persistent layer

[jolmedo]

Having a well described model and schema using Pydantic and Beanie syntax, there are some collections, represented by their Document classes, which are not been initialized by init_beanie function at the startup event of a Fastapi app.

Did someone know what could be the causes of such a behavior?

Fastapi-users set a very special class named User, which is one of the well initialized by the background Beanie engine. After that, I added my entire model which consists in several classes.

For example, from my product_category module:

from typing import Optional    
from beanie import Document, Indexed    

class ProductCategory(Document):
    category: Indexed(str, unique=True)
    description: Optional[str]

    class Settings:
        name = "product_categories"

From my product_subcategory module:

from typing import Optional    
from beanie import Document, Link    
from product_category import ProductCategory
    
class ProductSubcategory(Document):
    category_id: Link[ProductCategory]
    subcategory: str
    description: Optional[str]

    class Settings:
        name = "product_subcategories"

...and so. The outcome of init_beanie reflects an initialization of a collection named ProductCategory, not product_categories as I think it would happened, because of the Settings inner class with its property "name", and that's it. After several attempts, I really don't know how init_beanie creates proper collections, but it still gives me the same mistake:
beanie.exceptions.CollectionWasNotInitialized

Such a behavior is not documented, and that's why I assume I'm making something wrong. Can anyone know how to fix this?
If I remove all my Document classes, everything works fine. I'm trying to use the connection Fastapi_users set through Beanie to the MongoDB instance for my own purposes. Is it possible? Is there something "behind the scene" that prevents me from doing it?

Thanks in advance. I would appreciate any help you can provide.

Best regards,
Jorge

[frankie567]

Hi @jolmedo 👋

In a typical setup, you should have the following:

import motor.motor_asyncio
from beanie import PydanticObjectId
from fastapi_users.db import BeanieBaseUser, BeanieUserDatabase

DATABASE_URL = "mongodb://localhost:27017"
client = motor.motor_asyncio.AsyncIOMotorClient(
    DATABASE_URL, uuidRepresentation="standard"
)
db = client["database_name"]

This is the database client Beanie will use. Then, in the startup handler of FastAPI, you should have this:

from beanie import init_beanie


@app.on_event("startup")
async def on_startup():
    await init_beanie(
        database=db,  # This is the `db` variable from the snippet above
        # Declare here all your models so they are properly initialized
        document_models=[
            User,
            ProductCategory,
            ProductSubcategory,
        ],
    )

Is it what you have in your project?

[jolmedo]

That's exactly what I have in my project. I assume I did something wrong when Idexlsred my document model, but it seems well written. Do you need more of my code? Don't have any issues with it, it's a personal project.

Thanks for your answer. Any further help you can provide will be appreciated.

Best regards.

[jolmedo]

Hi @frankie567
A question, if I may, or some questions, to be honest:
I'm willing to assume you wrote this in your previous answer:

await init_beanie(
        database=db,  # This is the db variable from the snippet above
        # Declare here all your models so they are properly initialized
        document_models=[
            User,
            ProductCategory,
            ProductSubcategory,
        ],
    )

... because you read earlier in the code fragment I've posted that the ProductSubcategory class has a field named category_id, which is linked to ProductCategory, am I right?. So, I'm guessing the order I wrote the Document-based classes in the document_models array is relevant, am I right?

While a deeper debugging session through Beanie, I found where the exception beanie.exceptions.CollectionWasNotInitialized is thrown:

    @classmethod
    def get_settings(cls) -> DocumentSettings:

        #Get document settings, which was created on the initialization step
        #:return: DocumentSettings class

        if cls._document_settings is None:
            raise CollectionWasNotInitialized
        return cls._document_settings

So, according to this, there's a protected member named _document_settings which is None and it should be filled during the initialization step, meaning it should happen during the init_beanie call, am I right? That fragment code is at

... venv/lib/python3.9/site-packages/beanie/odm/documents.py

, started at line 871. My virtual environment Fastapi_users' deployment contains this:

beanie.__version__
'1.11.12'
fastapi.__version__
'0.85.0'
fastapi_users.__version__
'10.1.5'

At this point, I have to assume one of these things:

• my Document-based classes have something wrong, maybe some properties which are bad configured, something I've missed or
• there's something wrong inside my Fastapi_users deployment, particularly the Beanie adapter for Fastapi_users

My guess? the first one. Issue here is I can't find what I'm doing wrong, or where. Any help, any clarification you can provide will be appreciated.

Thanks in advance and best regards.
@jolmedo

[frankie567]

At which moment in your app this exception is raised? Directly when you launch the app? Or after when you call an API route?

The guess I have right now is that you maybe try to make read/write queries before init_beanie is called. This may happen if you have a module where you "hard-code" your product categories.

[jolmedo]

At which moment in your app this exception is raised? Directly when you launch the app? Or after when you call an API route?

The guess I have right now is that you maybe try to make read/write queries before init_beanie is called. This may happen if you have a module where you "hard-code" your product categories.

Exception is raised after when an API route is called. 'init_beanie' is called asynchronously, so, how can I be sure it finished whatever I must do before calling an API route? The code is really straight forward, I mean, easy to understand and follow... I also know Beanie uses Motor, which is asynchronous by definition. Any suggestions, please? I'll check again my code, see if there's something odd with that. I've read somewhere Beanie can delay more than a bit to complete it's task at initialization step, maybe that's the problem and hence, my issue. Thanks, @frankie567 , I'll keep you posted.

[frankie567]

If we await the initializer, we can be sure it's completed before FastAPI starts to accept request. Either way, it should take a few milliseconds at most.

Anyway, maybe it's worth to ask on Beanie's discussions: https://github.com/roman-right/beanie/discussions

[jolmedo]

Thanks for the advice, @frankie567. I just open a discussion there, let's see what those guys have to say about this. Keep you posted...

[jolmedo]

Hi, @frankie567. Problem solved, my friend. Forget about the files I've already sent you. The key for solving this was given by @roman-right:
..."a one-file small example..."

He asked me for a one-file small project for debugging purposes, meaning he wants to debug whatmy code. I did it myself, and after fixing some module imports, everything works fine. Then, I got back to my project and conceptually, the biggest difference was the import clause: all relative paths were changed to absolute, meaning that instead of

from .models.product_category import ProductCategory

I did

from dlivery.api.models.product_category import ProductCategory

And everything works fine. Maybe, Beanie can't follow relative paths, maybe there's a Pythonic-way to do this (to prefix some path in the relative ones, so Python inspection procedures can follow them, I really don't know), but simply putting the whole path was enough.
So, thank you very much for your attention and time. Best regards and I can promise you to keep my efforts with Beanie and Fastapi_users framework: it's really cool

[frankie567]

Ha, I was sure @roman-right would find the solution 🙂

As a rule of thumb, I tend to always use absolute imports in Python; I find it clearer and less error-prone.

Cheers!

[jolmedo]

that's an even better advice, @frankie567. I'll take, like you mentioned, as a rule of thumb from now on.