How to split models across multiple files

[michaeltoohig]

How to avoid circular imports when splitting models into multiple files? I didn't see in docs how to use a string representation of a model for a ForeignKey or a type annotation. I did see FutureRef but that still requires importing the FutureRef from the other file and creates the same circular import. This would be possible with SQLAlchemy or Marshmallow libraries for an example. I would like to avoid a models.py file from becoming a huge behemoth if possible.

[igormorgado]

As you mentioned have you tried FutureRef? IT solves those loopholes.

[michaeltoohig]

I did but maybe I'm doing something wrong? Anyways here's the docs from SQLModel demonstrating the functionality I am looking for in Ormar just so we are on the same page and the type of example I'd like to see in the docs for Ormar as well since all docs about ForwardRef show an example using a single file. https://sqlmodel.tiangolo.com/tutorial/code-structure/#make-circular-imports-work

Below is a simplified example of the models I'm working with. I have a models module with user.py, image.py, widget.py, acme.py inside and of course __init__.py. Images and Acmes are tied to Users and Widgets by foreign keys. I've tried defining the forward references in __init__.py in the same file as the model too. I get a variety of errors such as

If I declare the UserRef in the models/user.py file then try to import the UserRef to use it in acme.py I get the following

ImportError: cannot import name 'UserRef' from partially initialized module 'app.models' (most likely due to a circular import)

But if I declare the UserRef in the models/acme.py file then I get the following

NameError: name 'User' is not defined

Lastly, I'm not sure if the problem is where should I be updating forward refs in my given example? I have to wait until all the models are declared first so I tried updating forward refs in __init__.py where I import the models so I can call model.User or model.Widget easily in my codebase but that doesn't work either, I get the same NameError as before.

user.py

from fastapi_users.db import OrmarBaseUserModel
from app.db.base_class import BaseMeta

class User(OrmarBaseUserModel):
    class Meta(BaseMeta):
        tablename = "user"

image.py

from typing import Union, Dict, Any
import uuid
import ormar
from app.db.base_class import BaseMeta

class Image(ormar.Model):
    class Meta(BaseMeta):
        tablename = "image"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4, uuid_format="string")
    file_id: str = ormar.Text(unique=True, nullable=False)
    filename: str = ormar.Text()
    user: Union[User, Dict[str, Any]] = ormar.ForeignKey(User, nullable=False)
    widget: Union[Widget, Dict[str, Any]] = ormar.ForeignKey(Widget, nullable=False)

widget.py

import uuid
import ormar
from app.db.base_class import BaseMeta

class Widget(ormar.Model):
    class Meta(BaseMeta):
        tablename = "widget"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4, uuid_format="string")
    name: str = ormar.Text(nullable=False)

acme.py

from typing import Union, Dict, Any
import uuid
import ormar
from app.db.base_class import BaseMeta

class Acme(ormar.Model):
    class Meta(BaseMeta):
        tablename = "acme"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4, uuid_format="string")
    user: Union[User, Dict[str, Any]] = ormar.ForeignKey(User, nullable=False)
    widget: Union[Widget, Dict[str, Any]] = ormar.ForeignKey(Widget, nullable=False)

The above is the example only and I would replace instances of User with UserRef etc.

[maitham]

Did you manage to find a solution?

[collerek]

Note that update_forward_refs accepts not only globals, but also locals dictionary that you can pass.

So with dummy structure like this:

So you can do something like this:

user.py

from fastapi_users.db import OrmarBaseUserModel
from app.db.base_class import BaseMeta

class User(OrmarBaseUserModel):
    class Meta(BaseMeta):
        tablename = "user"

image.py

import uuid
from typing import Any, Dict, Union

import ormar
from pydantic.typing import ForwardRef

from base_class import BaseMeta
from widget import Widget

UserRef = ForwardRef("User")  # forward ref


class Image(ormar.Model):
    class Meta(BaseMeta):
        tablename = "image"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4,
                               uuid_format="string")
    file_id: str = ormar.Text(unique=True, nullable=False)
    filename: str = ormar.Text()
    user: Union[UserRef, Dict[str, Any]] = ormar.ForeignKey(UserRef, nullable=False)
    widget: Union[Widget, Dict[str, Any]] = ormar.ForeignKey(Widget, nullable=False)

widget.py

import uuid
import ormar
from base_class import BaseMeta


class Widget(ormar.Model):
    class Meta(BaseMeta):
        tablename = "widget"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4,
                               uuid_format="string")
    name: str = ormar.Text(nullable=False)

acme.py

import uuid
from typing import Any, Dict, Union

import ormar
from pydantic.typing import ForwardRef

from base_class import BaseMeta
from widget import Widget

UserRef = ForwardRef("User")  # forward ref


class Acme(ormar.Model):
    class Meta(BaseMeta):
        tablename = "acme"

    id: uuid.UUID = ormar.UUID(primary_key=True, default=uuid.uuid4,
                               uuid_format="string")
    user: Union[UserRef, Dict[str, Any]] = ormar.ForeignKey(UserRef, nullable=False)
    widget: Union[Widget, Dict[str, Any]] = ormar.ForeignKey(Widget, nullable=False)

base_class.py (for completeness)

import databases
import ormar
import sqlalchemy

DATABASE_URL = "sqlite:///test.db"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()


class BaseMeta(ormar.ModelMeta):
    database = database
    metadata = metadata

__init__.py

import sqlalchemy

from base_class import metadata, DATABASE_URL
from widget import Widget
from user import User
from acme import Acme
from image import Image

locals_ = {
    "User": User  # if you have more refs just add them here - note the key in dict must equal to the name used in ForwardRef
}
Image.update_forward_refs(**locals_)
Acme.update_forward_refs(**locals_)

engine = sqlalchemy.create_engine(DATABASE_URL)
metadata.drop_all(engine)
metadata.create_all(engine)

for table in metadata.sorted_tables:
    print(table.name, table.columns)

this sample will print

user ['user.id', 'user.email', 'user.hashed_password', 'user.is_active', 'user.is_superuser', 'user.is_verified']
widget ['widget.id', 'widget.name']
acme ['acme.id', 'acme.user', 'acme.widget']
image ['image.id', 'image.file_id', 'image.filename', 'image.user', 'image.widget']

Note that the creation of the tables should in reality be somewhere else (you don't want to do this each time your app starts, in production use migrations), example above just illustrates updating the forward references with locals dictionary and uses metadata as a way of proving that it works.

[michaeltoohig]

This solves my issue and thanks for clarifying it.