How to manage relationships in (pydantic) models

[ebreton]

EDIT: add proper greetings 🙄

Hi guys,

Many thanks for this fantastic repo. It rocks. Not to mention https://dockerswarm.rocks/, I am now in the process to review all my projects from this base 🥇

Here is my question :

How can I declare a one-to-many relationship in the pydantic models?

Context

I have a first object 'Rule', that is attached to a second 'City'.

I have tried the following without success :

rules.py

from app.models.cities import City

class RuleBase(BaseModel):
    mode: Optional[RuleMode] = None
    value: Optional[float] = None
    city: Optional[City] = None

cities.py

class City(BaseModel):
    id: int
    name: str
    rules: Optional[List['Rule']]

Error in tests:

ImportError while loading conftest '/app/app/tests/conftest.py'.
app/app/tests/conftest.py:4: in <module>
    from app.models.cities import City, Rectangle, Point
app/app/models/cities.py:50: in <module>
    class City(BaseModel):
usr/local/lib/python3.6/site-packages/pydantic/main.py:179: in __new__
    config=config,
usr/local/lib/python3.6/site-packages/pydantic/fields.py:118: in infer
    schema=schema,
usr/local/lib/python3.6/site-packages/pydantic/fields.py:87: in __init__
    self.prepare()
usr/local/lib/python3.6/site-packages/pydantic/fields.py:152: in prepare
    self._populate_sub_fields()
usr/local/lib/python3.6/site-packages/pydantic/fields.py:177: in _populate_sub_fields
    self.sub_fields = [self._create_sub_type(t, f'{self.name}_{display_as_type(t)}') for t in types_]
usr/local/lib/python3.6/site-packages/pydantic/fields.py:177: in <listcomp>
    self.sub_fields = [self._create_sub_type(t, f'{self.name}_{display_as_type(t)}') for t in types_]
usr/local/lib/python3.6/site-packages/pydantic/fields.py:210: in _create_sub_type
    model_config=self.model_config,
usr/local/lib/python3.6/site-packages/pydantic/fields.py:87: in __init__
    self.prepare()
usr/local/lib/python3.6/site-packages/pydantic/fields.py:153: in prepare
    self._populate_validators()
usr/local/lib/python3.6/site-packages/pydantic/fields.py:228: in _populate_validators
    else find_validators(self.type_, self.model_config.arbitrary_types_allowed)
usr/local/lib/python3.6/site-packages/pydantic/validators.py:326: in find_validators
    raise RuntimeError(f'error checking inheritance of {type_!r} (type: {display_as_type(type_)})') from e
E   RuntimeError: error checking inheritance of _ForwardRef('Rule') (type: _ForwardRef('Rule'))

Cheers,
Emmanuel

[euri10]

It won't answer your question exactly but I feel it's the same pattern where you have recursivity and declare an attribute of a model while that depends on another not yet created

my model looks like this :

<LoggerModel name='root' level=20 children=[<LoggerModel name='__main__' level=10 children=[]>, <LoggerModel name='__mp_ma…>

from __future__ import annotations

from typing import ForwardRef, List, Optional

from pydantic import BaseModel

ListLoggerModel = ForwardRef("List[LoggerModel]")

class LoggerModel(BaseModel):
    name: str
    level: Optional[int]
    children: ListLoggerModel = None


LoggerModel.update_forward_refs()

[ebreton]

Thanks for the hint!

I got an ImportError with python 3.6.8

>>> from typing import ForwardRef
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'ForwardRef'

Therefore I installed python 3.7.3 and that solved the issue.

Probably something to report to https://github.com/tiangolo/full-stack-fastapi-postgresql, from which I started from :)

[euri10]

Forwardref is a python 3 thing indeed, maybe there s a way to achieve the same without, @tiangolo will know for sure ! Le ven. 12 avr. 2019 à 7:59 PM, Manu <notifications@github.com> a écrit :

…

Thanks for the hint! I got an ImportError with python 3.6.8 >>> from typing import ForwardRef Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'ForwardRef' Therefore I installed python 3.7.3 and that solved the issue. Probably something to report to https://github.com/tiangolo/full-stack-fastapi-postgresql, from which I started from :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#153 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDZPnowf6UmatNDVrS7YpgPPMgkXMCnks5vgMlkgaJpZM4csUX5> .

[tiangolo]

Many thanks for this fantastic repo. It rocks. Not to mention https://dockerswarm.rocks/, I am now in the process to review all my projects from this base 🥇

That's cool! Thanks!

As @euri10 says, for that case, ForwardRef is the way to go.

The caveat is that ForwardRef is Python 3.7 only. And Python 3.7 is not compatible yet with tools like Celery and TensorFlow :/

[ebreton]

Hi @tiangolo ,

Thanks for the confirmation 👍

Fair enough for tensorflow. Celery seems to have catch up with python 3.7 and I have made a PR on the postgresql project Generator tiangolo/full-stack-fastapi-template#10 to test 3.7

Could you have a look at it ? Would it effectively allow the ForwardRef and ease the definition of relationships on a pydantic levels, that would be a great added value for a postgresql project 😇

Cheers,
Emmanuel

[tiangolo]

Awesome! I'll check it.

[tiangolo]

Just merged your PR. Now Celery and TensorFlow both support Python 3.7! 🎉

[fbidu]

Hi, I'm sorry for resurrecting an old issue but I just ran into @ebreton's problem and I'd like to check if this is still the best practice. Considering an example similar to the docs:

item.py:

class Item(ItemBase):
    id: int
    owner: User    # <------ Relationship with User

    class Config:
        orm_mode = True

user.py:

class User(UserBase):
    id: int
    is_active: bool
    items: List[Item] = [] # <------ Relationship with Item

    class Config:
        orm_mode = True

I'd have to import both item from user.py and user from item.py, which would cause a circular import. So I should do something like

ListItem = ForwardRef("List[Item]")
class User(UserBase):
    id: int
    is_active: bool
    items: ListItem = [] # <------ Relationship with Item

    class Config:
        orm_mode = True

User.update_forward_refs()

As my FastAPI application grows, I'd like to break the CRUD into a submodule and the schemas as well. Having back populated relationships like those make things difficult

---

Update: actually, I couldn't make it work...
Update 2: I think my problem is actually this one pydantic/pydantic#659

[rizkyarlin]

My problem is the same as @fbidu .

Or @tiangolo are you against splitting schema into multiple files?

[villqrd]

Same issue here. Can't manage to have 2 schemas referencing each others. What is the correct way to achieve this?

[szymonpru]

I have the same problem. Has anyone found a solution? @tiangolo

[Mause]

There is no correct way - two schemas referencing each other will cause an infinite loop, which you must break

[acidjunk]

What would be the best practice in this situation?

[jhrr]

To avoid the circular import problem you can use TYPE_CHECKING and postponed evaluation of annotations.

[ricardo-reis-1970]

This issue is ongoing and should not be closed until someone came up with an actual suggestion that runs. Ideally, this person should run their code and then post it here. @jhrr's previous comment is just dismissive and time-wasting.

I am having the precise same issue. My case deals with many-to-many relationships between the entities User, Role and Permission, which makes me dare say I might not be the only person dealing/struggling with this.

Saving the pure names to the SQLAlchemy models, I'm suffixing them with Schema in the Pydantic schemas. From both UserSchema and PermissionSchema I'm importing the RoleSchema directly and declaring something like this:

# schemas/user.py
class UserSchema(UserBase, CreatedModel):
    roles: List[RoleSchema]

    class Config:
        orm_mode = True

CreatedModel is just adding an id and created_date to all the models.

In the RoleSchema case, my attempt was:

# schemas/role.py
class RoleSchema(RoleBase, CreatedModel):
    """Schema validation for Role"""
    permissions: List[ForwardRef('PermissionSchema')]
    users: List[ForwardRef('UserSchema')]

    class Config:
        orm_mode = True

with ForwardRef, as importing either PermissionSchema or UserSchema would result in a circular import. This is the only schema that needs to use ForwardRef.

However, whenever and wherever I try to do RoleSchema.update_forward_refs(), I get the error:

NameError: name 'PermissionSchema' is not defined

Is there a solution/best practice/recommendation for this, or is @rizkyarlin correct to assume that this is all too small for real life scenarios where entities should be kept in separate files? I'm asking for a solution that looks like code, not a PEP link, please.

[jhrr]

@ricardo-reis-1970 I actually had no intention of being 'dismissive and time-wasting' when I left this comment. Even if you have a substantive criticism, please control your tone.

[ricardo-reis-1970]

@jhrr, tone is something that you read. If this sounds any milder, I'm actually open to understanding what your intentions were with the PEP link and how did you think that would help. This would amount to a better grade of 'substantive criticism', thus better consubstanciating your complaint, if I were to follow your reasoning there and see how precisely code would look like.

You see, when people are banging their heads trying to follow through sparse documentation across FastAPI, Pydantic, Starlette, SQLAlchemy and whatever else, they might be asking for code examples or templates, not yet another long document yielding little reward.

If you wish to take this offline, I'd be delighted to follow up and let you know where this opinion of mine stems from.

[ruancomelli]

Just for the record, there are similar issues in Pydantic: pydantic/pydantic#707 and pydantic/pydantic#1873.

[curtwagner1984]

@ricardo-reis-1970 I'm having the same issue. Did you find a solution or a workaround? When I'm not adding ForwarRef I get a circular import error, and when I do and then run update_forward_refs() I get NameError: name '[FORWARD REFED CLASS NAME]' is not defined

Like you, I too don't understand how @jhrr's comment

To avoid the circular import problem you can use TYPE_CHECKING and postponed evaluation of annotations.

Helps to solve the issue. As TYPE_CHECKING only helps with IDE hints and if you import something under if TYPE_CHECKING it isn't actually imported at runtime. The link to postponed evaluation of annotations also isn't helpful as I couldn't figure out how to actually import class B from class A and class A from B without triggering either a Forward Ref or circular import error

For example:

I have an Image class, like so:

class ImageSchema(IntIdSchemaMixin, UUIDSchemaMixin, CreateUpdateDateSchemaMixin, ThumbnailSchemaMixin,
                  MetricsSchemaMixin, MediaOnDiskSchemaMixin):
    """Represents an image file on disk"""

All those mixins are just regular fields for example id: int, uuid: str etc

Each image can be in a folder:

#folder.py
ImgSchema = ForwardRef("ImageSchema") #<- ForwarRef defined


class FolderSchema(IntIdSchemaMixin, UUIDSchemaMixin, CreateUpdateDateSchemaMixin, ThumbnailSchemaMixin):
    """Represents a folder on disk"""

    folder_path: Path
    """Path to the folder this class represents"""

    parent: Optional[FolderSchema] = None
    """Parent of this folder"""

    sub_folders: Optional[list[FolderSchema]] = None
    """SubFolders of this folder"""

    images: list[ImgSchema] #<- FowrawrdRef used.

And I have this simple test:

def test_folder_schema_initialization():
    from src.backend.database.pydantic_schemas.image_schema import ImageSchema
    from src.backend.database.pydantic_schemas.folder_schema import FolderSchema

    FolderSchema.update_forward_refs()

    image_schema = ImageSchema(name="somename", path_to_file=Path("mypath/to/file"),
                               path_to_dir=Path("another/file/path"), file_suffix=".mp4",
                               date_added=datetime.datetime.now(), uuid="string", id=4)

    folder_schema = FolderSchema(id=1,
                                 uuid="some_uuid",
                                 date_added=datetime.datetime.now(),
                                 folder_path=Path("some/folder/path"),
                                 parent=None, sub_folders=None, images=[image_schema])

    assert folder_schema.uuid == "some_uuid"

It fails with NameError: name 'ImageSchema' is not defined. If I remove FolderSchema.update_forward_refs() then it fails with pydantic.errors.ConfigError: field "parent" not yet prepared so type is still a ForwardRef, you might need to call FolderSchema.update_forward_refs().

If I replace the ForwardRef images: list[ImgSchema] to the string declaration of the actual class like this images: list["ImageSchema"] I get RuntimeError: error checking inheritance of 'ImageSchema' (type: str)

It's worth pointing out that the Folder class has self-reference, and running FolderSchema.update_forward_refs() does resolve the self-reference issues in a similar manner. But when I'm trying to add a forward ref to another class it crushes.

[NidalGhonaim]

This is what @jhrr is talking about -> https://www.youtube.com/watch?v=B5cjckVzY4g (doesn't work in runtime)

But in this case it seems to me like a true import cycle -> https://www.youtube.com/watch?v=UnKa_t-M_kM&t=151s

I just used children: list = [] instead of children: List[Child] = [] to avoid needing to import anything in the first place.

[accelleon]

Soooooo this is how I typically handle this scenario. This solution lets me keep my neat type hinting and all that jazz. This works in python3.9:

user.py

from typing import TYPE_CHECKING, List

from pydantic import BaseModel

if TYPE_CHECKING:
  from .item import Item


class User(BaseModel):
  name: str
  items: 'List[Item]'

item.py

from typing import List

from pydantic import BaseModel

from .user import User


class Item(BaseModel):
  name: str
  user: User

# This is the important bit, we explicitly bind the forward ref 'Item'
User.update_forward_refs(Item=Item)

However, this leads to recursion that breaks pydantic. Pydantic has no method to resolve the cyclic parsing that this leads to (i.e. I want to pull an Item from my orm, Pydantic parses the parent User, which parses the child Items which all parse the parent User again and so on....) SQLAlchemy is perfectly equipped to handle it, just Pydantic not so much. So your only 2 ways to deal with it is break the relationship on one of the models or define a child model that doesn't end up in an infinite loop of parsing its children:

# user.py stays the same

class Item(BaseModel):
  name: str
  user_id: int

class ItemDB(Item):
  user: User

So in that case User will contain a list of Items that doesn't attempt to recursively parse its parent User then itself etc... Done this way your max recursion depth ends up being 3. ItemDB still contains this relationship, which will return its parent User, and that parent User will have all its child Items and those will have their parent User. You could go full chooch and define 2 models of each, this'll reduce your recursion depth to 2, which is probably the deepest you'd ever want to go but is 1 more model to ensure you're returning in the right places.