Nested Pydantic models triggering all kind of errors? [Help Needed]

[KayJay89]

Hi all

I am sorry to disregard the template but it is quite daunting for someone who doesn't know exactly what is what. Secondly I would like to express my gratitude for this package as it has helped me many times in the past. However I have been stuck trying to find a solution for my problem for some days now.

I have looked across the internet and came across many topics but I still just do not understand a lot of what is being said (mainly due to my ignorance of both pydantic and fastapi). I believe though my problem is basically coming down to having nested pydantic schemas:

schemas.py

from datetime import datetime
from typing import Optional, List
from pydantic import BaseModel


class CountryBase(BaseModel):
    name: str
    iso_alpha_2_code: str
    iso_alpha_3_code: str
    numeric_code: int


class CountryCreate(CountryBase):
    pass


class Country(CountryBase):
    id: int
    companies: List[Company] = []

    class Config:
        orm_mode = True


class SectorBase(BaseModel):
    name: str


class SectorCreate(SectorBase):
    pass


class Sector(SectorBase):
    id: int
    companies: List[Company] = []

    class Config:
        orm_mode = True


class IndustryBase(BaseModel):
    name: str


class IndustryCreate(IndustryBase):
    pass


class Industry(IndustryBase):
    id: int
    companies: List[Company] = []

    class Config:
        orm_mode = True


class FormBase(BaseModel):
    name: str
    description: Optional[str] = None
    sec_number: Optional[str] = None
    link: Optional[str] = None


class FormCreate(FormBase):
    pass


class Form(FormBase):
    id: int
    filings: List[Filing] = []

    class Config:
        orm_mode = True


class CompanyBase(BaseModel):
    ticker: str
    name: Optional[str] = None
    ipo_year: Optional[datetime] = None


class CompanyCreate(CompanyBase):
    pass


class Company(CompanyBase):
    id: int
    ipo_year: datetime
    updated_on: datetime
    country: Country = None
    sector: Sector = None
    industry: Industry = None
    filings: List[Filing] = []

    class Config:
        orm_mode = True


class FilingBase(BaseModel):
    cik: int
    published_on: datetime
    link: str
    filed_on: datetime
    accepted_on: datetime
    edgar_filenumber: str
    edgar_accessionnumber: str


class FilingCreate(FilingBase):
    pass


class Filing(FilingBase):
    id: int
    form: Form
    company: Company

    class Config:
        orm_mode = True

main.py

# all imports depends...etc

app = FastAPI(dependencies=[Depends(verify_key)])

# Dependency
def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

@app.get("/ping", tags=["Test"])
async def test_connection():
    return {"pong"}

@app.get("/companies", response_model=List[schemas.Company])
async def get_all_companies(
    skip: int = 0, limit: int = 10, db: Session = Depends(get_db)
):
    companies = crud.get_companies(db, skip=skip, limit=limit)
    return companies

When I use the above and just launch fastapi (uvicorn main:app --reload) this triggers NameErrors like so:

File "./schemas.py", line 17, in <module>
    class Country(CountryBase):
  File "./schemas.py", line 19, in Country
    companies: List[Company] = []
NameError: name 'Company' is not defined

If I understand it correctly this is because I am obviously referencing an object which isn't created yet. After digging I found a reference to annotations PEP 563. I added from __future__ import annotations to the top of my schemas.py and no errors trigger. So I thought it was a simple fix.

When I go to my ping route, everything works as expected. When I go to my companies or to the docs, I get an internal server error:

pydantic.errors.ConfigError: field "companies" not yet prepared so type is still a ForwardRef, you might need to call Country.update_forward_refs().

When adding Country.update_forward_refs() below my Country class in schemas.py, I get another error:

File "./schemas.py", line 27, in <module>
    Country.update_forward_refs()
  File "pydantic/main.py", line 826, in pydantic.main.BaseModel.update_forward_refs
  File "pydantic/typing.py", line 377, in pydantic.typing.update_field_forward_refs
  File "pydantic/typing.py", line 62, in pydantic.typing.evaluate_forwardref
  File "/usr/lib/python3.8/typing.py", line 518, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'Company' is not defined

When adding Country.update_forward_refs() below my Country class in schemas.py, I get another error:

# many many times the same as below (given recursion error)
  File "pydantic/main.py", line 731, in pydantic.main.BaseModel.validate
  File "pydantic/main.py", line 623, in pydantic.main.BaseModel.from_orm
  File "pydantic/main.py", line 1034, in pydantic.main.validate_model
  File "pydantic/fields.py", line 723, in pydantic.fields.ModelField.validate
  File "pydantic/fields.py", line 906, in pydantic.fields.ModelField._validate_singleton
  File "pydantic/fields.py", line 913, in pydantic.fields.ModelField._apply_validators
  File "pydantic/class_validators.py", line 310, in pydantic.class_validators._generic_validator_basic.lambda12
  File "pydantic/validators.py", line 60, in pydantic.validators.str_validator
RecursionError: maximum recursion depth exceeded while calling a Python object

I read #2315 too but can't seem to understand how this is a solution and apply it to my own issue. And this is basically where I am completely lost.

If I do not add Country.update_forward_refs() and only keep the import statement from __future__ import annotations and go to docs, I get other errors too:

/home/kayjay/Desktop/Projects/my-repo/env/lib/python3.8/site-packages/fastapi/utils.py", line 24, in get_model_definitions
    m_schema, m_definitions, m_nested_models = model_process_schema(
  File "pydantic/schema.py", line 548, in pydantic.schema.model_process_schema
  File "pydantic/schema.py", line 589, in pydantic.schema.model_type_schema
  File "pydantic/schema.py", line 241, in pydantic.schema.field_schema
  File "pydantic/schema.py", line 495, in pydantic.schema.field_type_schema
  File "pydantic/schema.py", line 838, in pydantic.schema.field_singleton_schema
  File "/usr/lib/python3.8/abc.py", line 102, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

And just some more info from running python -c "import pydantic.utils; print(pydantic.utils.version_info())"

pydantic version: 1.8.1
            pydantic compiled: True
                 install path: /home/kayjay/Desktop/Projects/my-repo/env/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Jan 27 2021, 15:41:15)  [GCC 9.3.0]
                     platform: Linux-5.8.0-45-generic-x86_64-with-glibc2.29
     optional deps. installed: ['dotenv', 'typing-extensions']

I have been looking for solutions for days but I am probably not smart enough to figure it out or find the solution online so I am begging for your help. I am convinced it is not a bug but just my own plain ignorance but I an unable to resolve it myself.

I think you can clearly see I am really stuck and would hope someone could either nudge me in the right direction or help me move forward.

[hay-kot]

When adding Country.update_forward_refs() below my Country class in schemas.py, I get another error:

The problem is that you are nesting your models and referencing them in a recursive loop. When you grab a Company model it tries to populate company.country which then tries to populate country.companies which then tries to populate company.country and so on.

I think there are two approaches to solve this problem.

1. Create a new county model that excludes the companies property
2. Use the getter_dict method from pydanitc to pull specific values from the orm. For example...

class Company(CompanyBase):
    id: int
    ipo_year: datetime
    updated_on: datetime
    country: int = None
    sector: Sector = None
    industry: Industry = None
    filings: List[Filing] = []

    class Config:
        orm_mode = True

        @classmethod
        def getter_dict(_cls, name_orm: SqlAlchemyModel):
            return {
                **GetterDict(name_orm),
                "country":  name_orm.id
            }

Personally, I would probably go with the a new non-recursive model.

[KayJay89]

First of all many thanks for your reply!

I am happy that I understood correctly that the issue was the infinite loop but couldn't find a working solution.

With regards to the first solution, how does that resolve the issue? I mean I understand the loop is gone but I need that data to be returned. I suppose I could run another api call to get the company data per id from my front-end but that hardly seems efficient. And if I remove country, I just end up with the same error for the next field (sector) and if I remove sector, the next and so on until I have absolutely 0 references to any other models.

And as for the second solution, I tried but the id returned is always the id of the object, not the referenced one. And even if this would return the id, it still does not return the info I need which is the entire object.

I apologize for my ignorance in advance but it hardly seems practical to have to apply these "workarounds" for what in my view seems pretty basic DB related things for a normalized DB. Again as I am quite inexperienced at pydantic etc I might obviously be saying really stupid things here.

Edit: I have actually removed all references to other Objects with exception of Company and now it works as intended. I don't know why I did not think about this before. I'd like to thank Hay-Kot for your comment as this is the one that put me on the right track!

[hay-kot]

Glad I could help! If you've found the solution to your original problem, consider closing the issue. Thanks!

[tiangolo]

Thanks for the help here @hay-kot ! 👏 🙇

If that solves the original problem, then you can close this issue @KayJay89 ✔️

Sorry for the long delay! 🙈 I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.

[github-actions]

Assuming the original need was handled, this will be automatically closed now. But feel free to add more comments or create new issues or PRs.