created_cloned_field — slow performance with many models

[zanieb]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import fastapi
import pydantic


class NestedModel(pydantic.BaseModel):
    x: pydantic.BaseModel
    y: pydantic.BaseModel


def create_app():
    for _ in range(100):
        fastapi.routing.APIRoute(
            "/test", endpoint=lambda: ..., response_model=NestedModel
        )


# PROFILING

import yappi

yappi.set_clock_type("CPU")

with yappi.run():
    create_app()

stats = yappi.get_func_stats()
stats.save("fastapi.pprof", type="pstat")

Description

When building a FastAPI application with nested Pydantic models, the create_cloned_field utility in the APIRoute initialization is quite slow.

For the trivial example, you can see that create_cloned_field dominates the runtime with 90% of CPU time. The majority of this is spent deep copying.

Note, timing is CPU time not WALL time

If we replace this trivial application with the one from Prefect, from prefect.orion.api.server import create_app, we can see that this is significant in a real world example.

With a patch to retain the cache across calls to this function, we can get this down to 50% of the CPU time with a ~5x overall speedup.

This speedup persists and is even more significant in a real-world application with create_cloned_field accounting for only 11% of the CPU time.

Operating System

macOS

Operating System Details

No response

FastAPI Version

0.74.0

Python Version

3.8.12

Additional Context

This may also be resolvable with pydantic/pydantic#1008 as mentioned in #894 (comment)

[ddanier]

We are currently having this issue in a bigger FastAPI project. The startup time did increase to a minute till several minutes over the last months - just to get the server running and on every code change while doing development. As a quick test we did just "remove" the create_cloned_field call which did reduce the startup time drastically.

So I would love to get this fixed. ;-)

[ddanier]

Little update: I did use the patch provided in the pull request after all, but just reverted this on my local dev setup as it did produce problems when working with ForwardRef's. Those seem to be not resolved any more, possibly because an "old version" of the model is cached now. Have to dig into this - but the patch will introduce regression bugs for sure....so just you know.

[ddanier]

Update again - this is a real issue:
We "fixed" the ForwardRef issue by calling update_forward_refs before any routes are defined. This way the cache does not mess up things - although I still think that has to be redone in a clean way.

Thing is besides taking for basically ever to load without the patch and loading in about 30 seconds with the patch....also the RAM usage skyrockets....which the patch also fixes.

Patch enabled	RAM usage
No	1165 GiB
Yes	374 MiB

Again just FYI - we will need a good solution for that.

[zanieb]

Thanks for giving this a test @ddanier ! Can you share an minimal complete example that uses forward refs and demonstrates how it breaks your routes? I can investigate adding handling for that case.

[ddanier]

@madkinsz I will do that....sadly I did not have the time yet to really look into this. All details above are bits my colleagues gave me when trying to work on our own performance issue. The project we try to use this is pretty huge and even the ForwardRefs are no just plain normal ForwardRefs (they are part of dynamically generated pydantic model)...so I will need to setup a new project to get a simple example for this. What I can say for sure is that the problem occurred after we did try the patch and thus must be a direct result of this.

Sorry if this takes some time, much to do currently. :)

[teebu]

Any update on this?

[MarkusSintonen]

We are also heavily hit by this issue. But we added a hacky workaround for our tests which speeds up the test initialization hugely! It would be great to get the issue fixed.

The hack is here for anyone hit by the same issue:

FASTAPI_CLONED_TYPES_MEMO = {}

# Workaround for FastAPI initialization slowness causing slow test startup.
# https://github.com/tiangolo/fastapi/issues/4644
@pytest.fixture(scope="session")
def fix_slow_fastapi_issue(monkeypatch: MonkeyPatch) -> None:
    import fastapi.routing
    import fastapi.utils

    orig = fastapi.utils.create_cloned_field

    def patch(
        field: ModelField, *, cloned_types: Optional[Dict[Type[BaseModel], Type[BaseModel]]] = None
    ) -> ModelField:
        return orig(field, cloned_types=FASTAPI_CLONED_TYPES_MEMO if cloned_types is None else cloned_types)

    patcher.setattr(fastapi.utils, "create_cloned_field", patch)
    patcher.setattr(fastapi.routing, "create_cloned_field", patch)


@pytest.fixture(scope="session")
def http_app(fix_slow_fastapi_issue: None) -> FastAPI:
    return create_test_app()  # Does the FastAPI initialization

Profiles via pytest-profiling
Before:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
      818    0.009    0.000   13.524    0.017 routing.py:504(add_api_route)
      820    0.025    0.000   13.514    0.016 routing.py:308(__init__)
20409/778    0.123    0.000    9.486    0.012 utils.py:76(create_cloned_field)
...

After:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
      818    0.007    0.000    3.557    0.004 routing.py:504(add_api_route)
      820    0.023    0.000    3.548    0.004 routing.py:308(__init__)
 2839/778    0.016    0.000    1.086    0.001 utils.py:76(create_cloned_field)
...

cc @ddanier what issue are you seeing with the proposed fix do you have an example to give? We are also running a big FastAPI service but the patch seems to work for us.

[ddanier]

We are currently using a monkey patched version of the normal main.py like this:

# flake8: noqa
"""
This is a faster version of the main.py.

This version is faster by monkey patching the internals of FastAPI. It is
intended to NEVER be used in production. It is only for testing and
local development.

To can activate this file you may just create a `docker-compose.override.yml`
with the following contents:

---
version: "3.6"

services:
  api:
    command:
      [
        "poetry",
        "run",
        "uvicorn",
        "something.fast_main:app",
        "--host=0.0.0.0",
        "--reload",
      ]
---
"""

from dataclasses import is_dataclass
from typing import Optional, cast
from weakref import WeakKeyDictionary

from pydantic import BaseModel, create_model
from pydantic.fields import ModelField
from pydantic.utils import lenient_issubclass


def patched_create_cloned_field(
        field: ModelField,
        *,
        cloned_types: Optional[dict[type[BaseModel], type[BaseModel]]] = WeakKeyDictionary(),
) -> ModelField:
    # _cloned_types has already cloned types, to support recursive models
    if cloned_types is None:
        cloned_types = WeakKeyDictionary()
    original_type = field.type_
    if is_dataclass(original_type) and hasattr(original_type, "__pydantic_model__"):
        original_type = original_type.__pydantic_model__
    use_type = original_type
    if lenient_issubclass(original_type, BaseModel):
        original_type = cast(type[BaseModel], original_type)
        use_type = cloned_types.get(original_type)
        if use_type is None:
            use_type = create_model(original_type.__name__, __base__=original_type)
            cloned_types[original_type] = use_type
            for f in original_type.__fields__.values():
                use_type.__fields__[f.name] = patched_create_cloned_field(
                    f, cloned_types=cloned_types,
                )
    new_field = fastapi.utils.create_response_field(name=field.name, type_=use_type)
    new_field.has_alias = field.has_alias
    new_field.alias = field.alias
    new_field.class_validators = field.class_validators
    new_field.default = field.default
    new_field.required = field.required
    new_field.model_config = field.model_config
    new_field.field_info = field.field_info
    new_field.allow_none = field.allow_none
    new_field.validate_always = field.validate_always
    if field.sub_fields:
        new_field.sub_fields = [
            patched_create_cloned_field(sub_field, cloned_types=cloned_types)
            for sub_field in field.sub_fields
        ]
    if field.key_field:
        new_field.key_field = patched_create_cloned_field(
            field.key_field, cloned_types=cloned_types,
        )
    new_field.validators = field.validators
    new_field.pre_validators = field.pre_validators
    new_field.post_validators = field.post_validators
    new_field.parse_json = field.parse_json
    new_field.shape = field.shape
    new_field.populate_validators()
    return new_field


import fastapi  # noqa

fastapi.routing.create_cloned_field = patched_create_cloned_field


from something.main import app  # noqa

Note that the docs at the top only fit our own setup with running everything in docker and note that I did remove the app name (replaced by "something").

Anyways this seems to work for us now and we do not have any issues. I cannot reproduce the problems we had any more. RAM usage is still also down by a huge amount.

Nice thing about this additional file + the monkey patch is that we still can just build a normal production version that does include this.

[melvinkcx]

Thank you for the patch, @ddanier.

I can attest that this is an issue with one of the larger services we have.

Before patching:

After patching:

[Lawouach]

Just leaving a note that I'll try the patch as well.

When I start the app in itself, it's not so much of a problem per se but in my tests, this slowness is repeated for each test case and has rendered the tests very slow.

[varneyo]

Glad I found this issue thread, my project was getting too slow to debug. maybe it will get some love at some point @tiangolo ?

[spro]

Same here, especially annoying with --reload and trying to test small changes

[timabilov]

Although turning off(or caching here) response models will speed up startup a lot, it is still a problem with cold starts of Lambda.
Because internally FastAPI does a lot with dependency management and still using that create cloned fields somewhere.
So, if you a fat app it will take 10-30 seconds to initiate API. and thats already enough to get timeout in AWS

[olgeni]

My current startup time is around 3.5 minutes 😅

[emann]

I used the branch in #4645 and the time taken to register routers is quite literally 10x faster - from 8s to 0.8s. The PR has been waiting for review for a while... at least the author is kind enough to keep his branch (mostly) up to date with master, so thats what we're using

[zanieb]

@tiangolo please let me know if there's anything I can do to get this migrated back to an issue since there's quite a bit of detail regarding the specifics of the problem here.

[followben]

I too can confirm after encountering an existing FastAPI app that has 48 routers and 300 routes that this is a signifiant problem adversely affecting startup time in both testing/ development and production (we run fastapi on both AWS Lambda and ECS).

For example, on my intel i9 16" MBP, the app takes 25-30secs to load with a 70% of that time accumulated in create_cloned_field. Here it is running under pytest:

Using a WeakKeyDictionary via @ddanier's patch and/ or #4645 drops that to a respectable 8 seconds.

@tiangolo - what's the most helpful way to get the change tabled by @madkinsz reviewed and merged?

[michaelgmiller1]

We have the exact same situation as @followben. Impractically slow startup time when containers need to be spun up. @ddanier's patch works great for local development, but we're hesitant to implement this in production given how it alters the internals.

What will it take to get this merged in @tiangolo ?

[anton-shauchenia]

We noticed the exact same issue when running FastAPI application on AWS Lambda. Applying the above patch allowed us to get a significant speed-up during our cold starts (in our case we gained 1s)

[ulan-yisaev]

FastAPI application on AWS Lambda.

We have the same issue with our FastAPI application on AWS Lambda. I'm going to test above workaround, but in the meantime I would like an official fix from the maintainer :)

[MarkusSintonen]

We had to monkeypatch badly behaving created_cloned_field completely with this:

class FixedFastAPI(FastAPI):
    _create_cloned_field_patched = False

    @staticmethod
    def _fix_fastapi_routes_init_issue() -> None:
        """
        Workaround for FastAPI initialization slowness causing slow startup and memory issues.
        https://github.com/tiangolo/fastapi/discussions/8609
        """
        import fastapi.utils

        def patched_create_cloned_field(field: ModelField, **_kwargs: Any) -> ModelField:
            # We dont need to use badly behaving create_cloned_field as we have ensured that the returned Pydantic
            # instance and sub-fields match exactly the given model. We dont need to handle the derived class issues
            # like in create_cloned_field.
            return field

        fastapi.utils.create_cloned_field = patched_create_cloned_field
        fastapi.routing.create_cloned_field = patched_create_cloned_field

    def __init__(self: AppType, **kwargs: Any) -> None:
        if not FixedFastAPI._create_cloned_field_patched:
            FixedFastAPI._fix_fastapi_routes_init_issue()
            FixedFastAPI._create_cloned_field_patched = True

        super().__init__(**kwargs)

As a suggestion, could we have a way to override the create_cloned_field behaviour in APIRoute class? This would allow applications to define what "response securing" means for them also avoiding the issues with current very heavy logic. So eg having:

class APIRoute(routing.Route):
    def __init__(self, ...) -> None:
        ...
        if self.response_model:
            self.secure_cloned_response_field: Optional[
                ModelField
            ] = self.create_secure_response_field(self.response_field)

    @classmethod
    def create_secure_response_field(cls, field: ModelField) -> ModelField:
        # Applications can define what response securing means for them
        return create_cloned_field(field)

With the monkeypatch we got a big boost in startup time, it dropped from 40s to under 10s. Also the memory usage dropped by 3-4x!:

[tiangolo]

Thanks for the thorough report and for the discussion everyone!

This should be fixed in #4645, in FastAPI 0.96.0, just released 🚀

Thanks @madkinsz for the fix!

[orfisko]

I don't know about you guys, but after an upgrade to the very latest version of fastapi with pydantic v2 my coldstart times went up with factor 3. I had so many code changes that I am not 100% sure of the cause, but it seems similar to the issue we addressed here...

[Kludex]

pydantic/pydantic#6768

[orfisko]

@Kludex thanks! I will track that issue

[kefahi]

I'm facing what appears to be the same issue (not sure).
After the switch to pydantic 2 my python app takes 13 seconds to start (on pydantic 1.x it used to be couple of seconds).

Run-time performance is good, but start up is really slow. I also noticed a huge increase in memory consumption.

Using latest packages ... FastApi 0.103.1, pydantic 2.3.0, pydantic-core 2.6.3

Analyzing with cProfile gives this

[kefahi]

I figured it out. The issue is not directly related to pydantic, but it seems it is somehow related to rust and regular expressions.
In my code I use "\w" in the regular expressions with pydantic field pattern=; which is very expensive (memory + cpu). Replacing the \w with the equivalent reduced ASCII, the performance problem disappeared.

[michaelgmiller1]

There are also a number of performance issues with pydantic v2:

pydantic/pydantic#7409
pydantic/pydantic#7263
pydantic/pydantic#7054
pydantic/pydantic#6768

If you don't see your particular issue logged in pydantic, suggest you create a new one. They're trying to improve the performance of v2.

[earonesty]

regex's are basically always secretly dangerous.

[manorlh]

Hello,
Has the issue been resolved? We've also encountered it, taking around 1 minute to load. Our large application contains numerous models, and it appears that this is causing the slow startup.

Is there a fast workaround, perhaps for local development purposes?

Thank you.

[zanieb]

Yes this was resolved in #4645