✨ Support providing serializing context to response models

[alexcouper]

Context

pydantic/pydantic#9495 introduces passing context to TypeAdapter models.

Problem

Some fastapi based projects use model definitions that have multiple uses, and serializing for an API response is only one.
Up until now, fastapi/pydantic has supported exclusion of fields through the use of exclude keyword (example from sentry).

If however, a model needs to be serialized in different contexts - for example to be saved to dynamodb as well as to return via the API - it becomes limiting to have to opt in for inclusion/exclusion once and for all.

Solution

Pydantic Serialization Contexts provide a means to tell pydantic models the context of the serialization, and then they can act on that.

This PR makes it possible to reuse model definitions within a fastapi project for multiple purposes, and then simply state during route definition the context you want to be included when rendering.

Example

Simple example

class SerializedContext(StrEnum):
    DYNAMODB = "dynamodb"
    FASTAPI = "fastapi"

class Item(BaseModel):
    name: str = Field(alias="aliased_name")
    price: Optional[float] = None
    owner_ids: Optional[List[int]] = None

    @model_serializer(mode="wrap")
    def _serialize(self, handler, info: SerializationInfo | None = None):
        data = handler(self)
        if info.context and info.context.get("mode") == SerializedContext.FASTAPI:
            if "price" in data:
                data.pop("price")
        return data


@app.get(
    "/items/validdict-with-context",
    response_model=Dict[str, Item],
    response_model_context={"mode": SerializedContext.FASTAPI},
)
async def get_validdict_with_context():

    return {
        "k1": Item(aliased_name="foo"),
        "k2": Item(aliased_name="bar", price=1.0),
        "k3": Item(aliased_name="baz", price=2.0, owner_ids=[1, 2, 3]),
    }

And this model definition can be made more generalized like so:

class ContextSerializable(BaseModel):
    @model_serializer(mode="wrap")
    def _serialize(self, handler, info: SerializationInfo):
        d = handler(self)

        serialize_context = info.context.get("mode")
        if not serialize_context:
            return d
        for k, v in self.model_fields.items():
            contexts = (
                v.json_schema_extra.get("serialized_contexts", [])
                if v.json_schema_extra
                else []
            )
            if contexts and serialize_context not in contexts:
                d.pop(k)

        return d

class Item(ContextSerializable):
    name: str = Field(alias="aliased_name")
    price: Optional[float] = Field(serialized_contexts={SerializedContext.DYNAMODB})
    owner_ids: Optional[List[int]] = None

TODO:

• Wait for Support context being passed to TypeAdapter's dump_json/dump_python pydantic/pydantic#9495 to be merged
• Wait for Support context being passed to TypeAdapter's dump_json/dump_python pydantic/pydantic#9495 to be released (pydantic 2.8)
• Update version specification in this PR

[pierrecdn]

Hi @alexcouper 👋,

I'm wondering if your proposal can be used to handle my use-case: I'd like a parameter passed to an endpoint to expand some datamodels. By default I want one property being serialized with exclude_unset=True, but in case expansion is required, I want to turn it to False.
At first I though pydantic's serialization context would fix my issue, but I realize that without fastAPI integration of this to drive the context injection, it becomes yet another ugly hack.

class CustomObjectRead(SQLModel):
    identifier: int
    description: str | None = None
    config: Config

    @field_serializer('config')
    def expand_config(self, config: Config, info: SerializationInfo):
        context = info.context
        expand_config = context and context.get('expanded')
        return config.model_dump(exclude_unset=not expand_config)

@myrouter.get("")
async def customobject_get_all(
   expanded: bool = Query(default=False, description="Expand the configurations"),
) -> list[CustomObjectRead]:
     (...)
     # Find a way to pass expanded to the serialization context 
     return [my_object_collection]

The specifics here compared to your example is the fact the context should be hinted by the call itself.
Maybe I just lost myself and there's another elegant way to do it?

[alexcouper]

@pierrecdn

If you want the behaviour to be determined by a query param flag then I'm not sure this will help you -at least you won't be able to make use of the response_model_context parameter as in the example.

One way round this would be to have 2 route endpoints dependent on expansion that use the same underlying function to get the objects. If you do that you can use the response_model_exclude_unset flag already present in a route to determine behaviour.

[pierrecdn]

One way round this would be to have 2 route endpoints dependent on expansion that use the same underlying function to get the objects. If you do that you can use the response_model_exclude_unset flag already present in a route to determine behaviour.

Yeah, but the "2 route endpoint" is not something I'd consider. Basically I want the behavior to be driven by query, such as /api/resource?expanded=true.

I guess it's worth logging a new issue at that point.

[svlandeg]

Hi @alexcouper, thanks for the PR! We'll get this reviewed by the team and get back to you 🙏

[alexcouper]

There were linting issues when changing this to use dict approach from #11670 so leaving as is.

[alexcouper]

@svlandeg should I be regularly rebasing this PR, as conflicts come in, or is it OK to wait until a first review is done?

[agamble-oai]

Any update on this? This would be a useful feature and am keen to help land.

[YuriiMotov]

@alexcouper, thanks for working on this!

First we need to decide whether we want to implement this feature this way or we also want to support passing values from request.

Let's wait for Sebastián to look at this and take a decision

UPD:
Found the problem in this implementation.
Imagine we want to remove some field if we serialize model not in FastAPI context:

class MultiUseItem(BaseModel):
    name: str
    owner_ids: Optional[list[int]] = None

    @model_serializer(mode="wrap")
    def _serialize(self, handler, info: SerializationInfo):
        data = handler(self)
        if info.context and info.context.get("mode") == "FASTAPI":
            pass
        else:  # context is not passed or mode is not FASTAPI
            if "owner_ids" in data:
                data.pop("owner_ids")
        return data

If we try to use this model as response_model and return the instance of it from the endpoint, it will be serialized by jsonable_encoder without context passed and this way we loose owner_ids.

Full code example in details

Details

from typing import Optional

from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel, SerializationInfo, model_serializer


class MultiUseItem(BaseModel):
    name: str
    owner_ids: Optional[list[int]] = None

    @model_serializer(mode="wrap")
    def _serialize(self, handler, info: SerializationInfo):
        data = handler(self)
        if info.context and info.context.get("mode") == "FASTAPI":
            pass
        else:  # context is not passed or mode is not FASTAPI
            if "owner_ids" in data:
                data.pop("owner_ids")
        return data


app = FastAPI()


@app.get(
    "/items/exclude-if-not-fastapi",
    response_model=dict[str, MultiUseItem],
    response_model_context={"mode": "FASTAPI"},
)
async def get_validdict_with_context():
    return {"k": MultiUseItem(name="baz", owner_ids=[1, 2, 3])}


client = TestClient(app)


def test_exclude_if_not_fastapi_context():
    response = client.get("/items/exclude-if-not-fastapi")
    response.raise_for_status()

    expected_response = {"k": {"name": "baz", "owner_ids": [1, 2, 3]}}

    assert response.json() == expected_response

So, we also need to pass serialization context to jsonable_encoder.

This PR #13475 modifies jsonable_encoder to accept the context

[YuriiMotov]

I think we don't need this.
There are other places where the code may fail with older versions of Pydantic and if we try to cover all such cases we will end up having huge amount of runs

[YuriiMotov]

Suggested change

	#
	# context argument was introduced in pydantic 2.8
	if PYDANTIC_VERSION >= "2.8":
	return self._type_adapter.dump_python(
	value,
	mode=mode,
	include=include,
	exclude=exclude,
	by_alias=by_alias,
	exclude_unset=exclude_unset,
	exclude_defaults=exclude_defaults,
	exclude_none=exclude_none,
	context=context,
	)
	else:
	return self._type_adapter.dump_python(
	value,
	mode=mode,
	include=include,
	exclude=exclude,
	by_alias=by_alias,
	exclude_unset=exclude_unset,
	exclude_defaults=exclude_defaults,
	exclude_none=exclude_none,
	)
	kwargs: dict[str, Any] = (
	{"context": context} if PYDANTIC_VERSION >= "2.8" else {}
	)
	return self._type_adapter.dump_python(
	value,
	mode=mode,
	include=include,
	exclude=exclude,
	by_alias=by_alias,
	exclude_unset=exclude_unset,
	exclude_defaults=exclude_defaults,
	exclude_none=exclude_none,
	**kwargs,
	)

This doesn't give any linting issues

[YuriiMotov]

Suggested change

	Note: This feature is a noop on pydantic < 2.8
	Note: This feature requires Pydantic 2.8 or higher

[github-actions]

This pull request has a merge conflict that needs to be resolved.