exclude_none for specific fields only

[mar-muel]

Hi there! Apologies for asking stuff that is probably trivial, but couldn't find an answer to this.

I know it's possible to exclude None values globally. In my case, I'm generating a JSON response from FastAPI with Pydantic, and I would like to exclude only certain keys if None, but for all other fields, keep the default to showing null values, as sometimes they are meaningful.

What would be the easiest way to achieve this? Thanks for your help 🙏

[dmontagu]

In v2 this could be done with a model_serializer.

@samuelcolvin @adriangb is there any way we could make it possible for a field_serializer to exclude the field? Like, make it so the value doesn't end up in the final serialized object at all? I'm thinking the API could be something like raising a special exception in the field_serializer (raise pydantic.DoNotSerialize). That would prevent the need to handle certain return types specially, but maybe it's too weird. Or maybe it wouldn't work well with pydantic_core.

Either way, I think it would be somewhat powerful if we could support this using a field serializer, as it would allow you to get the exclude-if-none behavior through a type annotation.

[dmontagu]

Yeah, I think the big win comes from being able to use arbitrary logic to determine whether to include the field — it would mean that any use case a user might want can be handled by their own logic, rather than us having a bunch of types/flags/etc to maintain. (Especially if the info.context thing becomes a reality, in which case they could control it on a per-call basis.)

[mar-muel]

I fully agree with the need to be able to provide custom logic. Just as an example: If I'm using orm mode then I rely on logic that certain fields are not eagerly loaded. So depending on how you implement these flags Model.field is None (rather than something like inspect(self._obj).attrs.field.loaded_value == NO_VALUE) would trigger an SQL sub query, which is probably not wanted by the user.

[adriangb]

@dmontagu I think we could make something like this work:

from __future__ import annotations

from typing import Any, TypeVar

from pydantic_core import PydanticOmit
from typing_extensions import Annotated

import pydantic


def omit_if_none(v: Any, handler: pydantic.SerializerFunctionWrapHandler) -> str | None:
    # careful with omitting list items, see note below
    #   - raising a PydanticOmit in a list should exclude that item from the list not the list from the model...
    #     do we want a way to exclude the field based on a serializer of a list item? I don't think so...
    #   - ideally the same function can be used for both cases and differentiated them based on the info param
    #     that is: the function shouldn't create a bug where it omits list items when it meant to omit model fields
    if v is None:
        raise PydanticOmit
    return handler(v)


def omit_if_default(v: Any, handler: pydantic.SerializerFunctionWrapHandler, info: pydantic.SerializationInfo) -> str | None:
    # need:
    # - current field name
    # - know which fields were set and which were defaults
    raise NotImplementedError


T = TypeVar('T')

OmitIfNone = Annotated[T, pydantic.WrapSerializer(omit_if_none)]
OmitIfDefault = Annotated[T, pydantic.WrapSerializer(omit_if_default)]


class Model(pydantic.BaseModel):
    a: OmitIfNone[int | None] = None
    b: OmitIfDefault[int | None] = None

print(Model(a=1, b=2).model_dump())
print(Model(a=None).model_dump())

wdyt?

[TristanSpeakEasy]

Just following up to see if there has been anymore progress around this, we are currently trying to map the relationship of optional and nullable fields in an OpenAPI document where these concepts can be set independently so for a particular Model you may have some fields that are optional and if not set shouldn't be present in the output, if their optional and nullable then if they aren't set they shouldn't be in the output but they should if explicitly set to None and if just Nullable and not optional then an optional field should always be present as null in the output.

And I'm struggling to get this to work with the current tools available

[joshorr]

I would also love to see this feature, ie: some way to omit a value when serializing a field/custom-type.

At the moment, the only way I found that really works is deleting the attributes off of the model to prevent the field from being serialized. This works, but it's less than ideal because I want the user to still be able to access these attributes. Right now I implement a custom __getattr__ method to return a sentinel object that indicates an attribute is missing.

I tried using a custom model serializer to pop/delete dicts keys after calling the wrap function. I got it to work fine for model_dump() but not model_dump_json() because I get this error: error serializing to JSON: PydanticSerializationError: Unable to serialize unknown type on my sentinel object when I call the wrap(self) function in the serializer while doing a model_dump_json() on the model object. I could of course serialize the setinal object with some sort of special string or dictionary; but I don't want to do that because it takes a legal json value and treats it as 'missing', seems dangerous to do it that way.

If I could simply raise a PydanticOmit while Pydantic calls my serializer for the special sentinel object I am using to indicate value is currently missing, then it would solve this in a simple way. As it would cause Pydantic to not serialize the sentinel object at all, which is exactly what I want.

As a side-note: I also encountered a weird bug when using the validate_assignment=True option. When setting the value, and if you raise a PydanticOmit while that value is validating via core_schema.with_info_after_validator_function on the value's class/type it will cause an error inside of Pydantic. The validator works fine in all other situations, it only has the error while validating during assignment.

[larashores]

I have a very similar problem to this. I have a model of this form:

class SuperModel(pydantic.BaseModel):
    field: int | None = None
    submodel: SubModel

class SubModel(pydantic.BaseModel):
    field: int | None = None

When serializing this, I want to always include the fields in SubModel that are set to None. For example, I would like to achieve the following:

m = SuperModel(submodel=SubModel())
print(m.model_dump_json(exclude_none=True))
# > {"submodel": {"field": null}}

Basically, I want to ignore the exclude_none field for a specific model. I can achieve this with the following:

@pydantic.model_serializer(mode="wrap")
def always_include_none(self, wrap: Callable[[Self], dict[str, Any]], info: pydantic.SerializationInfo) -> dict[str, Any]:
    values = wrap(self)
    for key, value in self.model_fields.items():
        if info.exclude and key in info.exclude:
            continue
        if info.include and key not in info.include:
            continue
        if key not in values and getattr(self, key, None) is None:
            values[value.alias if info.by_alias else key] = None
    return values

It would be nice if this was a builtin feature. Perhaps it can be an override in the config dict?

class SubModel(pydantic.BaseModel):
    config = pydantic.ConfigDict(exclude_none=False)

Alternatively, maybe it can be a property of the field?

class SuperModel(pydantic.BaseModel):
    submodel: Annotated[SubModel, pydantic.Field(exclude_none=False)]

[mikenerone]

This feels like something that should be done with typing.Annotated. However, it also also occurs to me that an even simpler approach might be to define a sentinel value (pydantic.EXCLUDE) and allow any field serializer to return that to indicate that the field should be excluded. This makes it trivial to exclude via a @field_serializer method while also allowing existing annotations like WrapSerializer to be leveraged on a per-type basis.

Edit: I thought about it a bit more and realized this doesn't actually lend itself to Annotated immediately, because Pydantic's built-in serializer logic (where the sentinel value might be noticed) runs before the custom serializer function. There would need to be a little more code work to support it.

[KalleDK]

I have the same problem with an API where 2-3 fields should not be part of the json if they are None, but the 15 other fields should be included

[hampuskraft]

Inspired by previous responses, I came up with this solution, which works well enough for me at the moment—but I'd love to see a more ergonomic solution similar to what Adrian proposed.

@dataclass
class OmitIfNone:
    pass

class AppResponseModel(BaseModel):
    @model_serializer
    def _serialize(self):
        omit_if_none_fields = {
            k
            for k, v in self.model_fields.items()
            if any(isinstance(m, OmitIfNone) for m in v.metadata)
        }

        return {k: v for k, v in self if k not in omit_if_none_fields or v is not None}

class UserResponseModel(AppResponseModel):
    id: int
    deleted: Annotated[Literal[True] | None, OmitIfNone()]

[mikenerone]

I did something similar, though I also needed exclusions based on some more complex criteria. I made an ExcludeIf annotation that lets you pass in a predicate (like ExludeIf(lambda v: v < 5) (it also supports a when_used argument that works like Pydantic's), then a few subclasses of it where the predicates are predefined on the class (ExcludeIfNone(), ExcludeIfFalsy(), ExcludeIfEqual("a_value"), ExcludeIfIn(["some", "list"])).

[ashb]

This approach works for the serialization, but has the unfortunate side effect that the schema the the returned type is empty now.

[joshorr]

I've been experimenting with something like this as well. But there is a bit of a complication when you have multiple model serializers involved.

I wanted to create a very basic, library-style class model that would have this behavior generalized across all sorts of different kinds of models. This means some of these subclasses might want to further customize their model serialization behavior.

But if you set a model serializer on a subclass, the parent serializer does not get called automatically. You can call it via super().{parent-serializer-method-name}(...) but you have to ensure you configure your subclass model serializer the same way, and pass the same arguments and so on (like the 'wrap' argument for wrap mode). If the model changes how it wants to serialize things, the subclasses all have to be changed.

It would be nice if when calling the default serialization via mode='wrap' that it would call the superclasses model serializer for you, and then the subclass does not have to worry about how/if it should call the super classes serializer. The doc says it serializes it the same way it normally would when using 'wrap' mode, but it does skip the superclasses model serializer function unfortunately.

[heyglen]

Thanks for this, it worked for me though I had to tweak it to take into account Field aliases and to run PlainSerializer(s):

from dataclasses import dataclass

from pydantic import BaseModel, PlainSerializer, model_serializer

@dataclass
class OmitIfNone:
    pass

class AppResponseModel(BaseModel):
    @model_serializer
    def _serialize(self):
        skip_if_none = set()
        serialize_aliases = dict()

        # Gather fields that should omit if None
        for name, field_info in self.model_fields.items():
            if any(
                isinstance(metadata, OmitIfNone) for metadata in field_info.metadata
            ):
                skip_if_none.add(name)
            elif field_info.serialization_alias:
                serialize_aliases[name] = field_info.serialization_alias

        serialized = dict()

        for name, value in self:
            # Skip serializing None if it was marked with "OmitIfNone"
            if value is None and name in skip_if_none:
                continue
            serialize_key = serialize_aliases.get(name, name)

            # Run Annotated PlainSerializer
            for metadata in self.model_fields[name].metadata:
                if isinstance(metadata, PlainSerializer):
                    value = metadata.func(value)

            serialized[serialize_key] = value

        return serialized