Unexpected model serialization based on the field annotation and the extra configuration

[bcdurak]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

Hey there,

I am having an issue with model serialization in Pydantic v2.

To showcase it, let's say I have a model called MyModel with a config field of type BaseConfiguration. Depending on the use case, I want to be able to assign different types of configurations to this field, or in other words, other instances of subclasses inheriting from the BaseConfiguration class (such as the SubConfiguration class).

Creating such instances does not raise a ValidationError, however, while serializing these instances, I got some outputs I was not expecting. Moreover, the outcomes change again depending on the extra parameter, set in the ConfigDict of the configuration classes.

I created the following table to showcase the different behaviors:

param_1	param_2	param_3	Outcome
allow	allow	BaseConfiguration	{'config': {'a': 1}}
allow	forbid	BaseConfiguration	{'config': ({'a': 1, 'b': '2'}, None)}
forbid	allow	BaseConfiguration	{'config': {'a': 1}}
forbid	forbid	BaseConfiguration	{'config': {'a': 1}}
allow	allow	SubConfiguration	{'config': {'a': 1, 'b': '2'}}
allow	forbid	SubConfiguration	{'config': {'a': 1, 'b': '2'}}
forbid	allow	SubConfiguration	{'config': {'a': 1, 'b': '2'}}
forbid	forbid	SubConfiguration	{'config': {'a': 1, 'b': '2'}}

I originally ran into this issue while migrating our codebase from pydantic v1 to v2, and we were using the use case (2) where the config field got serialized as a tuple and I got the following error as well:

  Expected `general-fields` but got `tuple` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_json(

In the v1 equivalent of the same example code, all cases listed above return {'config': {'a': 1, 'b': '2'}} which was the expected output from my side.

Example Code

from typing import Optional

from pydantic import BaseModel, ConfigDict, Field

param_1 = "allow"
param_2 = "forbid"


class BaseConfiguration(BaseModel):
    a: int = Field(default=1)

    model_config = ConfigDict(extra=param_1)


class SubConfiguration(BaseConfiguration):
    b: str = Field(default="2")

    model_config = ConfigDict(extra=param_2)


param_3 = BaseConfiguration


class MyModel(BaseModel):
    config: Optional[param_3] = Field(default=None)

    model_config = ConfigDict(validate_assignment=True)


sub_config = SubConfiguration()
my_model = MyModel(config=sub_config)
print(my_model.model_dump())

Python, Pydantic & OS Version

pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=false
                 install path: /Users/bariscandurak/.pyenv/versions/3.9.16/envs/zenml-temp/lib/python3.9/site-packages/pydantic
               python version: 3.9.16 (main, Mar 11 2024, 18:29:02)  [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-13.5.1-arm64-arm-64bit
             related packages: fastapi-0.110.0 typing_extensions-4.10.0 mypy-1.7.1 pydantic-settings-2.2.1
                       commit: unknown

[sydney-runkle]

@bcdurak,

Thanks for your question. The reason for the discrepancy here compared to V1 behavior is because of changes in how subclasses are serialized. To achieve V1-like behavior in all of the above cases, you can use the SerializeAsAny annotation around your param3 type:

from typing import Optional

from pydantic import BaseModel, ConfigDict, Field, SerializeAsAny

param_1 = "allow"
param_2 = "forbid"


class BaseConfiguration(BaseModel):
    a: int = Field(default=1)

    model_config = ConfigDict(extra=param_1)


class SubConfiguration(BaseConfiguration):
    b: str = Field(default="2")

    model_config = ConfigDict(extra=param_2)


param_3 = SerializeAsAny[BaseConfiguration]


class MyModel(BaseModel):
    config: Optional[param_3] = Field(default=None)

    model_config = ConfigDict(validate_assignment=True)


sub_config = SubConfiguration()
my_model = MyModel(config=sub_config)
print(my_model.model_dump())

You can read more about this setting here.

Regarding the case you've linked above, the serialization warning does seem odd. Leaving this open until we figure that out...

[sydney-runkle]

Here's a more simple repro of this bug:

from pydantic import BaseModel, ConfigDict, Field, TypeAdapter


class Base(BaseModel):
    a: int = Field(default=1)
    
    model_config = ConfigDict(extra='allow')


class Sub(Base):
    b: str = Field(default="2")

    model_config = ConfigDict(extra='forbid')


sub = Sub()
print(repr(sub))
#> Sub(a=1, b='2')

print(sub.model_dump())
#> {'a': 1, 'b': '2'}

ta = TypeAdapter(Base)
result = ta.dump_python(sub)
"""
/Users/programming/pydantic_work/pydantic/pydantic/type_adapter.py:356: UserWarning: Pydantic serializer warnings:
  Expected `general-fields` but got `tuple` - serialized value may not be as expected
  return self.serializer.to_python(
"""
print(result)
#> ({'a': 1, 'b': '2'}, None)

[bcdurak]

Hey @sydney-runkle, thank you for the quick response, that was really helpful.

I will try to adapt our code in a way so that we can use SerializeAsAny in our models.

[sydney-runkle]

@bcdurak,

We'll be releasing a new serialize_as_any runtime flag as well in the next release!

[bcdurak]

That's great news :) This was a feature heavily used in our codebase in v1, that would help us a lot.