Why does subclassing TypedDict work with a field_validator even though this explicitly violates PEP 589 and type hinting guidelines?

[prrao87]

Background

A few days ago, Samuel posted a code example along these lines in my example Pydantic v2 code base. I've simplified it to a reproducible example below:

# pip install pydantic
from pydantic import TypeAdapter, field_validator
from typing_extensions import NotRequired, TypedDict


class Item(TypedDict):
    id: int
    price: float
    description: NotRequired[str]

    @field_validator("price", mode="before")
    def check_non_zero(cls, v):
        if v <= 0:
            raise ValueError("price cannot be zero")
        else:
            return v


if __name__ == "__main__":
    ItemsTypeAdapter: TypeAdapter[list[Item]] = TypeAdapter(list[Item])
    sample_data = [
        {"id": 1, "price": 10.0, "description": "A nice item"},
        {"id": 2, "price": 20.0, "description": "Another nice item"},
        {"id": 3, "price": 0.0, "description": "A bad item"},
    ]
    try:
        items = ItemsTypeAdapter.validate_python(sample_data)
        from pprint import pprint
        pprint(items, sort_dicts=False)
    except ValueError as e:
        print(e)

This produces the expected output (there will be a validation error as per the field validator constraints)

1 validation error for list[typed-dict]
2.price
  Value error, price cannot be zero [type=value_error, input_value=0.0, input_type=float]
    For further information visit https://errors.pydantic.dev/2.1.2/v/value_error

The rationale in not using BaseModel and instead subclassing TypedDict directly here is, as per Samuel:

Most of the time is spent in "serialization" namely converting the model to a dict. The model is quite simple in this case - just str and float types etc., no nested models etc. Since in this case you're not actually using the pydantic model, but converting it straight to a dict, I changed the code here to use a TypeDict, and do the excluding of None values during validation. This means there's no need for a formal "serialization" step - the output of validation to dicts is all you need.

Confusion with type checkers

When using VS Code, the Pylance language server (which it uses for linting) complains that this is incorrect usage.

Error: TypedDict classes can contain only type annotations

However, the code runs and there's clearly no syntax error.

I've filed an issue in the Pylance repo regarding this, and the folks there are of the opinion they're not sure if it's "correct" to subclass TypedDict and add custom methods to it, and that subclassing BaseModel is the right way to do it when field validators are involved. I realize it's still early days and that the Pydantic v2 docs are still catching up, but is there any documentation on how the inheritance from TypedDict works exactly? I'd love some tips in understanding how/why Pydantic is able to handle the specification of custom methods under TypedDict objects even though the object was originally intended to just house struct-like fields?

mypy rejected proposals to specify methods when TypedDict annotation is used

My concern is that there has been an issue rejected on the mypy repo, related to allowing custom methods within TypedDict that was closed as out of scope for this object type. I'm unsure if Pyright followed the same approach. So, if this is the case, how has Pydantic reconciled this and how does it pass type checkers if specifying field_validator alongside TypedDicts?

Update 1

I just ran mypy on the code snippet above, and indeed, it does fail type checking via mypy.

$ mypy example.py
example.py:10: error: Invalid statement in TypedDict definition; expected "field_name: field_type"  [misc]
Found 1 error in 1 file (checked 1 source file)

PEP 589 explicitly says method specification in TypedDict is not allowed

I just checked PEP 589 and it's clearly mentioned that methods are not allowed within TypedDict:

Methods are not allowed, since the runtime type of a TypedDict object will always be just dict (it is never a subclass of dict).

TypedDict isn’t extensible, and it addresses only a specific use case. TypedDict objects are regular dictionaries at runtime, and TypedDict cannot be used with other dictionary-like or mapping-like classes, including subclasses of dict. There is no way to add methods to TypedDict types. The motivation here is simplicity.

Questions

So I suppose my questions are:

1. Why does the above code snippet work? Is this because TypeDict is purely for type hinting and that the underlying dict is all that is seen during run time?
2. If the approach used in the above snippet, i.e., subclassing TypedDict and adding a field_validator to it, is non-standard and doesn't respect type hinting guidelines, can/should this approach even be used in a real Pydantic code base?
   • If not, how would one go about minimizing serialization burden while also using field validators? I don't see any other way to do this other than subclassing BaseModel, which is what's currently shown in the Pydantic docs

Update 2

I just stumbled across this old issue in Pydantic v1, and I'm now even more confused about what can/should be done in cases where I have a simple model like the above example 😅. I just want to validate the fields of a dict. Any clarification on what's going on at the Python level or the Pydantic level would be appreciated!

[alfawal]

I would suggest creating a new instance of TypedDict with a similar name (e.g. SerializedDict) and shipping it with Pydantic.

[adriangb]

That's an interesting idea, not sure how that'd interact with type checkers (i.e. will it get the benefits of a TypedDict like being able to use dict literal syntax?).

[alfawal]

I think something like SerializedDict = type.__new__(TypedDict, "SerializedDict", (), {}) would work fine. A bit deeper research is required to get it right.

[adriangb]

The short answer is that what you are using works at runtime but type checkers don't like it. Because it wasn't much extra work on our end / at runtime we thought it better to make it work even if users have to add a # type: ignore and figure out down the road if we can convince type checkers to allow it or look for alternatives.

I'll point out that there is an alternative for that particular example that:

1. Makes type checkers happy.
2. Is more composable and reusable.
3. Is much more human-readable and better at transmitting information of what is being validated.

import math
from typing import Annotated, TypeVar, Union

from annotated_types import Gt
from typing_extensions import NotRequired, TypedDict

from pydantic import TypeAdapter
from pydantic.functional_validators import AfterValidator

Number = TypeVar('Number', bound=Union[float, int])


def check_finite(v: Number) -> Number:
    if not math.isfinite(v):
        raise ValueError('Expected a finite number')
    else:
        return v


Positive = Annotated[Number, Gt(0)]
Finite = Annotated[Number, AfterValidator(check_finite)]

class Item(TypedDict):
    id: int
    price: Positive[Finite[float]]
    description: NotRequired[str]


if __name__ == '__main__':
    ItemsTypeAdapter: TypeAdapter[list[Item]] = TypeAdapter(list[Item])
    sample_data = [
        {'id': 1, 'price': 10.0, 'description': 'A nice item'},
        {'id': 2, 'price': 20.0, 'description': 'Another nice item'},
        {'id': 3, 'price': 0.0, 'description': 'A bad item'},
        {'id': 3, 'price': math.nan, 'description': 'Another bad one'},
        {'id': 3, 'price': math.inf, 'description': 'A big bad one'},
    ]
    try:
        items = ItemsTypeAdapter.validate_python(sample_data)
        from pprint import pprint
        pprint(items, sort_dicts=False)
    except ValueError as e:
        print(e)

3 validation errors for list[typed-dict]
2.price
  Value error, Expected a positive number [type=value_error, input_value=0.0, input_type=float]
    For further information visit https://errors.pydantic.dev/2.1.2/v/value_error
3.price
  Value error, Expected a finite number [type=value_error, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.1.2/v/value_error
4.price
  Value error, Expected a finite number [type=value_error, input_value=inf, input_type=float]
    For further information visit https://errors.pydantic.dev/2.1.2/v/value_error

[prrao87]

That's some extremely readable, beautiful code, thanks @adriangb!! This definitely addresses my concern regarding avoiding the type checker complaints. However, the bigger question that arises from all this is: when using Pydantic largely to validate dicts with standard value types (like floats, ints and strings) and no complex nested fields, does it ever make sense to subclass BaseModel? If validating millions of such dicts, I'm seeing a roughly 2x performance improvement by not using BaseModel and subclassing TypedDict instead (due to serialization overhead, presumably). From what Samuel mentioned, subclassing BaseModel is more powerful and allows access to the full power of Pydantic models, but for validating and coercing values in a dict, using struct-like objects like TypedDict makes more sense -- would you agree?

For newcomers to Pydantic, the docs don't show anything regarding this distinction, and on following the documentation examples where you unpack a dict into a BaseModel instance (Model(**data)) is what most people would follow, while being perfectly unaware that there is a better way to avoid all that overhead over millions of data points. Maybe it makes sense to highlight the use of struct-like objects like dataclasses and TypedDict more explicitly in the docs to educate newcomers who may want to use Pydantic for ETL use cases? I imagine the FastAPI crowd largely don't care about this distinction as FastAPI does most of this in the background anyway.