Serializing @property

[bradodarb]

Question

• OS: Linux(Ubuntu 18)
• Python version 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
• Pydantic version 1.0

I've pored over the documentation and issue and can't seem to find an example for serializing members other than class attributes.

class Rectangle(BaseModel):
    width: int
    length: int

    @property
    def area(self) -> int:
        return self.width * self.length

r = Rectangle(width=10, length=5)

print(r.json())

# What I'd like to see:
{ "width": 10, "length": 5, "area": 50 }

I'd also think that properties without setters would default to readOnly on the schema side as well.

It did seem there was a way to get something close to what I'm after via ORM mode and masking a property on the ORM model, but I'd hate to need to create a class just to contain properties.

Going over BaseModel._calculate_keys it seems BaseModel will not yield what I'm after.

Still, I'm hoping that it's just been a long day and I overlooked something :)

[dmontagu]

I think the only way you could serialize properties now would be to override the serialization-related methods to manually inject the property values. I don't think that should be hard, but I'm not sure. It's probably worth documenting if it is straightforward.

In general, it would be tough to do this "automatically" because properties are "stored" very differently than the model data. And for many people it would be a breaking change.

I think it could make sense to add support for this via config, but only if it didn't add more checks to the serialization process (which I think it might have to).

[bradodarb]

I wonder if implementing something like the schema/field as a decorator would be a good approach?

class Rectangle(BaseModel):
    width: bool
    length: int

    @property
    @field(title='Area')
    def area(self) -> int:
        return self.width * self.length

And on init, the internal fields dict could grab attributes that have the decorator and act accordingly.

I'd say the property above would set to readOnly on the schema output.

Could possibly used on methods to support writeOnly ?

I'd be happy to get a PR going with some guidance if this feature is desired by others (esp the property serializtion, not so much the method)

[samuelcolvin]

Happy to review a PR, hard to know how clean/performant it would be until someone tries it.

[dmontagu]

I think this is a really good idea, and like your proposed api @bradodarb, but adding as few checks as possible for non-users of the feature would be critical because of how the serialization functions are called recursively on each item in lists/dicts.

[stratosgear]

I was also looking for the serialization part of the @property fields

[MartinWallgren]

We do this in some of our models by overriding the dict method and injecting the properties to the resulting dict. I think the @field decorator is a really good idea to solve this in the framework in a stable way.

[MartinWallgren]

We do this in some of our models by overriding the dict method and injecting the properties to the resulting dict. I think the @field decorator is a really good idea to solve this in the framework in a stable way.

Example on how this can be done by overriding dict()

@classmethod
def get_properties(cls):
    return [prop for prop in cls.__dict__ if isinstance(cls.__dict__[prop], property)]

def dict(
         self,
          *,
          include: Union['AbstractSetIntStr', 'DictIntStrAny'] = None,
          exclude: Union['AbstractSetIntStr', 'DictIntStrAny'] = None,
          by_alias: bool = False,
          skip_defaults: bool = None,
          exclude_unset: bool = False,
          exclude_defaults: bool = False,
      ) -> Dict[str, Any]:
          """Override the dict function to include our properties"""
          attribs = super().dict(
              include=include,
              exclude=exclude,
              by_alias=by_alias,
              skip_defaults=skip_defaults,
              exclude_unset=exclude_unset,
              exclude_defaults=exclude_defaults
          )
          props = self.get_properties()

          # Include and exclude properties
          if include:
              props = [prop for prop in props if prop in include]
          if exclude:
              props = [prop for prop in props if prop not in exclude]

          # Update the attribute dict with the properties
          if props:
              attribs.update({prop: getattr(self, prop) for prop in props})
          return attribs

[matrixise]

Interesting feature, I was looking for it. Is there a PR for review/testing? Thank you

[samuelcolvin]

Not yet.

I think we still need to decide on exactly how it would work.

Questions:

• I think it would be good if this could be done via one decorator, will this upset mypy and if so can we extend the mypy plugin to support it?
• what should we call the decorator? computed_field, model_property?
• what arguments should it take? title, in_schema=True, alias?
• should the value be cached? I guess not, if it were to be cached we could compute all this at parse/init time and store the results in __dict__ but that would mean it wasn't updated on setting an attribute. maybe a cache=False argument to the decorator.

[jsoucheiron]

What if instead of decorators we use a default value? A bit similar to the field factory from dataclass.

Ex:

class Rectangle(BaseModel):
    width: int
    length: int
    area: int = property_value('area')

    @property
    def area(self) -> int:
        return self.width * self.length

We could even add a default validator for those properties that block setting the attribute value

[samuelcolvin]

I think that's uglier than the decorator approach outlined above.

[jsoucheiron]

Regarding the cache question. I'd go for no caching and use cached_property (from 3.8) if needed be. People in 3.7 would need to handle it manually, but it'd be pretty straightforward in future python versions.

[dmontagu]

Regarding mypy, the easiest way to deal with it would probably be to leverage the @property or @cached_property decorators, at least to "trick" mypy, if not to make direct use of them.

---

One possible approach -- I have no idea if the following would work, but something like:

# actual return type may or may not be the `property` type
def pydantic_property(*args, ** kwargs) -> Type[property]
    ...

Hopefully this would mean mypy would treat @pydantic_property(...) as equivalent to @property.

---

Another version to consider would be:

F = TypeVar("F")
def pydantic_property(...) -> Callable[[F], F]:
    ...

@pydantic_property(...)
@property
def blah(...)
    ...

This would allow the use of whichever property decorator you wanted (property, functools.cached_property in 3.8, cached_property.cached_property in 3.7, etc.), and I think would be fully mypy compatible.

---

I personally think it would be best to support both cached and un-cached versions, and it would be great if there was a way to leverage existing decorators so we didn't have to maintain that behavior ourselves.

If we want to move forward with this, I'm happy to try to look into the above more carefully.

[VianneyMI]

@samuelcolvin, @PrettyWood will V2 offer a solution to this issue ?

[nayef-livio-derwiche]

@rtunn

I did not check the interaction with fast api, thanks for the heads up

[Maydmor]

Wrote a little helper package, which also works for serialization.

Maybe this package helps some of you.

https://github.com/Maydmor/pydantic-computed

[gnkow]

How to deal with properties in mapped entity models when using ORM models?

[hofrob]

How to deal with properties in mapped entity models when using ORM models?

For some reason, it works in our code base with sqlalchemy ORM models. I just discovered this issue because we couldn't get it to work in a dataclass. 😕

[samuelcolvin]

This will be fixed via #5502 feedback their (or via new issues if the PR has been merged) very welcome.

[dmontagu]

Update: @computed_field is now available in the latest v2 alpha release v2.0a3; some docs are available in https://github.com/pydantic/pydantic/blob/main/docs/usage/computed_fields.md

[sshishov]

Hi @dmontagu and @samuelcolvin , is this feature going to be backported to v1? I am asking because v2 is still in alpha and this is quite important feature. Is it difficult to backport it or write the similar functionality on v1?

Or maybe for v1 you will recommend to use some third-party library like in this comment: #935 (comment)?

[samuelcolvin]

It won't be backported, v1 is receiving security and bug fixes only now.

[andrewmarcus]

In the meantime, here's a shim which you might be able to get working. I can't claim it's particularly efficient, and it only outputs dynamic properties into the output of functions like json() and dict(). It does not do anything about reading dynamic properties back into a model using property setters, so it's only a half solution. But it may meet your needs until you upgrade to Pydantic 2.

from collections.abc import Iterator
from inspect import getmro
from typing import TYPE_CHECKING, Optional, Union

from pydantic import BaseModel
from pydantic.utils import ValueItems

if TYPE_CHECKING:
    from pydantic.typing import AbstractSetIntStr, MappingIntStrAny, TupleGenerator


class BaseModelWithProperties(BaseModel):
    """
    Until we switch to Pydantic 2, this is a BaseModel class which includes dynamic properties
    in its JSON output.
    """

    @classmethod
    def property_iter(cls) -> Iterator[str]:
        """
        Iterates through all dynamic properties defined on the object using the @property decorator.
        It returns the names of all such properties in the current class as well as any parent
        classes.

        Yields:
            The names of any dynamic properties in this class and all superclasses.

        """
        # First yield any properties from superclasses
        for parent in getmro(cls):
            # Don't go higher than this base class
            if parent in {BaseModelWithProperties, BaseModel, object}:
                break
            if parent != cls and hasattr(parent, "property_iter"):
                yield from parent.property_iter()

        # Now yield additional properties in the subclass. Note that dynamic properties appear 
        # in the class `__dict__` but not in the instance `__dict__`.
        for name, prop in cls.__dict__.items():
            if isinstance(prop, property):
                yield name

    def __iter__(self):
        """
        Override the tuple iterator from Pydantic so `dict(model)` works and returns dynamic
        properties in addition to the defined Pydantic fields.
        """
        yield from super().__iter__()

        for prop in self.property_iter():
            yield prop, getattr(self, prop)

    def _iter(
        self,
        to_dict: bool = False,
        by_alias: bool = False,
        include: Optional[Union["AbstractSetIntStr", "MappingIntStrAny"]] = None,
        exclude: Optional[Union["AbstractSetIntStr", "MappingIntStrAny"]] = None,
        exclude_unset: bool = False,
        exclude_defaults: bool = False,
        exclude_none: bool = False,
    ) -> "TupleGenerator":
        """Override the base function used to iterate through all properties, used by `dict()` and
        `json()`. The dynamic properties will be included in the iteration after the Pydantic
        properties."""

        # First yield all the Pydantic properties
        yield from super()._iter(
            to_dict=to_dict,
            by_alias=by_alias,
            include=include,
            exclude=exclude,
            exclude_unset=exclude_unset,
            exclude_defaults=exclude_defaults,
            exclude_none=exclude_none,
        )

        # Filter certain properties in or out. This is borrowed directly from the super() method.
        # Merge field set excludes with explicit exclude parameter with explicit overriding
        # field set options.
        # The extra "is not None" guards are not logically necessary but optimizes performance
        # for the simple case.
        # This is copied directly from Pydantic
        if exclude is not None or self.__exclude_fields__ is not None:
            exclude = ValueItems.merge(self.__exclude_fields__, exclude)

        if include is not None or self.__include_fields__ is not None:
            include = ValueItems.merge(self.__include_fields__, include, intersect=True)

        allowed_keys = self._calculate_keys(
            include=include, exclude=exclude, exclude_unset=exclude_unset  # type: ignore
        )

        # Now iterate through the custom properties and export any which have not been excluded.
        for name in self.property_iter():
            value = getattr(self, name)

            if (allowed_keys is not None and name not in allowed_keys) or (
                exclude_none and value is None
            ):
                continue

            yield name, value