Lazy generated fields

[dgasmith]

We have a bit of a strange use case where we need to generated hundreds of thousands of objects which have an optional field that is not commonly used, has a way to automatically be generated based off other fields, is in an even rarer case be changed by the user and should retain that value. This necessitates a field that can be user provided, is not automatically generated on model initialization for speed and memory footprint, but can be lazily generated on access.

class LazyModel(BaseModel):
    symbols: List[int]
    mass: List[float] = None

    @lazy('mass', values)
    def _generate_mass(self) -> List[int]:
        return [MASS_LOOKUP[k] for k in values['symbols']]

>>> model = LazyModel(symbols=["H", "C"])
>>> model.mass
[1, 6]
>>> model.dict()
{"symbols": ["H", "C"]}

>>> model = LazyModel(lazy=[1, 2, 3], mass=[1.1, 6.1])
>>> model.mass
[1.1, 6.1]
>>> model.dict()
{"symbols": ["H", "C"], "mass": [1.1, 6.1]}

For normal classes we would do this with None fields and property decorators. We haven't found a good solution with the pydantic metaclass tech and would be curious if anyone had suggestions.

May be related to #935.

[samuelcolvin]

How about this:

from typing import List
from pydantic import BaseModel

class LazyModel(BaseModel):
    symbols: List[int]

    @property
    def mass(self) -> List[int]:
        mass = self.__dict__.get('mass')
        if mass is None:
            print('calculating mass...')
            mass = [42]  # [MASS_LOOKUP[k] for k in values['symbols']]
            self.__dict__['mass'] = mass
        return mass

m = LazyModel(symbols=[1, 2, 3])
print(m.dict())
#> {'symbols': [1, 2, 3]}
print(m.mass)
#> calculating mass...
#> [42]
print(m.mass)
#> [42]
print(m.dict())
#> {'symbols': [1, 2, 3], 'mass': [42]}

?

[loriab]

The suggestion looks good for the lookup and dictionary repr case, but then pydantic validation doesn't kick in for user-specified mass nor is the mass field represented in .schema() output?

[samuelcolvin]

That's not currently possible with pydantic.

#935 would do some of this but wouldn't be lazy and I don't think it would allow you to either a set mass or else calculate it.

[dgasmith]

Ah, I see the issues with the property order of operations. We can hack around this with aliases:

from typing import List
from pydantic import BaseModel

class LazyModel(BaseModel):
    symbols: List[int]
    mass_: List[float] = None

    class Config:
        fields = {'mass_': 'mass'}

    @property
    def mass(self) -> List[int]:
        mass = self.__dict__.get('mass_')
        if mass is None:
            mass = [42]  # [MASS_LOOKUP[k] for k in values['symbols']]
        return mass

Example:

m = LazyModel(symbols=[0], mass=[10.3])
print(m.mass)
#> [10.3]
print(m.dict(exclude_unset=True, by_alias=True))
#> {'symbols': [0], 'mass': [10.3]}

m = LazyModel(symbols=[0])
print(m.mass)
#> [42]
print(m.dict(exclude_unset=True, by_alias=True))
#> {'symbols': [0]}

This makes things a bit bulky but workable for this somewhat special class.

[samuelcolvin]

Yes, except you might want to save mass back into self.__dict__ to avoid re-doing the calculation.