Why no general implementation of polymorphic deserialization?

[argibbs]

Hi,

Just started using pydantic, and can't get over how good it is.

I've started using it to solve a problem that has often bugged me info python. And so, I have a question about the ability to do type-aware deserialisation of classes without the need to go via a container class and a discriminator.

I'm using pydantic 1.10 and python 3.10.

I started by shamelessly copying #3091 (h/t @PrettyWood & @jheddings) and then wondered if it was possible to pull out the logic into a general class.

So I tried (disclaimer - I'm bad at names):

class PolymorphicBaseModel(BaseModel):
    type: str

    _subtypes = dict()

    def __init_subclass__(subcls, type=None, **kwargs):
        super().__init_subclass__(**kwargs)
        if type:
            # n.b. if a subclass declares its own _subtypes dict, it'll take precedence over this one.
            # This would allow us to re-use the same type names across different classes.
            if type in subcls._subtypes:
                raise AttributeError(
                    f"Class {subcls} cannot be registered with polymorphic type='{type}' because it's already registered "
                    f" to {subcls._subtypes[type]}"
                )
            subcls._subtypes[type] = subcls

    @classmethod
    def _convert_to_real_type(cls, data):
        data_type = data.get("type")

        if data_type is None:
            raise ValueError(f"Missing 'type' for {cls}")

        subcls = cls._subtypes.get(data_type)

        if subcls is None:
            raise TypeError(f"Unsupported sub-type: {data_type}")
        if not issubclass(subcls, cls):
            raise TypeError(f"Inferred class {subcls} is not a subclass of {cls}")

        return subcls(**data)

    @classmethod
    def parse_obj(cls, data):
        return cls._convert_to_real_type(data)

Then the definition of Animal, Cat, etc from the linked discussion becomes:

class Animal(PolymorphicBaseModel):
    name: str
    color: str = None

class Cat(Animal, type="cat"):
    type: Literal["cat"] = "cat"
    hairless: bool = False

class Dog(Animal, type="dog"):
    type: Literal["dog"] = "dog"
    breed: str

class Person(BaseModel):
    name: str
    # Note the inclusion of the specific sub-types here. 
    # Round-tripping Person->Json->Person will degrade a Cat to Animal if Cat was missing from the union
    pet: Cat | Dog | Animal = None

And this seems to work. I can construct cat, dog, etc. explicitly, without needing to set type, and this round trips cleanly to/from json but keeping the specific Cat implementation as the pet member rather than degrading to a general Animal. e.g.:

    cat = Cat(name="Bacon Cat", color="Greyish")
    print(cat)
    print(cat.json(indent=2))
    print(Animal.parse_raw(cat.json(indent=2))) # returns a Cat instance
    print(Cat.parse_raw(cat.json(indent=2))) # returns a Cat instance
    print(Dog.parse_raw(cat.json(indent=2))) # Blows up because it's not a sub-class. Good dog.

    me = Person(name="Me, Myself", pet=cat)
    print(me)
    print(me.json(indent=2))
    print(Person.parse_raw(me.json(indent=2))) # still contains a Cat instance, not a mere Animal

Question

Sorry it took a while to get to it, but my question is, "what am I missing?".

Or to put it another way, this seems like a useful pattern to have, is there a good reason why this pattern isn't more widely published (I mean, it wasn't hard but it took several goes before I found the right combination of keywords that threw up #3091), or even more ambitiously, why a base class like this doesn't ship as part of pydantic? I will be the first to admit, this back-of-the-napkin implementation is a bit unpolished (suggestions welcome!), and the repetition of the type property three times is annoying, but the behaviour seems sound, and it's not really any worse than the sort of thing I have to do to make this work in jackson for Java.

One thing I did do different to the linked discussion #3091 was to remove

    @classmethod
    def __get_validators__(cls):
        yield cls._convert_to_real_type

As it didn't seem to do anything... Validation of the json still seemed to work (e.g. omitting breed from a Dog json still throws a missing value error). And in fact with this present, I couldn't even construct the Person object in code (the me = Person... bit above); it would blow up in _convert_to_real_type because the data object is no longer the deserialized dict from the json, but is in fact the Cat class. So maybe this is why; is that validator bit really important?

Anyhoo, would love to know if I'm missing something before I start using this everywhere and then come to regret it.

Thanks!

[jheddings]

For what it is worth, I've been using this pattern extensively in Notional (specifically in the core.py classes). I have yet to run into issues, but would definitely like to see a more general / supported approach in Pydantic if it makes sense!

[argibbs]

Niiice. Way more refined than my impl above.

I see you kept the __get_validators__ classmethod, and simply handle the case that was blowing me up (data is a class not a dict). If it's not too cheeky, can I ask what problem / edge case retaining that validators method handles? I played briefly without it, but it all seemed to work.

[jheddings]

(moving into thread) To be honest, at this point my unit tests won’t pass without it… That’s about all I can say 😉I’ll take a closer look and see if I can sort out what use case it is solving.

[dmontagu]

Glad to hear you are enjoying pydantic!

I'm struggling to follow the request here super closely, but my understanding is that you are looking for something that behaves like a discriminated union, but perhaps doesn't require a discriminator? And maybe is able to infer what the members of the discriminated union should be by registering subclasses during __init_subclass__? I see the benefits of the whole subclasses-get-registered thing, though ideally you'd be able to drop the pet: Cat | Dog | Animal = None and replace it with pet: Animal = None in that case.

That said, I'll warn that this will probably break in at least some subtle ways in the soon-to-be-released v2 due to the way parsing has been moved into pydantic-core.

BUT — in pydantic-core, we have much more general functionality for doing discriminated unions that goes beyond the presence of a single shared literal attribute. So, in principle in v2, it's possible to create a pydantic-core schema that will call a function on the input, and has a mapping of the return value of that function to the type it should be parsed as.

We haven't exposed an easy way to define this quite yet, but I do think we can make this whole pattern more ergonomic if we do things right. You can see some more detail in the documentation of pydantic_core.core_schema.tagged_union_schema, specifically here: https://github.com/pydantic/pydantic-core/blob/2a7a2f6393ad6eacaaa7204ac96ba244e0f64861/pydantic_core/core_schema.py#L2470-L2471

---

I'll also add that there was some discussion of adding a notion of "sealed" types in python at the bottom of PEP591, and I've seen this document proposing it in more depth, but I think it has been rejected (or at least deferred). I think that would make some of the implicit polymorphic stuff easier in common cases since it would be possible to know all the subclasses that should be considered without needing to manually create unions/etc.

[argibbs]

you are looking for something that behaves like a discriminated union, but perhaps doesn't require a discriminator

Close. A discriminated union requires a containing class for you to define the union in (the Person class in the example above). They work great, I have no problems with them. I'm trying to make it work without the containing class.

What I'm aiming for is a class ancestry like, class Parent, class Child1(Parent), class Child2(Parent) where I can accept json which might be a Child1 or a Child2 (I don't know which in advance, and I'm not allowed to peek at the json to find out), and I can do Parent.parse_raw(child_1_or_2_json) and it'll actually return an instance of the correct child class (rather than just an instance of Parent with all the surplus child properties thrown away).

though ideally you'd be able to drop the pet: Cat | Dog | Animal = None and replace it with pet: Animal = None in that case.

Indeed. And ideally find a way to use the type property in the init subclasses to set / default the type member of the model so I don't have to repeat repeat repeat myself. But that would be icing on the cake.

AIUI, the __init_subclass__ logic handles the Animal.parse_raw-returns-an-actual-Cat functionality which I'm so enamoured of, and then the pet: Cat | Dog | Animal = None is utilising the well known discriminated union behaviour. Put the two together and I get everything I want. Or at least it sounds like I do, given no one is telling me to put the pin back in the grenade.

this will probably break in at least some subtle ways in the soon-to-be-released v2

Awww, just when I was having fun 😆 . I guess I'll cross that bridge when I get to it.

[dmontagu]

If you don't want the containing class, in v2 you should be able to use TypeAdapter to parse data into a discriminated union:

from pydantic import TypeAdapter

adapter = TypeAdapter(Annotated[Union[Child1, Child2], Field(discriminator='type')])

child = adapter.validate_json(my_json_data)

You can also still use the __init_subclass__ pattern to get the behavior you want if you don't want to have to manually create the Union like this.

Either way, it should definitely be possible to make everything work as smoothly as you currently experience using __get_pydantic_core_schema__ or similar for this class, though may take some care due to the way the method is used for BaseModel subclasses now, and is called eagerly. (As a result, the __get_pydantic_core_schema__ you want might need to come from an annotation. I realize that's not a great description, but it will hopefully become more clear as we improve the v2 docs.)

If you want this thing to also work as a field, we are planning to add RootModel as a class in the coming days, and it should function similar to a BaseModel with __root__ from v1, but I get the sense that that isn't what you actually want.

In v2, you may also be able to achieve this with a pydantic.functional_validators.PlainValidator annotation. Basically, happy to help more with this when you are ready to migrate to v2 if you have questions then.

[argibbs]

Ah, smashing. Will probably try to jump to v2 soon after it releases, seems like there's some really good stuff in there.

Ty!

[hassec]

Sorry for being a bit late to the party and reactivating this thread, but I've been looking into how to best deserialize polymorphic classes in pydantic v2 and haven't found a good solution yet.

@dmontagu, the above sounds like you had an idea in mind that could work? But I've not been able to piece it together based on just the above.

I also think that this might be what @jheddings is asking in #7008, if I understand that correctly?

So I'd be super grateful for some guidance/tips on how to maybe get this to work in V2 😊