Add public BaseModel.iter() method

[Bobronium]

Feature Request

Add BaseModel.iter() method which will behave exactly like a dict, but without converting nested values to dict by default and will be a generator instead

Why

Imagine you want to iterate through values in model, but also need to exclude some, or use aliases instead of attributes. So what you need to do is write something like this:

for attr, value in model.dict(exclude={...}, by_alias=True).items():
    do_something(attr, value)

Obviously, in this case python will firet evaluate model.dict(exclude={...}, by_alias=True). part, going through all values and only then you will be able to iterate, going through the same values again.

Also, by using dict nested values are force-converted to dict, and this might be very slow when we just want to iterate through top-level values.

We also cannot use existing _iter(), because some work is actually done in dict()

How we can solve it

By having iter() method we can bypass this double-iteration problem and converting-to-dict problems:

for attr, value in model.iter(exclude={...}, by_alias=True):
    do_something(attr, value)

Selected Assignee: @hramezani

[samuelcolvin]

This already exists!

It's called ._iter().

[samuelcolvin]

Should be documented I guess.

[Bobronium]

This already exists!
It's called ._iter().

Yeah, but I mentioned, why it cannot be used in this case:

We also cannot use existing _iter(), because some work is actually done in dict()

By some work i mean resolving aliases keys and excluding the values

[samuelcolvin]

I don't want to add more public method to models, best I think to implement this yourself.

If there's a simple way to achieve this with changes to existing functions of happy review a PR.

[samuelcolvin]

Sorry, I missed your comment when replying in a hurry.

[Bobronium]

I don't want to add more public method to models, best I think to implement this yourself.

Just to be clear, thats valid even if ._iter() itself would become public without adding any new methods?

If there's a simple way to achieve this with changes to existing functions of happy review a PR.

I guess the simplest way would be to move those excluding and aliases stuff from .dict() to ._iter(), that will also simplify .dict() too.

I'll try to play with it soon and will be happy to make a PR once it'll be done.

[dmontagu]

For what it's worth, I would be in favor of moving logic from dict into _iter, assuming it didn't result in meaningful performance penalties in any public methods -- I think it would simplify/result in a smaller diff for the implementation of dump_as_type from #812, and the related serialization performance improvements (even when not dumping as a specified type).

[Kludex]

On V2, we can leverage .model_dump() as follows:

from pydantic import BaseModel


class Foo(BaseModel):
    a: str
    b: int


foo = Foo(a="a", b=1)

for key, value in foo.model_dump().items():
    print(key, value)

For that reason, I'll be closing this issue. 🙏

[gsakkis]

@Kludex model_dump is not equivalent to _iter if there are fields with custom serialization. Here's an example for a bson.ObjectId field serialized as string.

import bson
from pydantic import BaseModel, Field
from pydantic_core import core_schema


class PydanticObjectId(bson.ObjectId):
    @classmethod
    def __get_pydantic_core_schema__(cls, source_type, handler):
        serialization = core_schema.plain_serializer_function_ser_schema(str)
        return core_schema.is_instance_schema(cls, serialization=serialization)


class Foo(BaseModel):
    id: PydanticObjectId = Field(default_factory=PydanticObjectId)
    b: int


foo = Foo(b=1)
print(f"str: {foo}")
print(f"_iter: {dict(foo._iter())}")
print(f"model_dump: {foo.model_dump()}")

Output:

str: id=ObjectId('64e3afb883f5e2aaed388877') b=1
_iter: {'id': ObjectId('64e3afb883f5e2aaed388877'), 'b': 1}
model_dump: {'id': '64e3afb883f5e2aaed388877', 'b': 1}

So unless there is a way to prevent custom serialization, I'd suggest to reopen this issue.

[samuelcolvin]

no longer really relevant in V2 - _iter has gone, replaced by the underlying methods in pydantic-core.

[gsakkis]

An example of how the previous example would work by using pydantic-core methods instead of _iter would help.

[bojanbg]

@Kludex model_dump is not equivalent to _iter if there are fields with custom serialization. Here's an example for a bson.ObjectId field serialized as string.

So unless there is a way to prevent custom serialization, I'd suggest to reopen this issue.

I think there is a problem with your PydanticObjectId definition. If I use the one from here, this is the output I get running your example:

str: id=ObjectId('655f16f55d40bae3a225723f') b=1
_iter: {'id': ObjectId('655f16f55d40bae3a225723f'), 'b': 1}
model_dump: {'id': ObjectId('655f16f55d40bae3a225723f'), 'b': 1}

[thusithaC]

So what was the final verdict? Can we use model_dump instead of _iter?