add skip_none argument

[JakeSummers]

Question / Feature Request

Suppose that I have a class like this:

class Jake(BaseModel):
    foo: Optional[str]

When I serialize it, I only want to serialize optional values when they are not null. The skip_defaults flag does something similar to what I am looking for but has this weird case:

>>> j = Jake()
>>> 
>>> # This is the behavior I expect: 
... j.dict(skip_defaults=True)
{}
>>> 
>>> # Set the foo property: 
>>> j.foo = "baz"
>>> 
>>> # This is also the behaviour I expect: 
... j.dict(skip_defaults=True)
{'foo': 'baz'}
>>> 
>>> 
>>> # Unfortunately, this is not the behavior that I expect:
... j.foo = None
>>> j.dict(skip_defaults=True)
{'foo': None}

---

Right now after I serialize to dict, I walk the dict and remove all of the None values recursively, but I would rather not have to do this. Is this built into Pydantic? Is the behavior that I am seeing a bug?

[dmontagu]

This behavior is actually not a bug (I found it a little confusing at first too) -- it is intended so that you can tell precisely which fields were set on the specific object instance, rather than taking the default value based on the class definition. (I've included an example below to show why this might be useful.)

Removing None from the dicts seems most commonly relevant to serialization logic; based on this comment this seems explicitly outside the scope of pydantic.

For what it's worth, FastAPI has a function (fastapi.encoders.jsonable_encoder) that is specifically intended for converting (nested) pydantic models to dicts, and it has an include_none parameter that does what you want. If you want this capability in the context of serialization, you might find this function useful, even if only as a reference implementation (it handles a number of other serialization edge cases as well).

---

Here's a concrete example of how the skip_defaults=True logic can be useful in the case of a REST resource with optional properties:

Imagine you want to store a user's username, and optionally an email.

class User(BaseModel):
    user_id: uuid.UUID
    username: str
    email: Optional[str]

A clean way to implement updates is to have a corresponding model where all fields are optional. Imagine you have the following update function and model:

def update_user(user_id: uuid.UUID, **update_kwargs):
    for key, value in update_kwargs.items():
        set_user_attribute(user_id, key, value)

class UserUpdate(BaseModel):
    user_id: uuid.UUID
    username: Optional[str]
    email: Optional[str]

Then

user_update = UserUpdate(user_id=<user_id>, email="a@b.com")
update_user(**user_update.dict(skip_defaults=True))

makes sure that you only update the fields that were specified; in this case, only the email address. (This is a common pattern for the HTTP PATCH method.)

If the skip_defaults=True logic always removed None, then it would no longer be possible to specify that you want to change the email attribute from a non-None value back to None.

[samuelcolvin]

great answer @dmontagu

[JakeSummers]

@samuelcolvin - Is this the kind of functionality that you would consider for Pydantic? Could we leave this open as a feature request?

(Thanks @dmontagu great explanation of that behaviour!)

[samuelcolvin]

I'd accept skip_none keyword argument to dict() and friends.