Change of API in v2 to declare required and optional fields

[PrettyWood]

Hi everyone!

I'm playing a lot with the new builtin generic types in 3.9 and type union operator introduced in 3.10 (I even wrote a backport for 3.6+ since I love them so much 😆).
Playing with them and pydantic, I really feel like the API can be challenged for v2. The idea is:
All fields without a default value would be required and all fields with a default value would be optional.
For example

class Model(BaseModel):
    a: str  # required field
    b: str | None  # required nullable field
    c: str = ...  # optional field with no default value
    d: str = 'c'  # optional field with a default value
    e: str | None = ...  # optional nullable field with no default value

It would imply that Optional[str] does not mean "optional" but more "nullable" (which is what it is 😄)
To define an optional field without actually setting a default value, I suggest we use Ellipsis. I like the fact that we don't need to import a Undefined sentinel to say "no value at all".
It would be a bit confusing when switching from v1 to v2 since ... is currently used to define a required field but IMHO it makes more sense to write a model this way with a homogeneous API.

For Field it would just mean Field(description='') is required when Field(..., description='') is optional.
For create_model it's the same

create_model(
  'Model',
  my_required_field=str,
  my_optional_field=(str, ...),
  my_required_field_2=(str, Field(description='')),
  my_optional_field_2=(str, Field(..., description=''),
)

@samuelcolvin @StephenBrown2 @MrMrRobat others WDYT?

Partially related issue #990
Related issue #1223

---

EDIT: I tried for the first time to convert an issue into a discussion and it works well 🎉 Love this new feature! 😄
I also add directly up here the proposal for TypedDict (thanks @mpkocher!) with a new syntax Required[...].
If they go with it, it can be interesting to have the same interface as TypedDict for BaseModel and co

[StephenBrown2]

I was initially thrown off by the union operator, thinking "That's not valid!" but, after looking at it further, I must say, it is highly intriguing. It does look like it would solve the "What is Optional" issue that many have had. Would we even need/use Optional? Since it currently means Union[type, None], I presume it would line up with b: str | None # required nullable field. Hopefully not breaking many uses currently in the wild.

If we added future-typing as a dependency for python<3.10, I think it would indeed make a lot of things simpler.

[layday]

What would it mean for an optional field to not have a default value? Would the attribute be omitted from model instances and how would type checkers be made aware?

The ellipsis is also only valid as a placeholder in functions:

def foo(
    # this type checks
    bar: str = ...,
) -> None: ...

class Foo:
    # this doesn't: 'Incompatible types in assignment (expression has type "ellipsis", variable has type "str")'
    bar: str = ...

[Bobronium]

I see potential for encountering a lot of AttributeErrors here 😁

AFAIK, you can't tell mypy that specific attr might be undefined on some instances. So we'll be forced to check if hasattr(model, "foo").

What are the benefits of having such fields?

[lsorber]

There is certainly a use case for

optional fields that are "undefined" (≠ None) when they are not provided

as evidenced by the several existing workarounds in #1673.

[PrettyWood]

The ellipsis is also only valid as a placeholder in functions

Hmm that's already the case now when we force required fields with ... right?

AFAIK, you can't tell mypy that specific attr might be undefined on some instances

And indeed it would need some extra work on the mypy plugin to make it work but I feel like it should be doable ;)

My main focus is making models more explicit and the DX.
For me the main advantage is really differentiate required, nullable and optional fields where all the default values used in the code would be Undefined.

[Bobronium]

Hmm that's already the case now when we force required fields with ... right?

How's that?

So by saying «optional field with no default value», you mean that if field was not in data, attr will still be present on model and value would be Undefined instead of None?

[PrettyWood]

Sorry @MrMrRobat maybe I was not clear.
I just meant we already use ... in BaseModel so the "it's not understood by mypy" is already an issue now and it should be possible to tackle it by the plugin.

And yes I just mean the default value used behind the scene is always Undefined. And dict(), json(), ... methods would of course remove them. It would also make schema generation easier and closer to the model where you could have not required fields without a default value.

[layday]

Hmm that's already the case now when we force required fields with ... right?

'Forcing' required fields with the ellipsis is redundant which is why it hasn't been an issue so far, I guess.

And indeed it would need some extra work on the mypy plugin to make it work but I feel like it should be doable ;)

Well, there are other type checkers out there, some of which do not support plug-ins. Pyright in particular has been gaining in popularity owing to its integration in VS Code.

[PrettyWood]

Well, there are other type checkers out there, some of which do not support plug-ins

You're absolutely right. I only use mypy and tend to forget others may be a problem too 😅
Maybe the ... is not the best idea and we should go with something more robust like

from pydantic import BaseModel, Unset

class Model(BaseModel):
    a: int | None
    b: int | None | Unset  # or `Undefined` or something else...
    c: int | None | Unset

with class Unset: ... and a default value for every single field equal to Unset().
At least it would remove the type checking issues, be explicit and still quite concise

[layday]

That seems reasonable from a typing standpoint, but I'm not sure I fully understand the use case, if you could explain how this would be useful.

[PrettyWood]

Sorry if I'm not clear :(
Well for v2 there are already two forthcoming changes:

• switch to "required optional fields" in Required Optional fields #990, which imo makes more sense since Optional[x] only means Union[x, None] (which is even more obvious when you use the 3.10 syntax)
• support nullable in JSON schemas in JSON Schema Nullable Required Syntax? #1270

My main point is to be able to declare a field that may be omitted, but may not be null if the field is supplied
(which has already been discussed in #1761 for example). So we need a way to say "this field is not required but doesn't have a default value" hence the Unset or whatever else proposal

[layday]

I see, so this would be for fields that (a) can be omitted but (b) should also not have a default value in a JSON schema. The discussion of Omittable on typing-sig would be of some interest here - if the Omittable semantics could be extended to native classes in addition to TypedDict.

As an aside, if #990 would mean reverting to normal typing semantics and = None is required to make a typing.Optional field optional, that would resolve #1270 without having to add anything extra.

[mpkocher]

from pydantic import BaseModel, Unset

class Model(BaseModel):
    a: int | None
    b: int | None | Unset  # or `Undefined` or something else...
    c: int | None | Unset

I don't think this is a great idea to attempt to capture this at the type level by using Unset as a type. It might be better to capture the "unset" semantics as metadata using Python 3.9's Annotated feature.

https://www.python.org/dev/peps/pep-0593/

from typing import Annotated, TypeVar, Optional
from pydantic import BaseModel

class _Unset:
    pass

UNSET = _Unset()

class M1:
    a: Optional[int]
    b: Annotated[Optional[int], UNSET]

Or

T = TypeVar('T')
Unset = Annotated[T, UNSET]


class M2:
    a: Optional[int]
    b: Unset[Optional[int]]

Also, note that proper NoneType and EllipsisType types have been added to 3.10 and should be supported by mypy in a future release.

https://bugs.python.org/issue41810

python/typing#684 (comment)

[layday]

What would the value of M1(a=1).b be? If it is actually unset your type checker won't know. I assume the idea with Unset[] is that it would be a type-only descriptor and __get__ would return Optional[int] for instances of M2, meaning that it'd suffer from the same issue as M1.

[PrettyWood]

I see, so this would be for fields that (a) can be omitted but (b) should also not have a default value in a JSON schema. The discussion of Omittable on typing-sig would be of some interest here - if the Omittable semantics could be extended to native classes in addition to TypedDict.

Yes exactly! I also would like this for TypedDict (which I compare with TS interfaces). Thanks for sharing this thread!

[mpkocher]

Looks like the PEP the first version of the PEP has been posted the typing-sig mailing list.

https://mail.python.org/archives/list/typing-sig@python.org/thread/3NJZS7VCHSF54MD465SO7AF3AXGBGEDO/

It looks like the current version of the spec, typing.Required can only be used in the context of TypedDict.

It is an error to use Required[...] in any location that is not an item of a TypedDict:

[PrettyWood]

Thanks for the update! I hope this is something that we'll be able to leverage.
But I'm happy the community is putting some effort into making this easier to write and more explicit, which was the whole purpose of this issue.

[samuelcolvin]

Pipe operator for unions sounds great, I'm keen to support it, including backporting if it's reasonably easy.

---

Here's what I would propose for the rest of this (I'm using the current Optional/Union syntax for clarity, it should have the exact same functionality as |):

class Model(BaseModel):
    a: str  # value required, None not allowed
    b: Optional[str]  # value required, None are allowed
    c: str = None  # value not required, but if it is provided it may not be None
    c2: OptionalDisallowNone[str] = None  # same as c but could satisfy mypy and friends (see discussion below)
    d: Optional[str] = None  # value not required, if it is provided it may be None
    e: Optional[str] = Field(None, omit_default=True)  # optional field with no default value #1270

Regarding #1270 and the desire for fields that can disappear. I think it's a case of "the tail wagging the dog", I don't think this is something you'd ever really want, it's only come up because of trying to match JSONSchema. I think we can achieve this with a kwarg to Field without becoming unpythonic and more major interface changes in v2.

In particular I think reversing the meaning of ... in v2 is a mistake.

Again, I tend towards respecting the silent majority who want to keep it simple, pythonic and intuitive rather than the vocal minority who want to support specific behaviour.

I think the above (except e) is more pythonic and intuitive than what we have now, as well as more powerful.

---

For Annotated, see #2129. I think that covers all the options and the best way forward.

[layday]

Well, you’d then be lying to the type checker about what the value of c is. I think an explicit special Absent type is the way to go here, which would be omitted from the JSON
schema and would not be passable from JSON.

[samuelcolvin]

Well, you’d then be lying to the type checker about what the value of c is.

Good point. But I can't really think of a better solution.

A few options are:

• a clone of Optional which means the same thing to mypy and friends but means "not required but None is not allowed" to pydantic, e.g. c: OptionalDisallowNone[str] = None
• another kwarg to Field like c: Optional[str] = Field(None, allow_none=False)

I'm not clear how Absent works except wrt JSONSchema which is a secondary consideration.

[layday]

I don't know what the usefulness of fobidding None would be aside from JSON import and export. I thought what we wanted to do here was be able to express

{"properties": {"c": {"type": ["string"]}}}

in terms of Pydantic - that is, a property which can be omitted but must not default to None (or any other serialisable value) if omitted. allow_none=False would work if the property accepts None. But what if you have a property which must be a string if provided, can be omitted and does not have a default value? If the answer is it can't be done with Pydantic because Python is not able to represent optional attributes and arguments without defaults, I think that's perfectly acceptable.

[samuelcolvin]

I don't know what the usefulness of fobidding None would be

Me neither, but people want it, so I'm trying to support it. It is precisely because I think it's a relatively rare case that I'm happy for the solution to be somewhat involved/verbose.

But what if you have a property which must be a string if provided, can be omitted and does not have a default value?

Do you mean "does not have a default value on the model instance" or "does not have a default value in the JSON schema"?

As discussed elsewhere in this enormous thread, removing attributes from models is unpythonic and looses typing guarantees. I am absolutely not removing attributes from models. If you want to use your own Absent or Undefined default value, that should be fine, I won't be using it, but I'll try to support it.

If you mean the default (or lack thereof) in JSON schema, we can have sensible defaults then use kwargs for anything else. But that's not what this thread is really about.

[layday]

If you mean the default (or lack thereof) in JSON schema, we can have sensible defaults then use kwargs for anything else. But that's not what this thread is really about.

Further up in this enormous thread I was told that this is what about it's about, which is what led to my suggestion to use Absent or similar to encapsulate the distinction found in JSON. If all people want is to forbid None when constructing a model with no effect on the schema, can't they accomplish this trivially with a validator?

[Bobronium]

Is there a reason for additional None in Field() in case of e?

Can we use just omit_default=True like this, or it would mean something different?

e: Optional[str] = Field(omit_default=True)  # optional field with no default value #1270

[samuelcolvin]

yes, I guess you're right.

[bofm]

Hi. Since we are into this typing stuff, I think it may be interesting to take a look at Clojure’s spec/select. The idea is explained in depth in Rich Hickey’s “Maybe Not” talk. Essentially the concept is to define the model field types and the optionality of the fields separately.

In almost any FastAPI app there are many models for the same data: MyDataIn, MyDataOut, MyDataCreate, MyDataSearch. Usually each model has a subset of fields from the set of fields for MyData domain entity. This is a proliferation of types. What we might want to do instead is to define MyData type without specifying optionality for the fields in the model. The optionality then would be specified separately because it is context dependent. When we create MyData in the system we probably want to specify more fields than when we search MyData by MyData.ID. Same model but different field optionality depending on the context.

[Bobronium]

This is a proliferation of types. What we might want to do instead is to define MyData type without specifying optionality for the fields in the model. The optionality then would be specified separately because it is context dependent. When we create MyData in the system we probably want to specify more fields than when we search MyData by MyData.ID. Same model but different field optionality depending on the context.

I agree. But I can't think of a way to tell a type checker whether field optional or not, depending on context. This might be solved with a plugin, but I'm sure it will still require multiple models.

[koxudaxi]

I write this comment as a user of pydantic and an author of the pydantic PyCharm plugin.

Again, I tend towards respecting the silent majority who want to keep it simple, pythonic and intuitive rather than the vocal minority who want to support specific behaviour.

I respect the policy.
I think we should not increase the cost of learning on pydantic.
And, a unique style affects related tools like the pydantic PyCharm plugin.
For example, I'm working on supporting Annotated on the plugin.
Annotated is a built-in feature of Python which PyCharm supports.
I only write code related to Field() for supporting the feature.
Pythonic is an ecosystem in the community.
It's saving a lot of costs.

Someday, someone will implement a plugin for VSCode or a language server. It may be in Pyright.
I believe the policy will save costs when making it.

[PrettyWood]

Yes changing the use of ... was just a suggestion but it's not a good solution. Setting a Field with dedicated kwarg seems like the most reasonable solution

[PrettyWood]

So in summary we'd go with the "All fields without a default value would be required and all fields with a default value would be optional" rule (except if the default value is Ellipsis, which is pydantic way to represent Required)
To avoid setting a default value to an optional field, just use Field(omit_default=True)
Feels quite right

[PrettyWood]

The goal of this new API is to allow nullable required values and to differentiate "unset value" and None.
This is why in the internal, libraries often use a sentinel like Undefined for pydantic (or MISSING for dataclass or whatever...)

More than once did we have in my company a nullable field where we wanted to know whether or not the value had been set.
"Did the user actually sent None or is it the default value created by the model?"

Behind the scenes the model will always have all the fields that's fore sure!
But we could allow this Undefined value to be set once the model is initialized. Fields with this value would be ignored on dict(), json(), ... And only fields with omit_default=True or something equivalent would be allowed to have Undefined.

Since Undefined is only used behind the scenes, there is no need to expose it or its associated type UndefinedType for type checking. The Field(omit_default=True) is already supported and should do the trick!

So it would mean

class Model(BaseModel):
    required: str
    nullable_required: str | None

    optional_with_default_value: str = 'pika'
    optional_nullable_with_default_value: str | None = None
    optional_without_default_value: str = Field(omit_default=True)  # default is `Undefined` (already the case in `Field()`)
    optional_nullable_without_default_value: str | None = Field(omit_default=True)

    # INVALID as `None` is not valid for type `str`
    invalid_optional: str = None

Am I missing something? 🤔 Thanks for all your reactions by the way!

[koxudaxi]

Am I missing something? 🤔 Thanks for all your reactions by the way!

I think you have resolved all cases!!

I discussed the same theme in koxudaxi/datamodel-code-generator#327 last week.
I wanted to convert JSONShema/OpenAPI to Pydantic Model with strict-nullable.
(The content of link is not important, I share to tell the changes of v2 is needed by users)

Since Undefined is only used behind the scenes, there is no need to expose it or its associated type UndefinedType for type checking. The Field(omit_default=True) is already supported and should do the trick!

model = Model(required='abc', nullable_required=None)  # 1
model.optional_without_default_value  # 2
model.optional_nullable_without_default_value # 3

I have a question. you said no need to expose Undefined.
What can we get value from model.optional_without_default_value and model.optional_nullable_without_default_value? attributes have Undefined or None?
Or, Do we get errors by #1 or #2 or #3?

[PrettyWood]

What I meant is that we don't need to expose the sentinel type for mypy or other type checkers.
But I think raising a custom error or something is dangerous and we're better off using the sentinel in the code like dataclass

from dataclasses import dataclass, field, fields

@dataclass
class M:
    x: str = field(default_factory=lambda: 'q')

print(fields(M)[0].default)
# <dataclasses._MISSING_TYPE object at 0x10d4a3fd0>

I would hence expect all of your examples to be valid 😃
Thinking about it, if we have

model = Model(required='abc', nullable_required=None)
assert model.dict() == {
  'required': 'abc',
  'nullable_required': None,
  'optional_with_default_value': 'pika',
  'optional_nullable_with_default_value': None
}

would we need to add a model.dict(with_undefined=True) or something if we want the real internal state of the model? 🤔

[koxudaxi]

What I meant is that we don't need to expose the sentinel type for mypy or other type checkers.
But I think raising a custom error or something is dangerous and we're better off using the sentinel in the code like dataclass

Oh, I'm sorry I have misread your comment.
I agree 😄

would we need to add a model.dict(with_undefined=True) or something if we want the real internal state of the model? 🤔

I will want the real internal state. But, I don't know to need the option on model.dict(...).
I often use __fields_set__ to know whether the user set value.

We use model.dict(...) when giving the model to other files or databases.
It means the option provides to able to save the complete status to persistence storage or external machine.
However, I conjecture it's an Infrequent use case.

[gwerbin]

It seems like I missed the discussion, but it's probably worth pointing out that there's precedent here in the Attrs and Marshmallow APIs.

[Chasx]

Is there a way to specify an optional but not nullable field? I tried various options. My OpenAPI spec now seems to be correct. I am using FastAPI and in the /docs link the schema does not have a 'required' star next to the property and it also states nullable: false but when writing a test I would expect a 422 to return when I feed it a nulled value, but that is not the case.

test: Optional[float] = Field(nullable=False)

[samuelcolvin]

Is discussion is about a future change, for now use a validator for that.

[lsorber]

Is there a way to represent "non-required non-nullable" fields yet? The docs don't cover that case [1].

[1] https://docs.pydantic.dev/latest/migration/#required-optional-and-nullable-fields

[h0x32]

I don't understand how this works,
i have the following code that doesn't wish to work:

class Model(BaseModel, arbitrary_types_allowed=True):
    UUID: UUID = None


Model(UUID=UUID('12345678123456781234567812345678'))

with error message:

pydantic_core._pydantic_core.ValidationError: 1 validation error for Model UUID Input should be None [type=none_required, input_value=UUID('12345678-1234-5678-1234-567812345678'), input_type=UUID] For further information visit https://errors.pydantic.dev/2.0.3/v/none_required

How is it possible to have this UUID attribute that can be both None and UUID?

[rarestg]

I'm also facing the same issue, but I don't believe I'm overriding any existing types:

Meta.Source
  Input should be None [type=none_required, input_value={'ID': 'c48f8613', 'Name': 'My Test'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.2/v/none_required

I have a dict like so:

You can see the "Meta" dict and the "Source" sub-dict under that.

My Pydantic v2 model is defined as follows:

from typing import List, Optional

from pydantic import BaseModel


class Source(BaseModel):
    ID: Optional[str] = None
    Name: Optional[str] = None

class Meta(BaseModel):
    DataModel: str
    EventType: str
    EventDateTime: Optional[str] = None
    Test: Optional[bool] = None
    Source: Optional[Source] = None
    Destinations: Optional[List[Destination]] = None
...

I've clearly marked Source: Optional[Source] = None but it's telling me it should be None. When I follow the link to https://errors.pydantic.dev/2.2/v/none_required, it tells me this should only happen if I define it as Source: None.

I must be missing something here... right?

[joshy]

@rarestg I am facing exactly the same issue.

[Viicos]

I've clearly marked Source: Optional[Source] = None but it's telling me it should be None.

@rarestg @joshy Yes this is the same issue: in the type declaration (after the :), Source is evaluated to the left Source (not your previously defined Source model) which has a default value of None. So this is equivalent to Source: Optional[None] = None, or if simplified even more: Source: None = None.

More details can be found here: #7327. In the meanwhile, you'll have to adapt your type hint or use aliases

[joshy]

@Viicos Thanks for your answer but how would a possible different type hint look like to solve the problem? Thanks for your time!

[Viicos]

In the case of #2353 (reply in thread) it's a bit hard, as you can't play with the import (e.g. like in #2353 (reply in thread)). You can find my answer here: #7327 (comment) which provides alternatives, but they aren't perfect :/