Field(default=None) overrides a field's type annotation as Optional

[lsorber]

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

            pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /Users/.../miniconda3/envs/.../lib/python3.8/site-packages/pydantic
               python version: 3.8.3 (default, Jul  2 2020, 11:26:31)  [Clang 10.0.0 ]
                     platform: macOS-10.15.6-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions', 'email-validator']

Pydantic internally changes a field's type annotation from X to Optional[X] when applying Field(default=None).

Arguably, this is a bug because it is no longer possible to choose between (the semantically different) foo: int = Field(default=None) and foo: Optional[int] = Field(default=None). To me, the former reads as "foo is a non-required (default=None) field that can take on non-null int values (int)", while the latter reads as "foo is a non-required (default=None) field that can take on nullable int values (Optional[int])".

Consider the following example:

from pydantic import BaseModel, Field

class Foo(BaseModel):
    foo: int = Field(default=None)

Foo()  # OK: Don't expect to see a ValidationError because only supplied fields are validated.
Foo(foo=None)  # Not OK: Expect to see a ValidationError because `foo` is not an `int`, but no error raised.

# Why did this happen? Pydantic overrides the type of `foo` from `int` to `Optional[int]`:
print(Foo.__fields__["foo"])
# ModelField(name='foo', type=Optional[int], required=False, default=None)

One workaround is to use a different sentinel value (see below). Is there a better way to specify "non-required non-nullable fields of a given type"?

from pydantic import BaseModel, Field

_missing = object()

class Foo(BaseModel):
    foo: int = Field(default=_missing)

[PrettyWood]

Hello @lsorber
I agree the behaviour is not perfectly explicit but changing it will probably need to wait for v2. In pydantic code Undefined sentinel value is used for default value and I reckon it could be a good idea to use this more generally in the API and clearly separate Undefined (like not set) and None. It would avoid needing the trick you use.
Anyway I guess this topic can be discussed in #990 or in a new issue.
I close this ticket as it is not a bug but the expected behaviour (at least at the moment ;D)

[lsorber]

Understood @PrettyWood. Could you clarify why this is the expected behaviour in v1.6.1 though? To me at least this behaviour is not expected based on the documentation.

I suspect that this is the expected behaviour because "that's the way it's currently implemented". If that's the case I'm fine with that, but the reason I ask is that I'd like to understand if I misunderstood Pydantic's rule set.

[PrettyWood]

this is the expected behaviour because "that's the way it's currently implemented".

Exactly!

There is even a test to make sure the implemented behaviour here works. This is why I don't call it a bug. We can discuss the why and the possible improvements that could be made but for this @samuelcolvin will probably be better suited to answer ;)