Support Annotated type hints and extracting Field from Annotated

[JacobHayes]

Change Summary

As discussed elsewhere (1, 2), Pydantic doesn't support typing.Annotated/typing_extensions.Annotated. When using an Annotated field, Pydantic raises:

TypeError: Fields of type "<class 'typing.Annotated'>" are not supported.

This PR adds support for Annotated by simply unwrapping the root type in ModelField. The full Annotated information is still visible with get_type_hints(x, include_extras=True) on the BaseModel subclasses.

I'm not sure if this needs doc updates, perhaps just a small note saying "Annotated type hints are supported"?

Separately, I'm working on a prototype for #2129 in https://github.com/JacobHayes/pydantic-annotated which expands on this functionality. I have a small check (based on the presence of pydantic.typing.Annotated) in there to do the unwrapping until this or similar is merged.

Related issue number

• Decompose Field components into Annotated #2129

Checklist

• Unit tests for the changes exist
• Tests pass on CI and coverage remains at 100%
• Documentation reflects the changes where applicable
• changes/<pull request or issue id>-<github username>.md file added describing change
  (see changes/README.md for details)

[codecov]

Codecov Report

Merging #2147 (3645dbf) into master (78934db) will decrease coverage by 0.02%.
The diff coverage is 98.33%.

@@             Coverage Diff             @@
##            master    #2147      +/-   ##
===========================================
- Coverage   100.00%   99.97%   -0.03%     
===========================================
  Files           23       23              
  Lines         4426     4477      +51     
  Branches       888      903      +15     
===========================================
+ Hits          4426     4476      +50     
- Misses           0        1       +1     

Impacted Files	Coverage Δ
pydantic/typing.py	99.41% <95.65%> (-0.59%)	⬇️
pydantic/fields.py	100.00% <100.00%> (ø)
pydantic/schema.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78934db...3645dbf. Read the comment docs.

[JacobHayes]

This appears to be mostly working except when using a constrained type (gt, et al):

On field "x" the following field constraints are set but not enforced

Looks like I'll need a little more digging into the constraint logic to apply a get_origin or similar.

Tests are passing.

[samuelcolvin]

I think this looks like a great start, but I think before we merge we need:

• Annotated[int, Field(alias='foobar')] to work - I guess we should raise an error if you set the default value by via the positional argument to Field
• decent documentation on how to use Annotated, shouldn;t be that hard

[JacobHayes]

decent documentation on how to use Annotated, shouldn;t be that hard

I'll dig in. Planning to add two parts:

• 1 small comment on the Field Types page that they can be Annotated
• a slightly larger comment on Schema saying Fields can be set in Annotated and some of the error cases

[JacobHayes]

@samuelcolvin any chance we could make typing_extensions a required dependency? Would certainly clean up some code/cases for Annotated, Literal, and maybe some other stuff in typing.py (but certainly not everything, particularly on 3.6 it seems).

[JacobHayes]

Looks like we need to either bump the docs builder to py3.9, install typing_extensions, or I can just inline the example (so it's not run). Any preferences?

I also added a few failing tests around defaults - I need to add:

• error if the Field in Annotated has a default set
• use the rhs default value when Field is in Annotated

[JacobHayes]

I don't think we can (cleanly) "error if the Field in Annotated has a default set" since we also need to support default_factory, unless:

• we use default_factory from lhs Annotated[..., Field(...)] but default from rhs value (more complexity/error cases)
• or, we inspect isinstance(rhs, Callable) to infer default_factory (larger/breaking change that should be separate if desirable, potentially a v2 change)

For now, I'll allow the defaults to be set in the lhs, but let me know if you prefer one of the two above.

edit: Error cases cleaned up a bit with some more refactoring, so I opted for case 1 above.

[JacobHayes]

@samuelcolvin I think this is ready for another look!

There may be a bit of a usage disconnect between default/default_factory that might be improved with this potential v2 change.

[samuelcolvin]

this is looking great, a few things to fix.

[samuelcolvin]

I think get_args is already defined in this file, we should either:

• use that version
• add this logic to that method
• give this a different name

[JacobHayes]

1/3: We need the "Annotated aware" version in all places as the "default".

2: This probably gets tricky without larger "refactor now that typing-extensions is always available" changes, I think I'd have to add this logic to 3 versions of get_args and 2 versions of get_origin (the ones defined above) vs just wrapping whatever is decided above. Let me know if you'd like to just dup the check in those or refactor a bit (presuming #2147 (comment)).

[JacobHayes]

I was able to refactor the get_args/get_origin stuff a bit with typing_extensions, but it's not a panacea:

• those helper funcs aren't available on 3.6 so I kept the old "polyfills"
• I had to remove two tests using a bare Callable (the typing_extensions functions seem to expect the Callable generics to be [...]). Might get just as messy again to add the shim if those are truly valid cases (creating a Model with a bare Callable field hint seemed to still work)
• Looks like this makes the "test compiled without deps" fail (since we now import typing_extensions). Does this need to be tweaked somehow?

I'm content reverting those changes and/or doing something else if you prefer.

[JacobHayes]

Given that ^ messiness, I've pulled those changes (will reserve them for a future typing_extensions PR). How do the latest commits look?

[samuelcolvin]

I think we should move this to install_requires but perhaps that should be a separate PR.

[JacobHayes]

I'm happy to do this in this PR - it would may greatly simplify #2147 (comment). Perhaps I leave a lot of the other cleanup out of this PR though to reduce noise.

[JacobHayes]

I'll leave it out in the sake of getting this PR in. I have a separate branch with partial support I can push after this.

[JacobHayes]

Good on my end - open discussions:

• _get_field_info naming
• custom get_args/get_origin strategy

[thomascobb]

I've added a test for my use case in https://github.com/thomascobb/pydantic/blob/support-annotated-fields/tests/test_decorator.py:

def test_annotated_field_reuse():
    AOne = Annotated[int, Field(description='A long description of `one` we don\'t want to repeat')]

    class Model(BaseModel):
        one: AOne
        two: Annotated[int, Field(description='An optional integer')] = 3

        @classmethod
        @validate_arguments
        def alternative(
            cls,
            one: AOne,
            have_two: Annotated[bool, Field(description='Include two?')],
        ) -> 'Model':
            return cls(one=one, two=2 if have_two else 0)

    assert Model(one=1).dict() == dict(one=1, two=3)
    assert Model.alternative(one=1, have_two=False) == dict(one=1, two=0)

This is failing for a few reasons. First is in the type hint evaluations:

tests/test_decorator.py:134: in Model
    def alternative(
pydantic/decorator.py:61: in validate_arguments
    return validate(func)
pydantic/decorator.py:48: in validate
    vd = ValidatedFunction(_func, config)
pydantic/decorator.py:89: in __init__
    type_hints = get_type_hints(function)
/usr/lib/python3.8/typing.py:1264: in get_type_hints
    value = _eval_type(value, globalns, localns)
/usr/lib/python3.8/typing.py:270: in _eval_type
    return t._evaluate(globalns, localns)
/usr/lib/python3.8/typing.py:518: in _evaluate
    eval(self.__forward_code__, globalns, localns),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   NameError: name 'Model' is not defined

<string>:1: NameError

If I remove the -> 'Model' in the return type of alternative then I get:

tests/test_decorator.py:128: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pydantic/main.py:265: in __new__
    fields[ann_name] = inferred = ModelField.infer(
pydantic/fields.py:346: in infer
    field_info, value = cls._get_field_info(name, annotation, value, config)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

field_name = 'one'
annotation = typing_extensions.Annotated[int, FieldInfo(default=Ellipsis, description="A long description of `one` we don't want to repeat", extra={})]
value = PydanticUndefined, config = <class 'pydantic.main.Config'>

    @staticmethod
    def _get_field_info(
        field_name: str, annotation: Any, value: Any, config: Type['BaseConfig']
    ) -> Tuple[FieldInfo, Any]:
        """
        Get a FieldInfo from a root typing.Annotated annotation, value, or config default.
    
        The FieldInfo may be set in typing.Annotated or the value, but not both. If neither contain
        a FieldInfo, a new one will be created using the config.
    
        :param field_name: name of the field for use in error messages
        :param annotation: a type hint such as `str` or `Annotated[str, Field(..., min_length=5)]`
        :param value: the field's assigned value
        :param config: the model's config object
        :return: the FieldInfo contained in the `annotation`, the value, or a new one from the config.
        """
        field_info_from_config = config.get_field_info(field_name)
    
        field_info = None
        if get_origin(annotation) is Annotated:
            field_infos = [arg for arg in get_args(annotation)[1:] if isinstance(arg, FieldInfo)]
            if len(field_infos) > 1:
                raise ValueError(f'cannot specify multiple `Annotated` `Field`s for {field_name!r}')
            field_info = next(iter(field_infos), None)
            if field_info is not None:
                if field_info.default is not Undefined:
>                   raise ValueError(f'`Field` default cannot be set in `Annotated` for {field_name!r}')
E                   ValueError: `Field` default cannot be set in `Annotated` for 'one'

pydantic/fields.py:320: ValueError

Finally, if I run make lint I get:

tests/test_decorator.py:126:45: F722 syntax error in forward annotation "A long description of `one` we don't want to repeat"
tests/test_decorator.py:126:45: Q003 Change outer quotes to avoid escaping inner quotes
tests/test_decorator.py:130:47: F722 syntax error in forward annotation 'An optional integer'
tests/test_decorator.py:137:57: F722 syntax error in forward annotation 'Include two?'

Any ideas?

[PrettyWood]

@thomascobb The Q003 error is expected. Just change '...we don\'t...' into "...we don't..." (make format doesn't fix this automatically as it doesn't come from black but from flake8 🙄 )
All the other errors come from pyflakes. I already opened a PR and I just checked locally: it fixes all the rest of your issues.
Unfortunately this fix for pyflakes can't be merged yet because other issues have been spotted with it and I still haven't taken the time to dig into it

[JacobHayes]

@PrettyWood

Unfortunately this PR can't be merged because other issues have been spotted and I still haven't taken the time to dig into it

Thanks for looking into things! Do you mind linking to or describing some of the other issues (aside from @thomascobb's) in case I or others have a chance to check them out? If this gets too messy, I'm also happy to split this PR up into 2 parts - the first PR to just not break on unrelated Annotated uses (the first commit) and a second PR to support Field in Annotated (much messier 😁).

@thomascobb
I just pushed a commit that should fix that Field reuse issue - thanks for the find! I admittedly don't grok the distinction between Ellipsis and Undefined (anything aside from printing nicety?), but the FieldInfo mutation definitely adds a little complexity.

E NameError: name 'Model' is not defined

This part is unrelated to this PR/doesn't work on master unfortunately either.

broken minimum test_model_forward_ref

def test_model_forward_ref():
    from pydantic.decorator import validate_arguments

    class Model(BaseModel):
        one: int

        @classmethod
        @validate_arguments
        def alternative(cls, one) -> 'Model':  # still fails
            return cls(one=one)

I think it's due to the alternative method being created and validate_arguments called before Model is created. Perhaps there's a way to keep validate_arguments lazy here and do something in Model.update_forward_refs(), but in the short term you might have luck defining the function outside the class body.

working test_model_forward_ref

def test_model_forward_ref():
    from pydantic.decorator import validate_arguments

    class Model(BaseModel):
        one: int

    @classmethod
    @validate_arguments
    def alternative(cls, one) -> Model:
        return cls(one=one)

    Model.alternative = alternative

[PrettyWood]

The PR that can't be merged yet is the fix I made for pyflakes (I updated my message above to be more explicit. Sorry if this wasn't clear)

[thomascobb]

I just pushed a commit that should fix that Field reuse issue - thanks for the find! I admittedly don't grok the distinction between Ellipsis and Undefined (anything aside from printing nicety?), but the FieldInfo mutation definitely adds a little complexity.

Thanks, that fixes it

This part is unrelated to this PR/doesn't work on master unfortunately either.

Good point. The following works though, and it probably more correct...

from typing import TypeVar

def test_annotated_field_reuse():
    T = TypeVar("T")
    AOne = Annotated[int, Field(description='A long description of `one` we don\'t want to repeat')]

    class Model(BaseModel):
        one: AOne
        two: Annotated[int, Field(description='An optional integer')] = 3

        @classmethod
        @validate_arguments
        def alternative(
            cls: T,
            one: AOne,
            have_two: Annotated[bool, Field(description='Include two?')],
        ) -> T:
            return cls(one=one, two=2 if have_two else 0)

    assert Model(one=1).dict() == dict(one=1, two=3)
    assert Model.alternative(one=1, have_two=False) == dict(one=1, two=0)

[samuelcolvin]

I'm a bit unclear on this.

Is there anything (except conflicts which are trivial) which is blocking this? I would be keen to get it merged asap to include in v1.8 if possible.

[JacobHayes]

No blockers I'm aware of, just another round of reviews. I'll push a conflict fix for reqs in a min.

[samuelcolvin]

this is clever, even if it's sad we have to do it. 🙃

[samuelcolvin]

I think this is okay, but as a piece of feedback, it's generally a good idea to include a simple test that demonstrates the basic behaviour at the top of test file. All this parametrize stuff is very clever but it's not that easy to read or reason with when things break.

You could also use pytest.importorskip here to avoid lots of skipif, or just pytestmark = ....