Annotated and Optional get_type_hints buggy interaction

[med2277]

BPO	46195
Nosy	@gvanrossum, @JelleZijlstra, @sobolevn, @Fidget-Spinner
PRs	bpo-46195: Do not add Optional in get_type_hints #30304

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-02.05:31:58.542>
created_at = <Date 2021-12-29.18:38:47.765>
labels = ['type-bug', '3.9', '3.10', '3.11']
title = 'Annotated and Optional get_type_hints buggy interaction'
updated_at = <Date 2022-03-02.05:31:58.542>
user = 'https://bugs.python.org/med2277'

bugs.python.org fields:

activity = <Date 2022-03-02.05:31:58.542>
actor = 'JelleZijlstra'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-02.05:31:58.542>
closer = 'JelleZijlstra'
components = []
creation = <Date 2021-12-29.18:38:47.765>
creator = 'med2277'
dependencies = []
files = []
hgrepos = []
issue_num = 46195
keywords = ['patch']
message_count = 8.0
messages = ['409319', '409325', '409340', '409341', '409357', '409375', '414328', '414329']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'JelleZijlstra', 'sobolevn', 'kj', 'med2277']
pr_nums = ['30304']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46195'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[med2277]

This is two closely related issues with get_type_hints on an optional annotated member. I'm testing with Annotated/get_type_hints from typing extensions on 3.8 and assuming they're backport equivalent to current behavior.

The first issue is get_type_hints has inconsistent behavior depending on whether annotation comes from a function with a None default or an attribute with a None default.

class Foo:
    def __init__(
        self,
        x: 
            Annotated[Optional[str], "doc string"] = None,
    ):
        ...

class Foo2:
    x: Annotated[Optional[str], "doc string"] = None

get_type_hints(Foo.__init__) 
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}

get_type_hints(Foo2)
# {'x': typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string']}

Attributes with a None default are not wrapped by get_type_hints, but function parameters. Which of the two behaviors is correct I don't know, but I'd expect the two to be equivalent annotations.

The second issue is for function arguments with a None default the optional wrapper happens even if the type inside annotated already has optional. Example,

from typing_extensions import Annotated, get_type_hints

class Foo:
    def __init__(
        self,
        x: 
            Annotated[Optional[str], "doc string"] = None,
    ):
        ...

get_type_hints(Foo.__init__, include_extras=True)
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}

For Annotated types I would expect any type rules like wrapping to apply only to the first argument and not the entire annotation. I mainly ran into this for a runtime type introspection library (similar in spirit to pydantic).

As a note include_extras being True or False while it changes type is an issue in either case. With include_extras as False the Annotated goes away, but the type still gets double wrapped as an Optional.

[gvanrossum]

Could you try with 3.10 and the stdlib typing.Annotated please? There might
be changes (in the past a default of None automatically caused an Optional
to be added, but we changed our minds.--
--Guido (mobile)

[sobolevn]

I can verify that this happens on 3.10 and main branches:

from typing import Annotated, Optional, get_type_hints

class Foo:
    def __init__(self, x: Annotated[Optional[str], "d"] = None): ...

class Foo2:
    x: Annotated[Optional[str], "d"] = None

print(get_type_hints(Foo.__init__, include_extras=False))  # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo2, include_extras=False))  # ok
# {'x': typing.Optional[str]}

print(get_type_hints(Foo.__init__, include_extras=True))  # not ok?
# {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}
print(get_type_hints(Foo2, include_extras=True))  # ok
# {'x': typing.Annotated[typing.Optional[str], 'd']}

Notice that 3rd case does not look correct: {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}

In my opinion it should be {'x': typing.Annotated[typing.Optional[str], 'd']}

I will look into it! :)

[sobolevn]

And on 3.9 as well.

[sobolevn]

As Guido said, the root cause of this problem is because None default automatically adds Optional to the resulting type.

Source:

cpython/Lib/typing.py

Lines 1854 to 1856 in 8d7644f

	value = _eval_type(value, globalns, localns)
	if name in defaults and defaults[name] is None:
	value = Optional[value]

So, what happens there:

• correct `value` is passed to `_eval_type`, correct result `typing.Annotated[typing.Optional[str], 'd']` is returned at this point
• then `if name in defaults and defaults[name] is None:` adds extra `Optional` annotation on top of `Annotated`

in the past a default of None automatically caused an Optional
to be added, but we changed our minds

Guido, are you talking about python/typing#275 ?

Now all type-checkers (AFAIK) support something similar to --no-implicit-optional mode.

Having this in mind, I see different solutions to the current problem:

1. Remove Optional inference with None default. This is going to be a some-what breaking change. The only positive side of this is that we can really simplify our code (mainly because the other solution is to complicate our code even more).
2. Or we can change this place to explicitly check for Annotated type and its internal type. This should be the easiest to write and backport. But, it still has some complexity to it. I think that this is a better solution: we don't break existing behavior, change is local and pretty trivial.

Also caused by this:

• https://bugs.python.org/issue42921
• https://bugs.python.org/issue42288

[gvanrossum]

Yes, we changed PEP-484 in python/peps#689.

So get_type_hints() should follow suit.

[JelleZijlstra]

New changeset 20a1c8e by Nikita Sobolev in branch 'main':
bpo-46195: Do not add Optional in get_type_hints (GH-30304)
20a1c8e

[JelleZijlstra]

This is now fixed in 3.11, but we'll leave 3.10 and 3.9 alone. Thanks for your bug report!