typing.Literal[True] is implicitly converted to typing.Literal[1]

[vanburgerberg]

BPO	45679
Nosy	@gvanrossum, @rhettinger, @serhiy-storchaka, @miss-islington, @sobolevn, @Fidget-Spinner, @vanburgerberg
PRs	bpo-45679: Do not cache typing.Literal with more than one type arg #29333 bpo-45679: Fix caching of multi-value typing.Literal #29334 bpo-45679: add tuple tests with lru_cache to test_functools #29339 [3.10] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) #29340 [3.9] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) #29342

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-11-03.09:45:01.789>
created_at = <Date 2021-10-30.15:33:26.761>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'typing.Literal[True] is implicitly converted to typing.Literal[1]'
updated_at = <Date 2021-11-05.19:52:54.863>
user = 'https://github.com/vanburgerberg'

bugs.python.org fields:

activity = <Date 2021-11-05.19:52:54.863>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-03.09:45:01.789>
closer = 'kj'
components = ['Library (Lib)']
creation = <Date 2021-10-30.15:33:26.761>
creator = 'vanburgerberg'
dependencies = []
files = []
hgrepos = []
issue_num = 45679
keywords = ['patch']
message_count = 9.0
messages = ['405371', '405375', '405376', '405377', '405394', '405395', '405592', '405598', '405825']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'rhettinger', 'serhiy.storchaka', 'miss-islington', 'sobolevn', 'kj', 'vanburgerberg']
pr_nums = ['29333', '29334', '29339', '29340', '29342']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45679'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[vanburgerberg]

When you create Literal[1] annotation and then create Literal[True] annotation, in the second case you will actually get Literal[1] instead. This is happening because typing performs caching of the outcome of parameterizing generics and hash(True) is equal to hash(1). I think this behavior is incorrect and may lead to unexpected results.

Why is this inaccuracy important?
Consider the following example:

from typing import Literal

SomeUsefulAlias = Literal[1, "abc"]

def func(arg: Literal[True, "abc"]):
    if arg is not True:
        arg.capitalize()

If we look at func.__annotations__["arg"], we will see Literal[1, 'abc']. According to the new annotation, we can pass the value 1 to func, and this will lead to an attribute error, as you've already understood. Thus, proper runtime type checking cannot be performed.

[sobolevn]

This happens to all lru_cache calls, where we deal with container types.

Example:

from functools import lru_cache

@lru_cache(typed=True)
def test(arg):
    return str(arg)

print(test((1, 'a')))  # (1, 'a')
print(test((True, 'a')))  # (1, 'a')

Moveover, this behavior is not tested in test_functools: https://github.com/python/cpython/blame/aae18a17401dc36917c0f64f971d60ab1a5b477e/Lib/test/test_functools.py#L1482-L1496

Original issue: https://bugs.python.org/issue13227
Original commit: cd9fdfd

[sobolevn]

I see several ways of solving this:

1. Make lru_cache(typed=True) to look into iterable internals. This might be slow. And very slow for large iterables. Maybe even infinite for infinite ones
2. Adapt Literal / possibly other types to be treated differently. Because, most of the time, Literal take a small amount of args

I am going to send a PR with the second option.

[serhiy-storchaka]

I agree. lru_cache(typed=True) itself should not look into iterable internals. It would be not only slow, but a change of semantic.

The simplest way to solve this issue is to remove caching of __getitem__(). The more sophisticated way is to move caching to lower level and apply it to a function with a var-positional parameter.

[serhiy-storchaka]

New changeset 634984d by Serhiy Storchaka in branch 'main':
bpo-45679: Fix caching of multi-value typing.Literal (GH-29334)
634984d

[miss-islington]

New changeset 3997f3c by Miss Islington (bot) in branch '3.10':
bpo-45679: Fix caching of multi-value typing.Literal (GH-29334)
3997f3c

[Fidget-Spinner]

New changeset bbcf06b by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) (GH-29342)
bbcf06b

[Fidget-Spinner]

Closing this issue as the bug has been solved.

@nikita could you please open a new issue for your tests PR and link to that instead? It seems like an enhancement over the current test suite for lru_cache. Thanks!

[rhettinger]

New changeset 60b5333 by Nikita Sobolev in branch 'main':
bpo-45679: add tuple tests with lru_cache to test_functools (GH-29339)
60b5333