PEP 604 NewType

[wyfo]

BPO	44353
Nosy	@gvanrossum, @ambv, @serhiy-storchaka, @ilevkivskyi, @JelleZijlstra, @pablogsal, @miss-islington, @uriyyo, @wyfo, @domdfcoding, @Fidget-Spinner, @dlax, @AlexWaygood
PRs	bpo-44353: Refactor typing.NewType into callable class #27250 [3.10] bpo-44353: Refactor typing.NewType into callable class (GH-27250) #27258 bpo-44353: Add test to cover __or__ of two typing.NewType #27259 [3.10] bpo-44353: Add test to cover __or__ of two NewType (GH-27259) #27261 bpo-44353: Implement typing.NewType __call__ method in C #27262 bpo-44353: Add tests to cover typing.NewType pickling #27302 bpo-44353: Fix memory leak introduced by #27262 #27305 bpo-44353: Expand NewType tests for complex __qualname__. #27311 bpo-34963: Make the repr of the typing.NewType() result more meaningful. #9951 bpo-44353: Document that typing.NewType is now a class #27319 [3.10] bpo-44353: Document that typing.NewType is now a class (GH-27319) #27321 [3.10] bpo-44353: Expand NewType tests for complex __qualname__ (GH-27311) #27326 [3.10] bpo-44353: Improve tests covering typing.NewType pickling (GH-27302) #27328 bpo-44353: Correct docstring for NewType #29785 [3.10] bpo-44353: Correct docstring for NewType (GH-29785) #29796
Superseder	bpo-44642: Union of a type and the typing module function

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-24.15:29:29.597>
created_at = <Date 2021-06-08.20:21:48.412>
labels = ['3.10', '3.11']
title = 'PEP 604 NewType'
updated_at = <Date 2021-11-26.14:58:16.927>
user = 'https://github.com/wyfo'

bugs.python.org fields:

activity = <Date 2021-11-26.14:58:16.927>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-24.15:29:29.597>
closer = 'kj'
components = []
creation = <Date 2021-06-08.20:21:48.412>
creator = 'joperez'
dependencies = []
files = []
hgrepos = []
issue_num = 44353
keywords = ['patch']
message_count = 44.0
messages = ['395359', '395512', '395519', '395538', '397299', '397300', '397825', '397826', '397834', '397835', '397838', '397840', '397842', '397847', '397848', '397849', '397869', '397876', '397877', '397878', '397879', '397880', '397881', '397886', '398005', '398006', '398048', '398057', '398080', '398120', '398123', '398129', '398136', '398138', '398139', '398147', '398148', '398149', '398150', '398151', '398152', '398164', '407026', '407055']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'lukasz.langa', 'serhiy.storchaka', 'levkivskyi', 'JelleZijlstra', 'pablogsal', 'miss-islington', 'uriyyo', 'joperez', 'domdfcoding', 'kj', 'dlax', 'AlexWaygood']
pr_nums = ['27250', '27258', '27259', '27261', '27262', '27302', '27305', '27311', '9951', '27319', '27321', '27326', '27328', '29785', '29796']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '44642'
type = None
url = 'https://bugs.python.org/issue44353'
versions = ['Python 3.10', 'Python 3.11']

[wyfo]

typing.NewType doesn't support PEP-604.

>>> import typing
>>> A = typing.NewType("A", int)
>>> B = typing.NewType("B", str)
>>> A | B
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'function' and 'function'

[domdfcoding]

It is possible to implement NewType to return a class rather than a function as it does currently. However, the class version uses substantially more memory (1072 bytes vs 144 bytes with sys.getsizeof) and NewType is supposed to have almost no runtime overhead.

[uriyyo]

What about to return callable object instead of function from a typing.NewType?

It will look something like this:

class _NewType:
    __slots__ = ('__name__', '__supertype__')

    def __init__(self, name, supertype):
        self.__name__ = name
        self.__supertype__ = supertype

    def __call__(self, val):
        return val

    def __or__(self, other):
        return Union[self, other]

    def __ror__(self, other):
        return Union[other, self]



def NewType(name, tp):
    """NewType creates simple unique types with almost zero
    runtime overhead. NewType(name, tp) is considered a subtype of tp
    by static type checkers. At runtime, NewType(name, tp) returns
    a dummy callable object that simply returns its argument. Usage::

        UserId = NewType('UserId', int)

        def name_by_id(user_id: UserId) -> str:
            ...

        UserId('user')          # Fails type check

        name_by_id(42)          # Fails type check
        name_by_id(UserId(42))  # OK

        num = UserId(5) + 1     # type: int
    """
    return _NewType(name, tp)

With such implementation __or__ will be available for a NewType and actually because of slots size of object will be 48 bytes (with sys.getsizeof) which is less than current 144 bytes (if memory is important).

[JelleZijlstra]

python/typing#746 has some previous discussion of implementing NewType as a class (motivated by __repr__), including benchmarks.

[uriyyo]

Jelle I can try to implement it in C to see if it will give better performance.

[Fidget-Spinner]

@yurii, I would like to caution against a C accelerator for the typing module. My reasoning follows:

1. The maintenance burden is higher. typing is already somewhat complex in Python (lots of MRO/metaclass wizardry). A C module would require knowledge of the C API.

2. Following from 1., I fear it may raise the barrier for contributions. This is purely anecdotal but I think there are more contributors who know Python than Python + C API.

3. C API code is much more error-prone than Python code.

4. It's very hard to find available reviewers for typing-related C changes.

5. Backports become harder due to point 3. and 4. Also C code needs much more scrutiny. If we cause some obscure bug in Python, it raises an exception; in C it potentially segfaults and kills the interpreter.

6. Third-party monkey-patching becomes harder.

Unfortunately, I can't offer a good solution to this issue at the moment either.

[uriyyo]

@ken thanks for pointing this out.

The only solution which I see is to convert NewType into callable class but it will lead to performance degradation.

@ken what do you think about that?

[gvanrossum]

Do we have to solve it? Using from __future__ import annotations it should work, at least when used as an annotation. Or is that too inconvenient or inconsistent?

[JelleZijlstra]

Using from __future__ import annotations it should work

Not in type aliases or generic bases. (And if Larry's co_annotations PEP is accepted, it would break in normal annotations too.) I'd prefer to get rid of this sort of subtle difference where the obvious way to write a type only works in some contexts, though I realize that's not possible in all cases.

That means I think we should move forward with making NewTypes into instances with __call__ instead of functions, though according to my benchmarks in the typing issue that will incur a small performance penalty.

[uriyyo]

I opened PR with refactored typing.NewType into callable class.

Also I have added __module__ attr to typing.NewType so it will become pickable.

[gvanrossum]

Jelle, can you post more details about your benchmark?