[typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11

[vstinner]

BPO	38605
Nosy	@gvanrossum, @warsaw, @brettcannon, @terryjreedy, @ericvsmith, @jwilk, @stevendaprano, @methane, @ambv, @serhiy-storchaka, @vedgar, @ilevkivskyi, @asottile, @pablogsal, @miss-islington, @isidentical, @gousaiyang, @AlexWaygood
PRs	bpo-38605: Make postponed evaluation of annotations default #20434 bpo-38605: bump the magic number for 'annotations' future #22630 bpo-38605: Update "Future statements" docs since PEP 563 is always enabled #25236 bpo-38605: Revert making 'from __future__ import annotations' the default #25490 bpo-38605: Set the release for __future__.annotations to 3.11 #25596 bpo-38605: Update __future__ module doc as annotations is now "mandatory in 3.11" #25602

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-27.16:07:10.414>
labels = ['type-bug', '3.10', 'expert-unicode']
title = '[typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11'
updated_at = <Date 2022-02-07.11:12:54.549>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-02-07.11:12:54.549>
actor = 'jwilk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2019-10-27.16:07:10.414>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38605
keywords = ['patch']
message_count = 40.0
messages = ['355476', '355495', '355506', '355523', '355546', '360830', '366634', '366635', '369269', '369301', '369303', '370000', '370020', '370028', '370031', '370035', '370037', '370188', '370214', '370220', '370232', '375177', '377783', '378133', '378134', '378135', '378139', '378374', '378416', '390399', '391512', '391515', '391700', '391732', '391755', '391869', '394114', '394115', '404129', '404130']
nosy_count = 18.0
nosy_names = ['gvanrossum', 'barry', 'brett.cannon', 'terry.reedy', 'eric.smith', 'jwilk', 'steven.daprano', 'methane', 'lukasz.langa', 'serhiy.storchaka', 'veky', 'levkivskyi', 'Anthony Sottile', 'pablogsal', 'miss-islington', 'BTaskaya', 'gousaiyang', 'AlexWaygood']
pr_nums = ['20434', '22630', '25236', '25490', '25596', '25602']
priority = 'normal'
resolution = 'postponed'
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38605'
versions = ['Python 3.10']

[vstinner]

The PEP-563: Postponed evaluation of annotations was introduced an opt-in feature using "from __future__ import annotations". It is scheduled to become the default in Python 4.0.

I would prefer to limit the number of incompatible changes in Python 4.0: it should just a "regular" release, with a regular number of incompatible changes. The version number change is going to cause enough troubles...

Would it be possible possible to enable postponed evaluation of annotations either before or after Python 4.0? For example, can we imagine to enable it by default in Python 3.9? If not, what about Python 3.10?

See also the PEP-608 (Coordinated Python release) and bpo-38604 (Schedule Py_UNICODE API removal).

[ilevkivskyi]

IMO 3.10 would be better, since 3.9 would be too soon (it would be like a schedule for a normal deprecation).

Also if we are really doing this, I think it is better to announce this soon.

Also we should try to fix relevant issues related to string annotations (in typing and dataclasses), like python/typing#508, python/typing#574, https://bugs.python.org/issue37838, https://bugs.python.org/issue34776 and https://bugs.python.org/issue37948.

[gvanrossum]

We never should have mentioned 4.0 as the target date to make this the
default (and only) behavior -- who knows whether there will ever even be a
Python 4.0? Even 3.10 might be on the early side (assuming we'll switch to
a year-long release cycle per PEP-602 -- we will then make deprecations in
general take two release cycles).

I do agree that we should start the process of deprecating the
non-future behavior here in 3.9. I know of one project with a private
fork of Python (for other reasons) that has made this default.

[vstinner]

We never should have mentioned 4.0 as the target date to make this the
default (and only) behavior

I am fine with modifying __future__ documentation to only modify the "Mandatory" column to remove Python 4.0, and then close this issue:
https://docs.python.org/dev/library/__future__.html

--

But I like the "PEP-563: Postponed evaluation of annotations", IMHO it would be nice to get it as the default behavior :-) It's just a matter of properly organize the transition ;-)

[gvanrossum]

You can bring the deprecation schedule up on discourse or python-dev so more folks can let us know whether they'd be okay with 3.9 or 3.10.

[vstinner]

I started a thread on typing-sig:
https://mail.python.org/archives/list/typing-sig@python.org/thread/JG4IWHO3TZORNESR6ARXFL6OUZU2BEBX/

[brettcannon]

I personally like 3.10 as the target as that means users had at least 3 years to move to move over. Plus we can put a warning in the What's New for 3.9 about our plans for 3.10.

[vstinner]

This issue has been discussed during the Language Summit. A quick poll showed that the majority is in favor of changing the default in Python 3.9.

Lukasz proposed a PEP update to propose to switch the default in Python 3.9:
python/peps#1371

For me, the unclear part is which projects would be impacted if the default changes?

Someone mentioned attrs, but it seems like attrs is fine:
python-attrs/attrs#288 (comment)

In term of workflow, I would _prefer_ to get such incompatible in the very beginning of a devcycle, rather than just before the feature freeze. But I don't think that it's a blocker issue. Technically, changes are allowed until 3.9.0 beta1. Moreover, Lukasz is the 3.9 release manager, the author of the PEP-563 and he is in favor of changing the default in 3.9 :-)

[gvanrossum]

Too bad nobody took any action here after the positive outcome of the discussion at the summit.

[vstinner]

Too bad nobody took any action here after the positive outcome of the discussion at the summit.

I didn't understand it this way. I understood that some people were not 100% comfortable to target 3.9. The question was 3.9 or 3.10. Since the release cycle is now shorter (1 years), only targeting 3.10 is not a big deal ;-)

[ambv]

We'll make this an announced 3.10 feature early on. The discussion at the Summit wasn't as clear cut to me: 35% of participants would rather see this default later than 3.9.

[vstinner]

Is there anyone interested to implement this change in Python 3.10?

[isidentical]

I opened the PR 20434 as draft, but from what I understand, there is going to be some breakage (on our test suite). I'll try to narrow it down (currently ~4 tests instead of ~20) but I dont want to prevent anyone else from working on this, so feel free to ignore my PR if you have a working test suite with a low breakage level.

[isidentical]

After trying to complete a patch, there are a few issues that immediately showed itself (and this might lead to not to do this in 3.10, I dont know);

First one is double-forward-ref, which is usage of string-annotations when there is postponed evaluatation of annotations:
>>> import typing
>>> from __future__ import annotations
>>> def x(a: 'int'): pass
... 
>>> typing.get_type_hints(x)
{'a': ForwardRef('int')}

If we make annoatations feature default, this would be default behavior. The solution would be a workaround to the compiler;
static int
compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
{
    if (annotation->kind == Constant_kind && PyUnicode_CheckExact(annotation->v.Constant.value)) {
        PyObject *text = annotation->v.Constant.value;
        Py_INCREF(text);
        ADDOP_LOAD_CONST_NEW(c, text);
    } else {
        ADDOP_LOAD_CONST_NEW(c, _PyAST_ExprAsUnicode(annotation));
    }
    return 1;
}
But I am not sure if this is too silly or not. 

The second problem is inspect.signature. If we don't resolve annotations there and continue it is definitely going to break some code. If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

The third problem is various dataclass hacks. Like _type_{field.name} etc. annotations and how ClassVar/InitVar parsed.

There are also some little parts that need to change.

Any thoughts on these issues?

[vstinner]

The second problem is inspect.signature. If we don't resolve annotations there and continue it is definitely going to break some code.

Would you mind to elaborate why would it break some code? Consumers of annotations should already be prepared to get directly types or strings, no?

If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

I expect that resolving has an impact on performance, whereas the caller may not use annotations at all but only cares of the number of parameters or their name.

It would be resonable to not resolve annotations in signature() by default. If someone cares, maybe a new parameter can be added to resolve annotations?

[gvanrossum]

• Double forward ref: IMO this can be resolved in the get_type_hints() functions. (Łukasz do you agree?)

• inspect.signature(): Maybe this could switch to using typing.get_type_hints()? Then again if performance is important here maybe we cannot change anything.

• dataclasses hacks: these should just be resolved.

Another thought: maybe some of these issues can be considered bugs in 3.9 as well, and we should fix them there too? That might help us decide the right path forward. After all we should really encourage people to start using from __future__ import annotations in their code, to help them get ready for these issues in 3.10.

Final thought: I know at least the Dropbox client team, and possibly also Instagram, has already turned on from __future__ import annotations by default in their local fork of Python. Maybe we can ask them if they ever felt the need to change inspect.signature or typing.get_type_hints.

[ericvsmith]

To my knowledge, dataclasses works with from __future__ import annotations. If there are specific examples of problems, I'd like to hear about it: please open a separate issue.

There is a hack (discussed at PyCon 2018 with all of the relevant players) where it avoids importing typing to look at typing.ClassVar, but I think that code is all correct. Maybe I should just bite the bullet and import typing, since I believe importing it is faster than it used to be.

[isidentical]

From now on, should typing.get_type_hints automatically resolve arguments too? An example would be this;

import typing
T = typing.TypeVar("T")
class Loop(typing.Generic[T]):
    subloop: typing.Final["Loop[int]"]
print(typing.get_type_hints(Loop))
>>> {'subloop': typing.Final[__main__.Loop[int]]}
If we run the same code under future annotations
>>> {'subloop': typing.Final[ForwardRef('Loop[int]')]}

[gvanrossum]

I think in general it is more insightful to discuss the behavior of get_type_hints() given specific things in annotations.

We generally don't write forward refs inside forward refs, like "SomeClass['int']". So maybe that code was wrong? Where did you find it?

[isidentical]

An example would be this

cpython/Lib/test/test_typing.py

Lines 2744 to 2745 in 24bddc1

	class Loop:
	attr: Final['Loop']

. Either I can change tests in order to reflect now everything is a forward ref by default
class Loop:
attr: Final['Loop']
to
class Loop:
attr: Final[Loop]
or resolve everything on get_type_hints.

[gvanrossum]

There will still be a lot of code written that way, because people need compatibility with earlier versions of Python. So I think it should be fixed in get_type_hints().

[terryjreedy]

bpo-41314 changed the __future__ annotations default version to 3.10. 3.10.0a1 is scheduled for next Oct 5, less than 2 months from now. It would be good if PR 20434 were merged before that.

[serhiy-storchaka]

test_grammar and test_pydoc became failed on my machine until I removed all .pyc files. We need to bump the magic number for .pyc files.

[gvanrossum]

New changeset 22220ae by Batuhan Taskaya in branch 'master':
bpo-38605: bump the magic number for 'annotations' future (bpo-22630)
22220ae

[miss-islington]

New changeset 1be456a by Saiyang Gou in branch 'master':
bpo-38605: Update "Future statements" docs since PEP-563 is always enabled (GH-25236)
1be456a

[pablogsal]

New changeset b0544ba by Pablo Galindo in branch 'master':
bpo-38605: Revert making 'from __future__ import annotations' the default (GH-25490)
b0544ba

[vstinner]

New changeset b0544ba by Pablo Galindo in branch 'master':

Copy of the NEWS entry:

Revert making from __future__ import annotations the default. This follows
the Steering Council decision to postpone PEP-563 changes to at least Python
3.11. See the original email for more information regarding the decision:
https://mail.python.org/archives/list/python-dev@python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/.
Patch by Pablo Galindo.

[gvanrossum]

I haven't followed this precisely, but I recall that when we first made PEP-563 the default, we had to fix a number of bugs in various library modules (e.g. inspect) that were only apparent when from __future__ import annotations was used. I hope we're not throwing away those bugfixes. Hopefully Batuhan has a recollection of what I am thinking of, there was some significant delay while we figured out what to do about some of these.

[isidentical]

Hopefully Batuhan has a recollection of what I am thinking of, there was some significant delay while we figured out what to do about some of these.

The major one that I'd recall is that inspect.signature() just uses whatever is in __annotations__ instead of resolving those. Now that __future__.annotations is not the default one, we can add a new option named 'resolve_annotations' and call typing.get_type_hints when activated. Here is a quick demo;

from __future__ import annotations

import inspect

def foo(a: int, b: int) -> str:
    ...

def _get_annotations(func, **signature_opts):
    signature = inspect.signature(func, **signature_opts)
    return {
        param.name: param.annotation
        for param in signature.parameters.values()  
    }

print('bare: ', _get_annotations(foo))
print('annotations resolved: ', _get_annotations(foo, resolve_annotations=True))

bare: {'a': 'int', 'b': 'int'}
annotations resolved: {'a': <class 'int'>, 'b': <class 'int'>}

This would be a clear feature for both PEP-563 users + people who are still using string annotations mixed with normal ones. What do you think Guido?

[gvanrossum]

I don't like adding a flag. In the PEP-563-by-default world we didn't need a flag. Perhaps once Larry implements inspect.get_annotations() (bpo-43817) we can make inspect.signature() always call that?

[miss-islington]

New changeset f84f1b5 by Saiyang Gou in branch 'master':
bpo-38605: Update future module doc as annotations is now "mandatory in 3.11" (GH-25602)
f84f1b5

[vedgar]

May I ask, is this going forward? I installed 3.11-dev, it's not there (though __future__ claims it should be). I understand if it isn't done yet, just want to know if there's risk it will be postponed again (or even given up).

[isidentical]

May I ask, is this going forward? I installed 3.11-dev, it's not there (though __future__ claims it should be). I understand if it isn't done yet, just want to know if there's risk it will be postponed again (or even given up).

We are still waiting a ruling on PEP-649. If it gets rejected, and no more ideas arises (before beta cut for 3.11), I guess we could move on and resolve this issue again.

[stevendaprano]

Now that we're in 3.11, people are starting to notice that stringy annotations are not the default (see bpo-45499 for example).

What can we do to get PEP-649 moving forward?

[serhiy-storchaka]

I think it would help if we could enable some future feature globally by command line option or environment variable, without modifying all source files. It would allow users to quickly test their code base for compatibility with future changes. The problem currently is that nobody bothers to add "from __future__ import ...", so we have surprises every time when try to make it by default.

A tool which automatically adds or removes "from __future__ import ..." in files could help too.

[vstinner]

There are now multiple PEPs proposed to handle this problem differently.

PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11

The SC asked to hold this for now. So I prefer to close the issue. Please open a new issue once the SC took a decision on this topic.