Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11 #82786

Closed
vstinner opened this issue Oct 27, 2019 · 41 comments
Labels
topic-typing type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

BPO 38605
Nosy @gvanrossum, @warsaw, @brettcannon, @terryjreedy, @ericvsmith, @jwilk, @stevendaprano, @methane, @ambv, @serhiy-storchaka, @vedgar, @ilevkivskyi, @asottile, @pablogsal, @miss-islington, @isidentical, @gousaiyang, @AlexWaygood
PRs
  • bpo-38605: Make postponed evaluation of annotations default #20434
  • bpo-38605: bump the magic number for 'annotations' future #22630
  • bpo-38605: Update "Future statements" docs since PEP 563 is always enabled #25236
  • bpo-38605: Revert making 'from __future__ import annotations' the default #25490
  • bpo-38605: Set the release for __future__.annotations to 3.11 #25596
  • bpo-38605: Update __future__ module doc as annotations is now "mandatory in 3.11" #25602
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2019-10-27.16:07:10.414>
    labels = ['type-bug', '3.10', 'expert-unicode']
    title = '[typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11'
    updated_at = <Date 2022-02-07.11:12:54.549>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-02-07.11:12:54.549>
    actor = 'jwilk'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Unicode']
    creation = <Date 2019-10-27.16:07:10.414>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 38605
    keywords = ['patch']
    message_count = 40.0
    messages = ['355476', '355495', '355506', '355523', '355546', '360830', '366634', '366635', '369269', '369301', '369303', '370000', '370020', '370028', '370031', '370035', '370037', '370188', '370214', '370220', '370232', '375177', '377783', '378133', '378134', '378135', '378139', '378374', '378416', '390399', '391512', '391515', '391700', '391732', '391755', '391869', '394114', '394115', '404129', '404130']
    nosy_count = 18.0
    nosy_names = ['gvanrossum', 'barry', 'brett.cannon', 'terry.reedy', 'eric.smith', 'jwilk', 'steven.daprano', 'methane', 'lukasz.langa', 'serhiy.storchaka', 'veky', 'levkivskyi', 'Anthony Sottile', 'pablogsal', 'miss-islington', 'BTaskaya', 'gousaiyang', 'AlexWaygood']
    pr_nums = ['20434', '22630', '25236', '25490', '25596', '25602']
    priority = 'normal'
    resolution = 'postponed'
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue38605'
    versions = ['Python 3.10']

    @vstinner
    Copy link
    Member Author

    The PEP-563: Postponed evaluation of annotations was introduced an opt-in feature using "from __future__ import annotations". It is scheduled to become the default in Python 4.0.

    I would prefer to limit the number of incompatible changes in Python 4.0: it should just a "regular" release, with a regular number of incompatible changes. The version number change is going to cause enough troubles...

    Would it be possible possible to enable postponed evaluation of annotations either before or after Python 4.0? For example, can we imagine to enable it by default in Python 3.9? If not, what about Python 3.10?

    See also the PEP-608 (Coordinated Python release) and bpo-38604 (Schedule Py_UNICODE API removal).

    @vstinner vstinner added stdlib Python modules in the Lib dir 3.9 only security fixes labels Oct 27, 2019
    @ilevkivskyi
    Copy link
    Member

    IMO 3.10 would be better, since 3.9 would be too soon (it would be like a schedule for a normal deprecation).

    Also if we are really doing this, I think it is better to announce this soon.

    Also we should try to fix relevant issues related to string annotations (in typing and dataclasses), like python/typing#508, python/typing#574, https://bugs.python.org/issue37838, https://bugs.python.org/issue34776 and https://bugs.python.org/issue37948.

    @gvanrossum
    Copy link
    Member

    We never should have mentioned 4.0 as the target date to make this the
    default (and only) behavior -- who knows whether there will ever even be a
    Python 4.0? Even 3.10 might be on the early side (assuming we'll switch to
    a year-long release cycle per PEP-602 -- we will then make deprecations in
    general take two release cycles).

    I do agree that we should start the process of deprecating the
    non-future behavior here in 3.9. I know of one project with a private
    fork of Python (for other reasons) that has made this default.

    @vstinner
    Copy link
    Member Author

    We never should have mentioned 4.0 as the target date to make this the
    default (and only) behavior

    I am fine with modifying __future__ documentation to only modify the "Mandatory" column to remove Python 4.0, and then close this issue:
    https://docs.python.org/dev/library/__future__.html

    --

    But I like the "PEP-563: Postponed evaluation of annotations", IMHO it would be nice to get it as the default behavior :-) It's just a matter of properly organize the transition ;-)

    @gvanrossum
    Copy link
    Member

    You can bring the deprecation schedule up on discourse or python-dev so more folks can let us know whether they'd be okay with 3.9 or 3.10.

    @vstinner
    Copy link
    Member Author

    @brettcannon
    Copy link
    Member

    I personally like 3.10 as the target as that means users had at least 3 years to move to move over. Plus we can put a warning in the What's New for 3.9 about our plans for 3.10.

    @vstinner
    Copy link
    Member Author

    This issue has been discussed during the Language Summit. A quick poll showed that the majority is in favor of changing the default in Python 3.9.

    Lukasz proposed a PEP update to propose to switch the default in Python 3.9:
    python/peps#1371

    For me, the unclear part is which projects would be impacted if the default changes?

    Someone mentioned attrs, but it seems like attrs is fine:
    python-attrs/attrs#288 (comment)

    In term of workflow, I would _prefer_ to get such incompatible in the very beginning of a devcycle, rather than just before the feature freeze. But I don't think that it's a blocker issue. Technically, changes are allowed until 3.9.0 beta1. Moreover, Lukasz is the 3.9 release manager, the author of the PEP-563 and he is in favor of changing the default in 3.9 :-)

    @vstinner vstinner changed the title [typing] PEP 563: Postponed evaluation of annotations: enable it by default before Python 4.0 [typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.10 May 18, 2020
    @gvanrossum
    Copy link
    Member

    Too bad nobody took any action here after the positive outcome of the discussion at the summit.

    @gvanrossum gvanrossum added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error and removed stdlib Python modules in the Lib dir 3.9 only security fixes labels May 18, 2020
    @vstinner
    Copy link
    Member Author

    Too bad nobody took any action here after the positive outcome of the discussion at the summit.

    I didn't understand it this way. I understood that some people were not 100% comfortable to target 3.9. The question was 3.9 or 3.10. Since the release cycle is now shorter (1 years), only targeting 3.10 is not a big deal ;-)

    @ambv
    Copy link
    Contributor

    ambv commented May 18, 2020

    We'll make this an announced 3.10 feature early on. The discussion at the Summit wasn't as clear cut to me: 35% of participants would rather see this default later than 3.9.

    @vstinner
    Copy link
    Member Author

    Is there anyone interested to implement this change in Python 3.10?

    @vstinner vstinner added the 3.10 only security fixes label May 26, 2020
    @isidentical
    Copy link
    Sponsor Member

    I opened the PR 20434 as draft, but from what I understand, there is going to be some breakage (on our test suite). I'll try to narrow it down (currently ~4 tests instead of ~20) but I dont want to prevent anyone else from working on this, so feel free to ignore my PR if you have a working test suite with a low breakage level.

    @isidentical
    Copy link
    Sponsor Member

    After trying to complete a patch, there are a few issues that immediately showed itself (and this might lead to not to do this in 3.10, I dont know);

    First one is double-forward-ref, which is usage of string-annotations when there is postponed evaluatation of annotations:
    >>> import typing
    >>> from __future__ import annotations
    >>> def x(a: 'int'): pass
    ... 
    >>> typing.get_type_hints(x)
    {'a': ForwardRef('int')}
    If we make annoatations feature default, this would be default behavior. The solution would be a workaround to the compiler;
    static int
    compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
    {
        if (annotation->kind == Constant_kind && PyUnicode_CheckExact(annotation->v.Constant.value)) {
            PyObject *text = annotation->v.Constant.value;
            Py_INCREF(text);
            ADDOP_LOAD_CONST_NEW(c, text);
        } else {
            ADDOP_LOAD_CONST_NEW(c, _PyAST_ExprAsUnicode(annotation));
        }
        return 1;
    }
    But I am not sure if this is too silly or not. 

    The second problem is inspect.signature. If we don't resolve annotations there and continue it is definitely going to break some code. If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

    The third problem is various dataclass hacks. Like _type_{field.name} etc. annotations and how ClassVar/InitVar parsed.

    There are also some little parts that need to change.

    Any thoughts on these issues?

    @vstinner
    Copy link
    Member Author

    The second problem is inspect.signature. If we don't resolve annotations there and continue it is definitely going to break some code.

    Would you mind to elaborate why would it break some code? Consumers of annotations should already be prepared to get directly types or strings, no?

    If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

    I expect that resolving has an impact on performance, whereas the caller may not use annotations at all but only cares of the number of parameters or their name.

    It would be resonable to not resolve annotations in signature() by default. If someone cares, maybe a new parameter can be added to resolve annotations?

    @gvanrossum
    Copy link
    Member

    • Double forward ref: IMO this can be resolved in the get_type_hints() functions. (Łukasz do you agree?)

    • inspect.signature(): Maybe this could switch to using typing.get_type_hints()? Then again if performance is important here maybe we cannot change anything.

    • dataclasses hacks: these should just be resolved.

    Another thought: maybe some of these issues can be considered bugs in 3.9 as well, and we should fix them there too? That might help us decide the right path forward. After all we should really encourage people to start using from __future__ import annotations in their code, to help them get ready for these issues in 3.10.

    Final thought: I know at least the Dropbox client team, and possibly also Instagram, has already turned on from __future__ import annotations by default in their local fork of Python. Maybe we can ask them if they ever felt the need to change inspect.signature or typing.get_type_hints.

    @ericvsmith
    Copy link
    Member

    To my knowledge, dataclasses works with from __future__ import annotations. If there are specific examples of problems, I'd like to hear about it: please open a separate issue.

    There is a hack (discussed at PyCon 2018 with all of the relevant players) where it avoids importing typing to look at typing.ClassVar, but I think that code is all correct. Maybe I should just bite the bullet and import typing, since I believe importing it is faster than it used to be.

    @isidentical
    Copy link
    Sponsor Member

    From now on, should typing.get_type_hints automatically resolve arguments too? An example would be this;

    import typing
    T = typing.TypeVar("T")
    class Loop(typing.Generic[T]):
        subloop: typing.Final["Loop[int]"]
    print(typing.get_type_hints(Loop))
    >>> {'subloop': typing.Final[__main__.Loop[int]]}
    If we run the same code under future annotations
    >>> {'subloop': typing.Final[ForwardRef('Loop[int]')]}

    @gvanrossum
    Copy link
    Member

    I think in general it is more insightful to discuss the behavior of get_type_hints() given specific things in annotations.

    We generally don't write forward refs inside forward refs, like "SomeClass['int']". So maybe that code was wrong? Where did you find it?

    @isidentical
    Copy link
    Sponsor Member

    An example would be this

    class Loop:
    attr: Final['Loop']
    . Either I can change tests in order to reflect now everything is a forward ref by default
    class Loop:
    attr: Final['Loop']
    to
    class Loop:
    attr: Final[Loop]
    or resolve everything on get_type_hints.

    @gvanrossum
    Copy link
    Member

    There will still be a lot of code written that way, because people need compatibility with earlier versions of Python. So I think it should be fixed in get_type_hints().

    @terryjreedy
    Copy link
    Member

    bpo-41314 changed the __future__ annotations default version to 3.10. 3.10.0a1 is scheduled for next Oct 5, less than 2 months from now. It would be good if PR 20434 were merged before that.

    @gvanrossum
    Copy link
    Member

    New changeset 22220ae by Batuhan Taskaya in branch 'master':
    bpo-38605: bump the magic number for 'annotations' future (bpo-22630)
    22220ae

    @miss-islington
    Copy link
    Contributor

    New changeset 1be456a by Saiyang Gou in branch 'master':
    bpo-38605: Update "Future statements" docs since PEP-563 is always enabled (GH-25236)
    1be456a

    @pablogsal pablogsal reopened this Apr 20, 2021
    @pablogsal
    Copy link
    Member

    New changeset b0544ba by Pablo Galindo in branch 'master':
    bpo-38605: Revert making 'from __future__ import annotations' the default (GH-25490)
    b0544ba

    @vstinner vstinner changed the title [typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.10 [typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11 Apr 21, 2021
    @vstinner
    Copy link
    Member Author

    New changeset b0544ba by Pablo Galindo in branch 'master':

    Copy of the NEWS entry:

    Revert making from __future__ import annotations the default. This follows
    the Steering Council decision to postpone PEP-563 changes to at least Python
    3.11. See the original email for more information regarding the decision:
    https://mail.python.org/archives/list/python-dev@python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/.
    Patch by Pablo Galindo.

    @gvanrossum
    Copy link
    Member

    I haven't followed this precisely, but I recall that when we first made PEP-563 the default, we had to fix a number of bugs in various library modules (e.g. inspect) that were only apparent when from __future__ import annotations was used. I hope we're not throwing away those bugfixes. Hopefully Batuhan has a recollection of what I am thinking of, there was some significant delay while we figured out what to do about some of these.

    @isidentical
    Copy link
    Sponsor Member

    Hopefully Batuhan has a recollection of what I am thinking of, there was some significant delay while we figured out what to do about some of these.

    The major one that I'd recall is that inspect.signature() just uses whatever is in __annotations__ instead of resolving those. Now that __future__.annotations is not the default one, we can add a new option named 'resolve_annotations' and call typing.get_type_hints when activated. Here is a quick demo;

    from __future__ import annotations
    
    import inspect
    
    def foo(a: int, b: int) -> str:
        ...
    
    def _get_annotations(func, **signature_opts):
        signature = inspect.signature(func, **signature_opts)
        return {
            param.name: param.annotation
            for param in signature.parameters.values()  
        }
    
    print('bare: ', _get_annotations(foo))
    print('annotations resolved: ', _get_annotations(foo, resolve_annotations=True))

    bare: {'a': 'int', 'b': 'int'}
    annotations resolved: {'a': <class 'int'>, 'b': <class 'int'>}

    This would be a clear feature for both PEP-563 users + people who are still using string annotations mixed with normal ones. What do you think Guido?

    @gvanrossum
    Copy link
    Member

    I don't like adding a flag. In the PEP-563-by-default world we didn't need a flag. Perhaps once Larry implements inspect.get_annotations() (bpo-43817) we can make inspect.signature() always call that?

    @miss-islington
    Copy link
    Contributor

    New changeset f84f1b5 by Saiyang Gou in branch 'master':
    bpo-38605: Update future module doc as annotations is now "mandatory in 3.11" (GH-25602)
    f84f1b5

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented May 21, 2021

    May I ask, is this going forward? I installed 3.11-dev, it's not there (though __future__ claims it should be). I understand if it isn't done yet, just want to know if there's risk it will be postponed again (or even given up).

    @isidentical
    Copy link
    Sponsor Member

    May I ask, is this going forward? I installed 3.11-dev, it's not there (though __future__ claims it should be). I understand if it isn't done yet, just want to know if there's risk it will be postponed again (or even given up).

    We are still waiting a ruling on PEP-649. If it gets rejected, and no more ideas arises (before beta cut for 3.11), I guess we could move on and resolve this issue again.

    @stevendaprano
    Copy link
    Member

    Now that we're in 3.11, people are starting to notice that stringy annotations are not the default (see bpo-45499 for example).

    What can we do to get PEP-649 moving forward?

    @serhiy-storchaka
    Copy link
    Member

    I think it would help if we could enable some future feature globally by command line option or environment variable, without modifying all source files. It would allow users to quickly test their code base for compatibility with future changes. The problem currently is that nobody bothers to add "from __future__ import ...", so we have surprises every time when try to make it by default.

    A tool which automatically adds or removes "from __future__ import ..." in files could help too.

    @ahmedsayeed1982 ahmedsayeed1982 mannequin added topic-unicode 3.8 only security fixes and removed interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.10 only security fixes labels Nov 4, 2021
    @eryksun eryksun added 3.10 only security fixes and removed 3.8 only security fixes labels Nov 4, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @AlexWaygood AlexWaygood added topic-typing and removed 3.10 only security fixes labels Apr 13, 2022
    @vstinner
    Copy link
    Member Author

    vstinner commented Nov 3, 2022

    There are now multiple PEPs proposed to handle this problem differently.

    PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.11

    The SC asked to hold this for now. So I prefer to close the issue. Please open a new issue once the SC took a decision on this topic.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    topic-typing type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    15 participants