`from __future__ import annotations` has no effect inside `ast.parse`

[kayhayen]

BPO	35143
Nosy	@ericvsmith, @benjaminp, @ambv, @ilevkivskyi, @pablogsal, @isidentical

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2019-12-28.14:04:14.565>
created_at = <Date 2018-11-02.08:28:53.021>
labels = ['extension-modules', 'type-bug', '3.8']
title = '`from __future__ import annotations` has no effect inside `ast.parse`'
updated_at = <Date 2019-12-28.14:04:14.564>
user = 'https://bugs.python.org/kayhayen'

bugs.python.org fields:

activity = <Date 2019-12-28.14:04:14.564>
actor = 'eric.smith'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2019-12-28.14:04:14.565>
closer = 'eric.smith'
components = ['Extension Modules']
creation = <Date 2018-11-02.08:28:53.021>
creator = 'kayhayen'
dependencies = []
files = []
hgrepos = []
issue_num = 35143
keywords = []
message_count = 7.0
messages = ['329114', '329904', '356827', '358907', '358910', '358941', '358945']
nosy_count = 7.0
nosy_names = ['eric.smith', 'benjamin.peterson', 'lukasz.langa', 'kayhayen', 'levkivskyi', 'pablogsal', 'BTaskaya']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35143'
versions = ['Python 3.8']

[kayhayen]

Hello,

in trying to know what to put into "__annotations__" at run time, the "from __future__ import annotations" pose a new problem. It is now necessary to "unparse" to ast code that is still there.

Code to do so, is in C API "_PyAST_ExprAsUnicode", but won't work with the Python level ast objects.

I have a kludge, using "compile" and "exec", to get past that. It's pretty ugly to do it like this, and Nuitka cannot be alone in trying to predict the value of "__annotations__".

This code is my "ast.unparse" replacement:

def unparse(node):
        _host_node = ast.parse("x:1")

    _host_node.body[0].annotation = node

        r = compile(_host_node, "<annotations>", "exec", 1048576, dont_inherit = True)

        # Using exec here, to compile the ast node tree back to string,
        # there is no accessible "ast.unparse", and this works as a hack
        # to convert our node to a string annotation, pylint: disable=exec-used
        m = {}
        exec(r, m) 

    return m["__annotations__"]['x']

I am caching "_host_node" in the real code.

Having a real "ast.unparse" would be better however. It seems that the building blocks are all there, just not in that form.

Yours,
Kay

[ambv]

I think this is a bug, actually. It's going to be an incompatibility in ast.parse so we'll probably want to only enable it in Python 3.8.

This should have stringified annotations:

>>> p = ast.parse("""
... from __future__ import annotations
... a: 1
... """)
>>> p.body[1].annotation  # ACTUAL
<_ast.Num object at 0x1048cf470>

>>> p.body[1].annotation  # EXPECTED
<_ast.Str object at 0x1048cf6a0>

[isidentical]

Hey @lukasz.langa, I want to work on this. Should we add an interface to _PyAST_ExprAsUnicode or just the bugfix for annotations?

[isidentical]

Having a real "ast.unparse" would be better however. It seems that the building blocks are all there, just not in that form.

Now we have that :) https://docs.python.org/3.9/whatsnew/3.9.html#improved-modules

[pablogsal]

I don't think this is a bug. Technically the 'from __future__ import annotations' is a compiler option, not an AST one. Also, the flag says that "annotations become strings at runtime" and at the point, we have an AST is not still runtime. The CO_FUTURE_ANNOTATIONS only takes place in the compiler that reverses the AST object into a string. If we make the AST aware of CO_FUTURE_ANNOTATIONS, then what the compiler will see in the presence of 'from __future__ import annotations' would be different if you compile a module than if you parse it using ast.parse() and then compile the result (in the second case the compiler will see strings in the annotations, not AST nodes while in the first case it will see nodes). Additionally, doing this will limit tools that want to analyze the contents of those annotations before they get turned into strings (like type checker, linters...etc).