Indentation error while getting annotation of function with an template-string as return type

[15r10nk]

Bug report

Bug description:

I found the following bug while fuzzing with pysource-codegen

from annotationlib import get_annotations, Format

def name_3() -> t'{ {}}':
    pass
get_annotations(name_3, format=Format.STRING)

output (Python 3.14.0rc2+):

Traceback (most recent call last):
  File "/home/frank/projects/pysource-playground/crash.py", line 5, in <module>
    get_annotations(name_3, format=Format.STRING)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frank/projects/cpython/install/lib/python3.14/annotationlib.py", line 914, in get_annotations
    ann = _get_and_call_annotate(obj, format)
  File "/home/frank/projects/cpython/install/lib/python3.14/annotationlib.py", line 1034, in _get_and_call_annotate
    ann = call_annotate_function(annotate, format, owner=obj)
  File "/home/frank/projects/cpython/install/lib/python3.14/annotationlib.py", line 683, in call_annotate_function
    key: _stringify_single(val)
         ~~~~~~~~~~~~~~~~~^^^^^
  File "/home/frank/projects/cpython/install/lib/python3.14/annotationlib.py", line 814, in _stringify_single
    return ast.unparse(_template_to_ast(anno))
                       ~~~~~~~~~~~~~~~~^^^^^^
  File "/home/frank/projects/cpython/install/lib/python3.14/annotationlib.py", line 573, in _template_to_ast
    value=ast.parse(part.expression),
          ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/frank/projects/cpython/install/lib/python3.14/ast.py", line 46, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    {}
IndentationError: unexpected indent

I think the problem is the space before {} but this space is required to distinguish it from an escaped {.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-138558: Improve handling of Template annotations in annotationlib #139072
• gh-138558: Improve description of Interpolation.expression in library docs #139187
• [3.14] gh-138558: Improve handling of Template annotations in annotationlib (GH-139072) #139272

[AA-Turner]

Simpler reproducer:

def f() -> t'{ 0}':
    pass

get_annotations(f, format=Format.STRING)

or even:

>>> import ast
>>> tuple(t'{ 0}')[0].expression
' 0'
>>> ast.parse(tuple(t'{ 0}')[0].expression)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    ast.parse(tuple(t'{ 0}')[0].expression)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\ast.py", line 46, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    0
IndentationError: unexpected indent

cc @JelleZijlstra

[JelleZijlstra]

I suppose we should do ast.parse(part.expression.strip()) instead?

Arguably this should also be documented on the template side. https://docs.python.org/3.14/library/string.templatelib.html#string.templatelib.Interpolation.expression says it is "The text of a valid Python expression, or an empty string." But " 0" is neither.

[brianschubert]

Just stripping leading whitespace before the offending ast.parse call would make these cases work, but I don't know that's the "right" fix

diff --git a/Lib/annotationlib.py b/Lib/annotationlib.py
index bee019cd515..34e6a321d0b 100644
--- a/Lib/annotationlib.py
+++ b/Lib/annotationlib.py
@@ -570,7 +570,7 @@ def _template_to_ast(template):
             case _:
                 interp = ast.Interpolation(
                     str=part.expression,
-                    value=ast.parse(part.expression),
+                    value=ast.parse(part.expression.lstrip()),
                     conversion=(
                         ord(part.conversion)
                         if part.conversion is not None

[AA-Turner]

I think something like ast.parse(textwrap.dedent(expr).lstrip()) is needed. Consider something along the lines of the following:

>>> def f() -> t'''{  0
...  + 0 # comment \
...  - 0 \
...  * 0}''':
...     pass
>>> expr = tuple(t)[0].expression
>>> expr
'  0\n + 0 \n - 0 \\\n * 0'
>>> ast.parse(textwrap.dedent(expr).strip())
Module(body=[Expr(value=Constant(value=0, kind=None)), ..., Expr(value=BinOp(left=UnaryOp(...), op=Mult(...), right=Constant(...)))], type_ignores=[])

However, as Jelle says, this could be a implementation problem with t-strings -- is it more important to have exact whitespace preserved, or to conform to being a valid Python expression? cc @lysnikolaou @davepeck for thoughts here.

A

[JelleZijlstra]

textwrap.dedent is not enough either:

>>> t = t'''{ 0
... & 1
... | 2
... }'''
>>> tuple(t)[0].expression
' 0\n& 1\n| 2'
>>> ast.parse(_)
Traceback (most recent call last):
  File "<python-input-15>", line 1, in <module>
    ast.parse(_)
    ~~~~~~~~~^^^
  File "/Users/jelle/py/cpython/Lib/ast.py", line 46, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    0
IndentationError: unexpected indent

A robust solution probably would be to enclose the expression in parentheses:

>>> ast.parse("(" + tuple(t)[0].expression + ")")
Module(body=[Expr(value=BinOp(left=BinOp(...), op=BitOr(...), right=Constant(...)))], type_ignores=[])

[davepeck]

Interesting.

"The text of a valid Python expression, or an empty string."

I'll update this. If the Interpolation comes from a t-string literal, expression will be valid source text for inside of interpolation curly braces, whitespace included. If the Interpolation is constructed directly, expression is an arbitrary string; we recommend that developers set it to a valid expression or leave it empty, but can't prevent other values.

(Update: After a bit more time dwelling on this issue, I've removed most of my reply here in favor of the one that follows.)

[davepeck]

@JelleZijlstra Can you help me understand annotationlib's requirements here?

The _template_to_ast() function tries to take a Template instance and return the AST for a t-string literal.

But not every Template instance has a t-string literal equivalent:

1. It might might have an invalid Interpolation.expression, as in Template(Interpolation(42, '😅'))
2. It might have a valid expression that does not evaluate to the corresponding value, as in Template(Interpolation(42, '99')). (Currently, _template_to_ast() incorrectly returns an AST for t'{99}' in this case.)

Trying to recover a t-string literal from an instance seems... problematic.

What if _template_to_ast() returned an AST for the non-literal form instead? (Call(func='Template', args=[...]) etc.) Would this uphold annotationlib's contract?

Stepping further back, it looks to me like we maybe always end up calling ast.unparse() on the AST to get a string form? If that's the case, what if we updated Template.__repr__ to return this string directly? Could we avoid special casing Template entirely in annotationlib? (Right now eval(repr(some_template)) raises an exception.)

Thanks!

[JelleZijlstra]

The main thing it tries to do is that retrieving annotations in the STRING format returns something as close as possible to the original source:

>>> def f(x: t"x", y: t"{y}"): pass
... 
>>> annotationlib.get_annotations(f, format=annotationlib.Format.STRING)
{'x': "t'x'", 'y': "t'{y}'"}

It would be undesirable if this returned Template("x") or something like that.

For more exotic code, annotationlib should make a best-effort attempt to get close to the original code, but it's not terrible if for something like a manually created Template instance, it returns something slightly different.

[davepeck]

@JelleZijlstra Got it, thanks!

So perhaps _template_to_ast() could:

1. Attempt to parse all Interpolation.expressions (with your suggested parens fix for whitespace)
2. If all parses succeed, return a t-string literal AST
3. If any parse fails, return a non-literal AST

This will:

1. Always return a t-string literal AST if the Template started life as a literal
2. Always return a non-literal AST in cases where expressions aren't parsable (Template(Interpolation(42, '😅')))
3. Return an incorrect literal AST in cases where .expression wouldn't evaluate to .value (Template(Interpolation(42, '99')))

If we're okay with the oddity of that last case, then perhaps this could work?

[JelleZijlstra]

Yes, that makes sense to me. The Template(Interpolation(42, '99'))) seems like a case of garbage in, garbage out to me.