Expressions with format specifiers in f-strings give wrong code position in AST

[numirias]

BPO	35212
Nosy	@blueyed, @ericvsmith, @asmeurer, @numirias, @yangyangxcf, @DanielNoord
PRs	bpo-35212: fix col_offset in f-strings #19398 bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions #27729
Superseder	bpo-44885: Incorrect exception highlighting for fstring format

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2022-03-16.12:23:28.718>
created_at = <Date 2018-11-11.12:34:00.842>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Expressions with format specifiers in f-strings give wrong code position in AST'
updated_at = <Date 2022-03-16.12:23:28.718>
user = 'https://github.com/numirias'

bugs.python.org fields:

activity = <Date 2022-03-16.12:23:28.718>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2022-03-16.12:23:28.718>
closer = 'eric.smith'
components = ['Interpreter Core']
creation = <Date 2018-11-11.12:34:00.842>
creator = 'arminius'
dependencies = []
files = []
hgrepos = []
issue_num = 35212
keywords = ['patch']
message_count = 4.0
messages = ['329672', '353908', '376163', '415337']
nosy_count = 6.0
nosy_names = ['blueyed', 'eric.smith', 'asmeurer', 'arminius', 'fhsxfhsx', 'danielnoord']
pr_nums = ['19398', '27729']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '44885'
type = 'behavior'
url = 'https://bugs.python.org/issue35212'
versions = ['Python 3.7']

[numirias]

ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier.

Compare the trees of f'{a}' and f'{a:b}' :

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets,

Name(lineno=1, col_offset=3, id='a', ctx=Load())

and

Name(lineno=1, col_offset=1, id='a', ctx=Load())

.

Expected behavior: col_offset=3 for name "a" in both f-strings.

(In my specific use case, this breaks a semantic highlighting plugin for vim.)

[numirias]

There is another instance of incorrectly reported offsets in f-strings, in this case affecting multi-line strings.

Example:

(
f'aaa{foo}bbb'
f'ccc{bar}ddd'
f'eee{baz}fff'
)

Relevant part of the syntax tree:

JoinedStr(
lineno=2,
col_offset=4,
values=[
Str(lineno=2, col_offset=4, s='aaa'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=10, id='foo', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='bbbccc'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=5, id='bar', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='dddcccaaaaaaaaaaaaaa'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=5, id='baz', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='ddd'),
],
),

The reported code position for "foo" is correct (2,10), but not for "bar" (2,5) and "baz" (2,5).

Could you give an estimate as to whether this issue will be looked at any time soon?

[asmeurer]

The same thing occurs with specifiers like {a!r}.

[DanielNoord]

This was fixed by #27729.

On 3.10:

>>> ast.parse("f'{a}'").body[0].value.values[0].value.col_offset
3
>>> ast.parse("f'{a:b}'").body[0].value.values[0].value.col_offset
3

The other issue also no longer reproduces.