Incorrect exception highlighting for fstring format

[pablogsal]

BPO	44885
Nosy	@gvanrossum, @ammaraskar, @lysnikolaou, @pablogsal, @isidentical
PRs	bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions #27729 [3.10] bpo-44885: Correct the ast locations of f-strings with format … #27743 [3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) #27744

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-12.16:42:19.299>
created_at = <Date 2021-08-11.07:59:02.051>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'Incorrect exception highlighting for fstring format'
updated_at = <Date 2021-08-12.17:46:49.800>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2021-08-12.17:46:49.800>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-12.16:42:19.299>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-08-11.07:59:02.051>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 44885
keywords = ['patch']
message_count = 8.0
messages = ['399372', '399392', '399397', '399399', '399400', '399469', '399471', '399475']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ammar2', 'lys.nikolaou', 'pablogsal', 'BTaskaya']
pr_nums = ['27729', '27743', '27744']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44885'
versions = ['Python 3.11']

[pablogsal]

Given this code:

print(f"Here is that pesky {xxx/2:.3f} again")

The traceback prints:

Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/lel.py", line 1, in <module>
    print(f"Here is that pesky {xxx/2:.3f} again")
           ^^^
NameError: name 'xxx' is not defined

Removing the formatting part ":.3f" makes it work as expected

[gvanrossum]

(I originally reported this.)

[pablogsal]

The problem lies here:

cpython/Parser/string_parser.c

Lines 374 to 389 in f66d00f

	// The call to fstring_find_expr_location is responsible for finding the column offset
	// the generated AST nodes need to be shifted to the right, which is equal to the number
	// of the f-string characters before the expression starts. In order to correctly compute
	// this offset, strstr gets called in fstring_find_expr_location which only succeeds
	// if curly braces appear before and after the f-string expression (exactly like they do
	// in the f-string itself), hence the following lines.
	str[0] = '{';
	memcpy(str+1, expr_start, len);
	str[len+1] = '}';
	str[len+2] = 0;
	int lines, cols;
	if (!fstring_find_expr_location(t, str, &lines, &cols)) {
	PyMem_Free(str);
	return NULL;
	}

The problem is that the strstr call will fail because the string containing the expression doesn't have the formatting part in it, so the offsets never get corrected.

[pablogsal]

Actually, this has even more problems. Because we are using strstr to find the start of the expression in the parent string, if the expression is repeated the offsets are incorrectly generated:

For example:

print(f"Here is that {xxx} pesky {xxx} again")

This produces:

...
FormattedValue(
value=Name(
id='xxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...
FormattedValue(
value=Name(
id='xxxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...

while

print(f"Here is that {xxx} pesky {xxxx} again")

(different variables) produces:

...
FormattedValue(
value=Name(
id='xxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...
FormattedValue(
value=Name(
id='xxxx',
ctx=Load(),
lineno=1,
col_offset=34,
end_lineno=1,
end_col_offset=38),
...

[pablogsal]

Ha, we have test that knows is broken:

cpython/Lib/test/test_fstring.py

Line 217 in f66d00f

NOTE: this is currently broken, always sets location of the first

[pablogsal]

New changeset 8e832fb by Pablo Galindo Salgado in branch 'main':
bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729)
8e832fb

[pablogsal]

New changeset c28c2e1 by Pablo Galindo Salgado in branch '3.10':
[3.10] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27743)
c28c2e1

[pablogsal]

New changeset 4b86c9c by Pablo Galindo Salgado in branch '3.9':
[3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27744)
4b86c9c