Interpolation.expression drops trailing whitespace despite documentation saying it includes any whitespace

[kid-lxy]

Bug report

Bug description:

For t-string literals, Interpolation.expression preserves whitespace after the opening {, but removes whitespace immediately before the closing }.

This appears inconsistent with the string.templatelib documentation, which states that, for interpolations created from t-string literals, expression contains the expression text inside the curly braces, including any whitespace.

Minimal reproducer

x = 42

cases = [
    t"{x}",
    t"{x }",
    t"{ x}",
    t"{ x }",
    t"{  x  }",
    t"""{
  x
}""",
]

for template in cases:
    print(repr(template.interpolations[0].expression))

Actual behavior

On CPython main commit b86a41c:

'x'
'x'
' x'
' x'
'  x'
'\n  x'

Whitespace following { is preserved, but whitespace before } is removed.

Expected behavior

Based on the current documentation, I expected:

'x'
'x '
' x'
' x '
'  x  '
'\n  x\n'

In particular:

t"{x }".interpolations[0].expression == "x "
t"{ x }".interpolations[0].expression == " x "
t"{  x  }".interpolations[0].expression == "  x  "

Possible cause

A possible cause is the way the lexer reconstructs interpolation source text in Parser/lexer/string.c.

_PyLexer_update_ftstring_expr() records last_expr_end when it reaches the closing }, !, or : token:

case '}':
case '!':
    tok_mode->last_expr_end = strlen(tok->start);
    break;
case ':':
    if (tok_mode->last_expr_end == -1) {
        tok_mode->last_expr_end = strlen(tok->start);
    }
    break;

_PyLexer_set_ftstring_expr() then creates the metadata from:

tok_mode->last_expr_buffer,
tok_mode->last_expr_size - tok_mode->last_expr_end

This appears to remove the suffix represented by last_expr_end. In the ordinary } case, that suffix seems to include whitespace immediately before the closing brace, while whitespace immediately after the opening brace remains in last_expr_buffer.

That would explain the observed asymmetry:

t"{ x }"  -> " x"

leading whitespace preserved
trailing whitespace removed

This is only a proposed cause based on the observed behavior and a source-code review.

CPython versions tested on:

3.16, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-154719: Preserve trailing whitespace in t-string interpolation expressions #154736