Python 3.12 tokenize generates invalid locations for f'\N{unicode}'

[smartbomb]

Bug report

Bug description:

from tokenize import untokenize, generate_tokens
from io import StringIO

untokenize(generate_tokens(StringIO("f'\\N{EXCLAMATION MARK}'").readline))

ValueError: start (1,22) precedes previous end (1,24)

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-115154: Fix untokenize handling of unicode named literals #115171
• [3.12] gh-115154: Fix untokenize handling of unicode named literals (GH-115171) #115662

[Eclips4]

cc @pablogsal

[terryjreedy]

The tokens parsing f'\\N{EXCLAMATION MARK}' and traceback unparsing the list (toks) are

[TokenInfo(type=59 (FSTRING_START), string="f'", start=(1, 0), end=(1, 2), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=60 (FSTRING_MIDDLE), string='\\N{EXCLAMATION MARK}', start=(1, 2), end=(1, 22),
   line="f'\\N{EXCLAMATION MARK}'"), 
TokenInfo(type=61 (FSTRING_END), string="'", start=(1, 22), end=(1, 23), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=4 (NEWLINE), string='', start=(1, 23), end=(1, 24), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
#
Traceback (most recent call last):
  File "F:\dev\tem\tem2.py", line 6, in <module>
    print(untokenize(toks))
  File "C:\Programs\Python313\Lib\tokenize.py", line 294, in untokenize
    out = ut.untokenize(iterable)
  File "C:\Programs\Python313\Lib\tokenize.py", line 223, in untokenize
    self.add_whitespace(start)
  File "C:\Programs\Python313\Lib\tokenize.py", line 176, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (1,22) precedes previous end (1,24)

The problem is that when Untokenizer.untokenize, line215 sees the FSTRING_MIDDLE token, it replaces '{' and '}' with '{{' and '}}' and bumps the end position by 2, making the end column 2 more than the next start column. This works when the presence of curly brackets results from the reverse process, but not when the lexer recognizes \N{name} as a unicode named literal without replacing it with the indicated character. Other escapes are resolved, as with '\ueeee' being tokenized as a single character. Unless the tokenizer replaces \N{name} with a character, the untokenizer must recognize it also and not do the replacement.

[pablogsal]

CC: @isidentical