Error parsing Python raw f-string with backslash

[jmd-dk]

Python(3) code using raw f-strings containing a backslash within escaped braces (i.e. double braces) cannot be parsed. A minimum example is

fr'{{\S'

Some pointers:

• Works: f'{{\S'.
• Works: r'{{\S'
• Works: rf'{{'.
• Works: rf'\S'.
• Also do not work: rf'{{\S'.
• The S is not important, though some character must be there in order for the syntax to be valid.

This might seem like a pretty esoteric case. For me it comes up a lot when writing LaTeX in Matplotlib.

[jmd-dk]

I think I now understand the problem and have the solution.

In lexers/python.py under # raw f-strings, only quotes ('tdqf', 'tsqf', 'dqf', 'sqf') are given as the third element of each tuple. Looking below at "non-raw f-strings" (normal f-strings) and "non-raw strings" (normal strings), their third elements are combined('fstringescape', <quote>) and combined('stringescape', <quote>), respectively. Furhter down we find the definition of fstringescape:

'fstringescape': [
    (r'\{\{', String.Escape),
    (r'\}\}', String.Escape),
    include('stringescape'),
],

i.e. fstringescape is just stringescape with the addition of {{ and }}.

We can try adding the missing third elements, resulting in the new code:

# raw f-strings
('(?i)(rf|fr)(""")',
 bygroups(String.Affix, String.Double),
 combined('fstringescape', 'tdqf')),
("(?i)(rf|fr)(''')",
 bygroups(String.Affix, String.Single),
 combined('fstringescape', 'tsqf')),
('(?i)(rf|fr)(")',
 bygroups(String.Affix, String.Double),
 combined('fstringescape', 'dqf')),
("(?i)(rf|fr)(')",
 bygroups(String.Affix, String.Single),
 combined('fstringescape', 'sqf')),

The rf'{{...}}' with backslashes will now lex without error. We've however lost the "raw-ness", as e.g. \n in rf'{{\n}}' will be understood as a formatted unit. The culprit is the normal stringescape included by fstringescape. We thus have to make a new such "escape" containing only {{ and }}, call it rfstringescape:

'rfstringescape': [
    (r'\{\{', String.Escape),
    (r'\}\}', String.Escape),
],

Now using

# raw f-strings
('(?i)(rf|fr)(""")',
 bygroups(String.Affix, String.Double),
 combined('rfstringescape', 'tdqf')),
("(?i)(rf|fr)(''')",
 bygroups(String.Affix, String.Single),
 combined('rfstringescape', 'tsqf')),
('(?i)(rf|fr)(")',
 bygroups(String.Affix, String.Double),
 combined('rfstringescape', 'dqf')),
("(?i)(rf|fr)(')",
 bygroups(String.Affix, String.Single),
 combined('rfstringescape', 'sqf')),

my raw f-strings containing backslashes within {{...}} seems to be lexed properly.

Thank you for leading the way.

I have a version that more or less solves all these problems for all bytes and string literals, I still need to clean up a bit but results are great so far.

Updated examples from https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals.

[Anteru]

That looks quite promising -- looking forward to the PR!

I also have nested lambdas in function definitions (IMHO the most trickiest token to catch)

As you can see, I've used some non standard types (about 5 or so) such as Name.Function.Parameter (fp class here) or Text.LineBreak(br class). That's what i need to clean up.

I'll send one when it's ready 🖖

[jmd-dk]

Question

I wonder why Pygments (in the case of Python code) doesn't just use the built-in machinery of Python, where all of the tokenization/lexing etc. is fully exposed? Why reimplement all of this from scratch, which is obviously hard to get 100% right?

@jmd-dk i dunno i was looking at the tokenize module to be honest.
Comparatively, retrieving tokens using the module and dispatching these to pygments internal tokens would greatly reduce the number of operations required for the same result.

[birkenfeld]

@jmd-dk a highlighter is not the same as a language tokenizer.

tokenize's output (as you can see in TrippyHank's screenshot) is much less detailed than Pygments'. That's because many distinctions (e.g. between raw and non-raw strings) are handled later in the interpreter's pipeline.

On the other hand, Pygments needs to be more lenient since it doesn't only support the one version of the langauge it happens to run on: using tokenize, new syntax in 3.11 would fail to highlight if Pygments runs on 3.10.

[birkenfeld]

I also have nested lambdas in function definitions (IMHO the most trickiest token to catch)

As you can see, I've used some non standard types (about 5 or so) such as Name.Function.Parameter (fp class here) or Text.LineBreak(br class). That's what i need to clean up.

I'll send one when it's ready vulcan_salute

Please see #1654 for why this is not a good idea.

@birkenfeld Hmmm ..

a highlighter is not the same as a language tokenizer.

Pygments is literally tokenizing input text in order to output to other formats (HTML up here).

We're just talking about replacing the tokenizing process for Python (could be an opt-in) since there's a built-in module for that, that will:

1. Simplify the code base
2. Receive updates with python version changes
3. Eliminate all the burden of tokenizing the whole python language by "hand" with regexes

Even tho, i have a pretty complete version that I'm very happy with right now. I know there might be some special edge case I've graciously missed because ey I've got only two hands and I'm living on a blue planet after all.

And i definitly don't care if this is either a good or a bad idea, if it works, it works 😌