Support PEP 701: Syntactic formalization of f-strings in 3.12

[JelleZijlstra]

https://peps.python.org/pep-0701/, which will be in Python 3.12, adds support for several f-string variations that our current parser won't handle:

>>> f"{f"{"f"}"}"
'f'

We should support this new syntax, somehow.

I don't know if this is doable with blib2to3.

[zero00072]

I have same problem.

My Python code:

#!/usr/bin/env python3.12

print(f"The result: {"1" if True else "0"}")

black it:

user@localhost:~$ black test.py
error: cannot format test.py: Cannot parse: 3:22: print(f"The result: {"1" if True else "0"}")

Oh no! 💥 💔 💥
1 file failed to reformat.

With Python console:

>>> print(f"The result: {"1" if True else "0"}")
The result: 1

The AST:

import ast

_ = ast.Module(
    [
        ast.Expr(
            ast.Call(
                ast.Name("print"),
                [
                    ast.JoinedStr(
                        [
                            ast.Constant("The result: "),
                            ast.FormattedValue(
                                ast.IfExp(
                                    ast.Constant(True),
                                    ast.Constant("1"),
                                    ast.Constant("0"),
                                ),
                                -1,
                            ),
                        ]
                    )
                ],
                [],
            )
        )
    ],
    type_ignores=[],
)

[DarkAlchy]

Apparently, this was never fixed as I have the same issue with the web demo or ran locally.

python --version
Python 3.10.12

[sodul]

@hauntsaninja do you have an update on this issue and if there is work to get it resolved? I see that it seems to be a common problem which has new reports popping up every few weeks. I do understand that this is probably tricky to fix but it is pretty bad that black is unable to parse what is now valid python. Our workaround is to extract variables out of f-strings so that there are no parsing issues, but that defeats the purpose of PEP 701.

I know the new syntax is not valid with 3.11 or lower but the same could have been said of the := operator introduced with 3.8.

[JelleZijlstra]

There is an open PR, #3822, but it hasn't made much progress recently. Perhaps you can work with the PR author to move it along.

This grammar change is much trickier than most previous ones because it involves changes to the tokenizer. That's why it's been harder to get into Black than most previous grammar changes.

[sodul]

Thanks for the update @JelleZijlstra, I've worked with 'quoting' libraries several lifetimes ago, so I fully appreciate how difficult supporting PEP 701 can be and I'm afraid I would not be much help here, except for moral support. Maybe whomever implemented PEP 701 for 3.12 could help?

Thank you for your work on this!