lib2to3 should parse f-strings

[sk]

BPO	33991
Nosy	@ericvsmith, @benjaminp, @stevendaprano, @sk-
Superseder	bpo-45544: Close 2to3 issues and list them here

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.22:50:46.701>
created_at = <Date 2018-06-28.22:16:04.205>
labels = ['3.8', '3.7', 'expert-2to3']
title = 'lib2to3 should parse f-strings'
updated_at = <Date 2021-10-20.22:50:46.701>
user = 'https://github.com/sk'

bugs.python.org fields:

activity = <Date 2021-10-20.22:50:46.701>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.22:50:46.701>
closer = 'iritkatriel'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2018-06-28.22:16:04.205>
creator = 'skreft'
dependencies = []
files = []
hgrepos = []
issue_num = 33991
keywords = []
message_count = 5.0
messages = ['320687', '320691', '320707', '320711', '320790']
nosy_count = 4.0
nosy_names = ['eric.smith', 'benjamin.peterson', 'steven.daprano', 'skreft']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45544'
type = None
url = 'https://bugs.python.org/issue33991'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[sk]

Currently f-strings are parsed just as regular strings by lib2to3. However, this has the problem that the invariant of a string node being a literal is now broken.

This poses two problems. On one hand, if I want to compare that two string nodes are equivalent I would need to do something like:

def string_nodes_are_eqivalent(node1, node2):
  if is_f_string(node1) and is_f_string(node2):
    # This would require to parse the nodes using ast.parse
    return f_strings_are_equivalent(node1, node2)
  if not is_f_string(node1) and not is_f_string(node2):
    return ast.literal_eval(node1.value) == ast.literal_eval(node2.value)
  return False

Note that ast.literal_eval does not accept f-strings.
Also note that ast.parse returns an ast.JoinedString for f-strings, whose elements are either ast.Str or ast.FormattedValue, the latter being ast expressions.

On the other hand, this has the problem of not being able to refactor the expressions inside an f-string.

[stevendaprano]

I'm curious how you are getting f-strings (introduced in Python 3.6) in Python 2 code that you are passing to 2to3.

[sk]

@steven lib2to3 is no longer specifically for python 2 code, it also parses python 3 code and is the building block of tools like YAPF.

See Eli's comment on this https://bugs.python.org/issue23894: "AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that".

I added the versions 3.6-3.8 as previous ones are not able to parse f-strings.

[stevendaprano]

@steven lib2to3 is no longer specifically for python 2 code, it also
parses python 3 code

Ah, all good then. Thanks.

[sk]

Note also, that lib2to3 will parse invalid f-strings like f"hello {", whereas ast.parse will raise a SyntaxError exception.

See below for reproduction cases:
In [2]: lib2to3.tests.support.parse_string('f"hello {"')
Out[2]: Node(file_input, [Node(simple_stmt, [Leaf(3, 'f"hello {"'), Leaf(4, '\n')]), Leaf(0, '')])

In [4]: ast.parse('f"hello {"')
Traceback (most recent call last):

File "/Users/skreft/.virtualenvs/pyfaster/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)

File "<ipython-input-4-78a4f2773f7f>", line 1, in <module>
ast.parse('f"hello {"')

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)

File "<unknown>", line 1
SyntaxError: f-string: expecting '}'