Add f-string support to tokenize.py

[skrah]

BPO	25311
Nosy	@ericvsmith, @skrah, @vadmium
Files	tokenize.patch issue25311.diff issue25311-1.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2015-10-26.08:44:45.248>
created_at = <Date 2015-10-04.17:23:27.578>
labels = ['easy', 'type-bug', 'library']
title = 'Add f-string support to tokenize.py'
updated_at = <Date 2015-10-26.08:44:45.247>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2015-10-26.08:44:45.247>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2015-10-26.08:44:45.248>
closer = 'eric.smith'
components = ['Library (Lib)']
creation = <Date 2015-10-04.17:23:27.578>
creator = 'skrah'
dependencies = []
files = ['40712', '40730', '40821']
hgrepos = []
issue_num = 25311
keywords = ['easy']
message_count = 13.0
messages = ['252274', '252275', '252295', '252479', '252485', '252522', '252610', '252613', '252619', '253109', '253236', '253461', '253463']
nosy_count = 5.0
nosy_names = ['eric.smith', 'skrah', 'python-dev', 'martin.panter', 'Nan Wu']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25311'
versions = ['Python 3.6']

[skrah]

I think tokenize.py needs to be updated to support f-strings.

BTW, the f-string implementation seems to be incredibly robust. Nice work!

[ericvsmith]

Thanks for noticing tokenize.py. And thanks for the kind note!

[vadmium]

I was just about to make the same bug report :) I guess it would be fine to tokenize F-strings as the same string objects as others, it probably just needs adding an F to the right regular expression.

$ ./python -btWall -m tokenize
"string"
1,0-1,8:            STRING         '"string"'     
1,8-1,9:            NEWLINE        '\n'           
b"string"
3,0-3,9:            STRING         'b"string"'    
3,9-3,10:           NEWLINE        '\n'           
f"string"
4,0-4,1:            NAME           'f'            
4,1-4,9:            STRING         '"string"'     
4,9-4,10:           NEWLINE        '\n'

[NanWu]

Added 'f'/'F' to the StringPrefix regex and also update the quote dictionary.

[vadmium]

Thanks for the patch. Do you want to try adding a test case. See TokenizeTest.test_string() at /Lib/test/test_tokenize.py:189 for a guide, though I would suggest a new test_fstring() method.

Also, F-strings can be combined with the raw string syntax. I wonder if you need to add support for things like rf". . ." and FR'''. . .'''.

[ericvsmith]

Yes, both 'fr' and 'rf' need to be supported (and all upper/lower variants). And in the future, maybe 'fb' (and 'rfb', 'bfr', ...).

Unfortunately, the regex doesn't scale well for all of the combinations.

[ericvsmith]

I think the best way to approach this is to generate (in code) all of the places where string prefixes appear. There's StringPrefix, endpats, triple_quotes, and single_quoted.

With the currently valid combinations of f, b, r, and u, I count 24 combinations:
['B', 'BR', 'Br', 'F', 'FR', 'Fr', 'R', 'RB', 'RF', 'Rb', 'Rf', 'U', 'b', 'bR', 'br', 'f', 'fR', 'fr', 'r', 'rB', 'rF', 'rb', 'rf', 'u']

If I add "fb" strings (plus raw), I count 72 combinations:
['B', 'BFR', 'BFr', 'BR', 'BRF', 'BRf', 'BfR', 'Bfr', 'Br', 'BrF', 'Brf', 'F', 'FBR', 'FBr', 'FR', 'FRB', 'FRb', 'FbR', 'Fbr', 'Fr', 'FrB', 'Frb', 'R', 'RB', 'RBF', 'RBf', 'RF', 'RFB', 'RFb', 'Rb', 'RbF', 'Rbf', 'Rf', 'RfB', 'Rfb', 'U', 'b', 'bFR', 'bFr', 'bR', 'bRF', 'bRf', 'bfR', 'bfr', 'br', 'brF', 'brf', 'f', 'fBR', 'fBr', 'fR', 'fRB', 'fRb', 'fbR', 'fbr', 'fr', 'frB', 'frb', 'r', 'rB', 'rBF', 'rBf', 'rF', 'rFB', 'rFb', 'rb', 'rbF', 'rbf', 'rf', 'rfB', 'rfb', 'u']

Coding these combinations by hand seems insane.

[ericvsmith]

Oops, make that 80 combinations (I forgot the various 'fb' ones):

['B', 'BF', 'BFR', 'BFr', 'BR', 'BRF', 'BRf', 'Bf', 'BfR', 'Bfr', 'Br', 'BrF', 'Brf', 'F', 'FB', 'FBR', 'FBr', 'FR', 'FRB', 'FRb', 'Fb', 'FbR', 'Fbr', 'Fr', 'FrB', 'Frb', 'R', 'RB', 'RBF', 'RBf', 'RF', 'RFB', 'RFb', 'Rb', 'RbF', 'Rbf', 'Rf', 'RfB', 'Rfb', 'U', 'b', 'bF', 'bFR', 'bFr', 'bR', 'bRF', 'bRf', 'bf', 'bfR', 'bfr', 'br', 'brF', 'brf', 'f', 'fB', 'fBR', 'fBr', 'fR', 'fRB', 'fRb', 'fb', 'fbR', 'fbr', 'fr', 'frB', 'frb', 'r', 'rB', 'rBF', 'rBf', 'rF', 'rFB', 'rFb', 'rb', 'rbF', 'rbf', 'rf', 'rfB', 'rfb', 'u']

import itertools as _itertools

def _all_string_prefixes():
    # The valid string prefixes. Only contain the lower case versions,
    #  and don't contain any permuations (include 'fr', but not
    #  'rf'). The various permutations will be generated.
    _valid_string_prefixes = ['b', 'r', 'u', 'f', 'br', 'fr', 'fb', 'fbr']
    result = set()
    for prefix in _valid_string_prefixes:
        for t in _itertools.permutations(prefix):
            # create a list with upper and lower versions of each
            #  character
            for u in _itertools.product(*[(c, c.upper()) for c in t]):
                result.add(''.join(u))
    return result

[ericvsmith]

My first attempt. Many more tests are needed.

I'm going to need to spend some time trying to figure out how parts of tokenize.py actually works. I'm not sure, for example, that endpats is initialized correctly. There definitely aren't enough tests, since if I comment out parts of endpats the tests still pass.

[ericvsmith]

Multi-line string tests were added in changeset 91c44dc35dfd. That will make changes for this issue safer. Updated patch to come.

[ericvsmith]

This patch cleans up string matching in tokenize.py, and adds f-string support.