lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6

[elibendersky]

BPO	23894
Nosy	@gpshead, @benjaminp, @ned-deily, @jab, @ambv, @serhiy-storchaka
PRs	bpo-23894: lib2to3 doesn't recognize rb'...' and f'...' #593 bpo-23894: Make rb'' strings work in lib2to3 #1724 [3.6] bpo-23894: Make rb'' strings work in lib2to3 (GH-1724) #1730 bpo-23894: make lib2to3 recognize f-strings #1733 [3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) #1737 bpo-23894: add Misc/NEWS entry. #2255 bpo-23894: add Misc/NEWS entry. #2256
Files	issue23894.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2017-05-23.00:04:34.799>
created_at = <Date 2015-04-09.13:18:15.097>
labels = ['easy', 'type-bug', '3.7']
title = "lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6"
updated_at = <Date 2017-06-17.02:43:33.286>
user = 'https://bugs.python.org/elibendersky'

bugs.python.org fields:

activity = <Date 2017-06-17.02:43:33.286>
actor = 'ned.deily'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2017-05-23.00:04:34.799>
closer = 'lukasz.langa'
components = []
creation = <Date 2015-04-09.13:18:15.097>
creator = 'eli.bendersky'
dependencies = []
files = ['41382']
hgrepos = []
issue_num = 23894
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['240322', '240444', '240445', '256831', '289343', '289415', '289423', '291321', '291572', '294196', '294202', '294203', '294223', '295836', '296227', '296228']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'benjamin.peterson', 'ned.deily', 'eli.bendersky', 'jab', 'lukasz.langa', 'serhiy.storchaka', 'josephgordon']
pr_nums = ['593', '1724', '1730', '1733', '1737', '2255', '2256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23894'
versions = ['Python 3.6', 'Python 3.7']

[elibendersky]

lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is

[serhiy-storchaka]

Should it?

>>> rb'abc'
  File "<stdin>", line 1
    rb'abc'
          ^
SyntaxError: invalid syntax

According to https://docs.python.org/2/reference/lexical_analysis.html#string-literals "rb" is not valid string prefix.

[elibendersky]

Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that.

rb'...' is valid in Python 3:

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
b'abc'
>>>

[josephgordon]

With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings.

[ambv]

I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1.

Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think?

[ned-deily]

Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final. Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release. I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced. The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests.

[ambv]

Alright, we'll merge this for 3.6.2 then!

[serhiy-storchaka]

Should this issue be closed now?

[ambv]

No Serhiy, I need a new patch with tests, etc.

[ambv]

New changeset 0c4aca5 by Łukasz Langa in branch 'master':
Make rb'' strings work in lib2to3 (bpo-1724)
0c4aca5

[ambv]

New changeset 1398b1b by Łukasz Langa in branch '3.6':
[3.6] Make rb'' strings work in lib2to3 (GH-1724) (bpo-1730)
1398b1b

[ambv]

New changeset 1b9530c by Łukasz Langa in branch 'master':
bpo-23894: make lib2to3 recognize f-strings (bpo-1733)
1b9530c

[ambv]

New changeset e8412e6 by Łukasz Langa in branch '3.6':
[3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) (bpo-1737)
e8412e6

[ned-deily]

Misc/NEWS entries for these changes?

[ned-deily]

New changeset b7c59f0 by Ned Deily in branch '3.6':
bpo-23894: add Misc/NEWS entry. (bpo-2255)
b7c59f0

[ned-deily]

New changeset ceb817d by Ned Deily in branch 'master':
bpo-23894: add Misc/NEWS entry. (bpo-2256)
ceb817d