-
-
Notifications
You must be signed in to change notification settings - Fork 31.6k
lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6 #68082
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc') This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is |
Should it? >>> rb'abc'
File "<stdin>", line 1
rb'abc'
^
SyntaxError: invalid syntax According to https://docs.python.org/2/reference/lexical_analysis.html#string-literals "rb" is not valid string prefix. |
Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that. rb'...' is valid in Python 3: $ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
b'abc'
>>> |
With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings. |
I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1. Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think? |
Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final. Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release. I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced. The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests. |
Alright, we'll merge this for 3.6.2 then! |
Should this issue be closed now? |
No Serhiy, I need a new patch with tests, etc. |
Misc/NEWS entries for these changes? |
Removes the "-g" option from "pydoc", for some reason; I believe (dmalcolm 2010-01-29) that this was introduced in this change: - fix pydoc (python#68082) in 2.2.1-12 as a response to the -g option needing TkInter installed (Red Hat Linux 8)
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: