chokes on rb"literal"

[jab]

yapf currently chokes on raw bytestring literals:

bash-4.3$ cat /tmp/rb_str_literal.py
rb'literal'
bash-4.3$ python3 -m yapf -i /tmp/rb_str_literal.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/pytree_utils.py", line 96, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'literal'", context=('', (1, 2))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.4/site-packages/yapf/__main__.py", line 18, in <module>
    sys.exit(yapf.main(sys.argv))
  File "/usr/local/lib/python3.4/site-packages/yapf/__init__.py", line 109, in main
    print_diff=args.diff, verify=args.verify)
  File "/usr/local/lib/python3.4/site-packages/yapf/__init__.py", line 133, in FormatFiles
    print_diff=print_diff, verify=verify)
  File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 72, in FormatFile
    verify=verify)
  File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 95, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n')
  File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/pytree_utils.py", line 101, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'literal'", context=('', (1, 2))

This is using revision 7a01d00.

[bwendling]

The problem is that the original code isn't correct. "rb" isn't accepted by "br" is. One of the limitations of lib2to3 (and therefore YAPF) is that it requires the code to be well-formed. Here's what I saw:

>>> rb'hello world'
  File "<stdin>", line 1
    rb'hello world'
                  ^
SyntaxError: invalid syntax
>>> br'hello world'
'hello world'
>>> 

[jab]

$ python3
Python 3.4.3 (default, Mar  1 2015, 15:44:16)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'this is valid python3 syntax'
b'this is valid python3 syntax'

Useful for e.g. not having to escape backslashes in bytestrings.

[bwendling]

Ah! My mistake. I was using 2.7.

[bwendling]

Marking this as a bug, but I'm not sure what we can do here. Might be the same issue as yield from?

[eliben]

Yeah, I'll have to see what we do about places where lib2to3 fails. Maybe deeper monkey-patching will help

[eliben]

This is an upstream lib2to3 bug. Reported http://bugs.python.org/issue23894

Monkey patching this in lib2to3 is pretty complex -- I don't see this as very high priority at this time.

[eliben]

Adding a lib2to3 bug label.

[bwendling]

Python 3 seems to have fixed this.