Literal raw string/bytes backslashes are highlighted as escapes

[t-mart]

Environment data

• VS Code version:

Version: 1.48.2 (user setup)
Commit: a0479759d6e9ea56afa657e454193f72aef85bd0
Date: 2020-08-25T10:13:11.295Z
Electron: 7.3.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.19041

• Extension version (available under the Extensions sidebar): v2020.8.105369
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.5
• Type of virtual environment used: virtualenv stdlib package
• Relevant/affected Python packages and their versions: Python 3.8.5
• Relevant/affected Python-related VS Code extensions and their versions: v2020.8.105369
• Value of the python.languageServer setting: v2020.8.105369 Pylance language server 2020.8.3 (pyright 8c5415fd) (Sorry, unsure how to query this value, so previous values are just copy-pastes from extension page and Python Language Server output log.)

Expected behaviour

Backslashes (and the characters that follow them) in raw string literals and raw byte literals should not be highlighted. Highlighting in this case should be reserved for non-raw literals, where the backslashes create an escaped character.

Both string and bytes literals may optionally be prefixed with a letter 'r' or 'R'; such strings are called raw strings and treat backslashes as literal characters.

from https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

Actual behaviour

Backslashes are highlighted in these cases as if they are escape characters, which they are not.

Steps to reproduce:

1. Create a new python file and ensure the language server has started.
2. Insert this text into the file:

RAW_STRING = r"C:\Users\tim\code\imagesearch\tests\fixtures"
RAW_BYTES = rb"C:\Users\tim\code\imagesearch\tests\fixtures"

3. Note the errant escape highlighting.

[t-mart]

I was not sure if I posted this bug in the right place, so I dug around a little to try to find the token/grammar specification in this repo. I couldn't find anything. Maybe this comes from TextMate, the grammar provider for VSCode?

Also, to provide a little more context, I found a nifty command that brings up a tooltip about the tokenization that occurred at the current position in a document: Developer: Inspect Editor Tokens and Scopes (which can be accessed from the Ctrl + Shift + P menu). Here's what it shows for these escapes:

Perhaps this file https://github.com/textmate/python.tmbundle/blob/master/Syntaxes/Python.tmLanguage. This file defines the scopes/tokens with the same names as those in the tooltip above, which adds confidence that this is the right file. Also, I did see some specific handling for raw literals in there. However, these files are complex and I'm not sure how they work.

[brettcannon]

We don't control syntax highlighting for Python files. That comes from VS Code which gets it from MagicPython. It's best to report an issue there.

[t-mart]

Thanks for pointing me in the right direction @brettcannon.

For posterity, one can workaround to the expected behavior by using capitalized "R"s in front of the literals (even though this distinction is not part of the Python grammar).

More info here: MagicStack/MagicPython#114