gh-135676: Add a summary of source characters #138194
Draft
+76
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| The lexical analyzer determines the program text's :ref:`encoding <encodings>` | ||
| (UTF-8 by default), and decodes the text into | ||
| :ref:`source characters <lexical-source-character>`. | ||
| If the text cannot be decoded, a :exc:`SyntaxError` is raised. |
There was a problem hiding this comment.
Can this also not raise a Unicode*Error? For example, running a file with a surrogate:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 1-2: surrogates not allowed
| * formfeed | ||
| * * :ref:`Whitespace <whitespace>` | ||
|
|
||
| * * * CR, LF |
There was a problem hiding this comment.
Should we also list CRLF, as we list in the formal grammar:
newline: <ASCII LF> | <ASCII CR> <ASCII LF> | <ASCII CR>
| .. (the following uses zero-width-joiner characters to render | ||
| .. a literal backquote) | ||
|
|
||
| backquote (`````) |
There was a problem hiding this comment.
Could a substitution be used here? See example for nbsp in math.rst.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The lexical analysis docs have notes like this at the end:
The period can also occur in floating-point and imaginary literals.
The following printing ASCII characters have special meaning as part of other tokens or are otherwise significant to the lexical analyzer:
' " # \The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error:
$ ? `The intent behind these seems to be providing a "map" of what all the ASCII characters do in Python, but that map is incomplete as it is, and isn't really kept up to date.
This instead provides a summary of source characters -- nominally the ones that start tokens, with notes for other notable cases.
The table can also serve as an alternate "table of contents".
The presentation -- a table of bulleted lists -- is a bit wacky but I think it gets the job done.
📚 Documentation preview 📚: https://cpython-previews--138194.org.readthedocs.build/