|
1 | 1 | .. _grammar:
|
2 | 2 |
|
3 |
| -========================== |
4 |
| -Changing CPython's grammar |
5 |
| -========================== |
6 |
| - |
7 |
| -Abstract |
8 |
| -======== |
9 |
| - |
10 |
| -There's more to changing Python's grammar than editing |
11 |
| -:cpy-file:`Grammar/python.gram`. Here's a checklist. |
12 |
| - |
13 |
| -.. note:: |
14 |
| - These instructions are for Python 3.9 and beyond. Earlier |
15 |
| - versions use a different parser technology. You probably shouldn't |
16 |
| - try to change the grammar of earlier Python versions, but if you |
17 |
| - really want to, use GitHub to track down the earlier version of this |
18 |
| - file in the devguide. |
19 |
| - |
20 |
| -For more information on how to use the new parser, check the |
21 |
| -:ref:`section on how to use CPython's parser <parser>`. |
22 |
| - |
23 |
| -Checklist |
24 |
| -========= |
25 |
| - |
26 |
| -Note: sometimes things mysteriously don't work. Before giving up, try ``make clean``. |
27 |
| - |
28 |
| -* :cpy-file:`Grammar/python.gram`: The grammar, with actions that build AST nodes. |
29 |
| - After changing it, run ``make regen-pegen`` (or ``build.bat --regen`` on Windows), |
30 |
| - to regenerate :cpy-file:`Parser/parser.c`. |
31 |
| - (This runs Python's parser generator, :cpy-file:`Tools/peg_generator`). |
32 |
| - |
33 |
| -* :cpy-file:`Grammar/Tokens` is a place for adding new token types. After |
34 |
| - changing it, run ``make regen-token`` to regenerate |
35 |
| - :cpy-file:`Include/internal/pycore_token.h`, :cpy-file:`Parser/token.c`, |
36 |
| - :cpy-file:`Lib/token.py` and :cpy-file:`Doc/library/token-list.inc`. |
37 |
| - If you change both ``python.gram`` and ``Tokens``, |
38 |
| - run ``make regen-token`` before ``make regen-pegen``. |
39 |
| - On Windows, ``build.bat --regen`` will regenerate both at the same time. |
40 |
| - |
41 |
| -* :cpy-file:`Parser/Python.asdl` may need changes to match the grammar. |
42 |
| - Then run ``make regen-ast`` to regenerate |
43 |
| - :cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`. |
44 |
| - |
45 |
| -* :cpy-file:`Parser/lexer/` contains the tokenization code. |
46 |
| - This is where you would add a new type of comment or string literal, for example. |
47 |
| - |
48 |
| -* :cpy-file:`Python/ast.c` will need changes to validate AST objects |
49 |
| - involved with the grammar change. |
50 |
| - |
51 |
| -* :cpy-file:`Python/ast_unparse.c` will need changes to unparse AST |
52 |
| - involved with the grammar change ("unparsing" is used to turn annotations |
53 |
| - into strings per :pep:`563`). |
54 |
| - |
55 |
| -* The :ref:`compiler` has its own page. |
56 |
| - |
57 |
| -* ``_Unparser`` in the :cpy-file:`Lib/ast.py` file may need changes |
58 |
| - to accommodate any modifications in the AST nodes. |
59 |
| - |
60 |
| -* :cpy-file:`Doc/library/ast.rst` may need to be updated to reflect changes |
61 |
| - to AST nodes. |
62 |
| - |
63 |
| -* Add some usage of your new syntax to ``test_grammar.py``. |
64 |
| - |
65 |
| -* Certain changes may require tweaks to the library module :mod:`pyclbr`. |
66 |
| - |
67 |
| -* :cpy-file:`Lib/tokenize.py` needs changes to match changes to the tokenizer. |
68 |
| - |
69 |
| -* Documentation must be written! Specifically, one or more of the pages in |
70 |
| - :cpy-file:`Doc/reference/` will need to be updated. |
| 3 | +This document is now part of the |
| 4 | +`CPython Internals Docs <https://github.com/python/cpython/blob/main/InternalDocs/changing_grammar.md>`_. |
0 commit comments