Move PyArena C API to the internal C API

[vstinner]

BPO	43244
Nosy	@vstinner, @shihai1991
PRs	bpo-43244: Move pyarena.h from include/cpython to include/internal/. #24688 bpo-43244: Rename pycore_ast.h to pycore_ast_state.h #24907 bpo-43244: Add pycore_ast.h header file #24908 bpo-43244: Remove symtable.h header file #24910 bpo-43244: Remove PyAST_Validate() function #24911 bpo-43244: test_peg_generator defines _Py_TEST_PEGEN macro #24912 bpo-43244: Fix test_peg_generators on Windows #24913 bpo-43244: GitHub Action builds Python in debug mode on Windows #24914 bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers #24933 bpo-43244: Add pycore_compile.h header file #25000 bpo-43244: Remove parser_interface.h header file #25001 bpo-43244: Remove pyarena.h header #25007 bpo-43244: Fix compiler warning in compile.c about Yield #25242 bpo-43244: Remove Yield macro from pycore_ast.h #25243 bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() #25252 bpo-43244: Remove aliases from pycore_ast.h #25256

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-24.01:26:54.289>
created_at = <Date 2021-02-17.18:31:30.182>
labels = ['expert-C-API', '3.10']
title = 'Move PyArena C API to the internal C API'
updated_at = <Date 2021-05-26.22:44:01.883>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-26.22:44:01.883>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-24.01:26:54.289>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-02-17.18:31:30.182>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43244
keywords = ['patch']
message_count = 19.0
messages = ['387180', '388973', '388976', '388981', '389001', '389016', '389018', '389075', '389408', '389416', '389419', '389423', '389424', '389501', '390416', '390444', '390472', '390473', '394489']
nosy_count = 2.0
nosy_names = ['vstinner', 'shihai1991']
pr_nums = ['24688', '24907', '24908', '24910', '24911', '24912', '24913', '24914', '24933', '25000', '25001', '25007', '25242', '25243', '25252', '25256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43244'
versions = ['Python 3.10']

[vstinner]

Include/cpython/pyarena.h declares 4 functions which are implemented in Python/pyarena.c. The header file exists to access these functions from the compiler. IMO there was no intent to expose these functions to the public Python C API.

Moreover, the PEP-384 explicitly excludes this API from the stable ABI (from the limited C API).

I propose to move these functions to the internal C API.

Since these functions are no even documented, I don't think that a deprecation period is needed.

Note: The PR 24550 moved Include/pyarena.h to Include/cpython/pyarena.h.

[vstinner]

New changeset b4536e1 by Victor Stinner in branch 'master':
bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907)
b4536e1

[vstinner]

New changeset 526fdeb by Victor Stinner in branch 'master':
bpo-43244: Add pycore_ast.h header file (GH-24908)
526fdeb

[vstinner]

New changeset e0bf70d by Victor Stinner in branch 'master':
bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
e0bf70d

[vstinner]

New changeset 6af528b by Victor Stinner in branch 'master':
bpo-43244: Fix test_peg_generators on Windows (GH-24913)
6af528b

[vstinner]

New changeset eec8e61 by Victor Stinner in branch 'master':
bpo-43244: Remove the PyAST_Validate() function (GH-24911)
eec8e61

[vstinner]

The work on this issue started in Python 3.9 with bpo-21120 which excluded Python-ast.h, ast.h and asdl.h from the limited C API: commit 421a72a.

[vstinner]

New changeset 28ad12f by Victor Stinner in branch 'master':
bpo-43244: Remove symtable.h header file (GH-24910)
28ad12f

[vstinner]

New changeset 94faa07 by Victor Stinner in branch 'master':
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
94faa07

[vstinner]

New changeset a81fca6 by Victor Stinner in branch 'master':
bpo-43244: Add pycore_compile.h header file (GH-25000)
a81fca6

[vstinner]

New changeset 57364ce by Victor Stinner in branch 'master':
bpo-43244: Remove parser_interface.h header file (GH-25001)
57364ce

[vstinner]

New changeset 8370e07 by Victor Stinner in branch 'master':
bpo-43244: Remove the pyarena.h header (GH-25007)
8370e07

[vstinner]

Ok, the whole C API related to parser, compiler, AST, ASDL and PyArena moved to the internal C API. So we will have more freedom to evolve it!

[vstinner]

In short, this issue is a follow-up of bpo-40939.

These C API removal are related to the final step of the PEP-617 "New PEG parser for CPython": removal of the PyParser C API.

• The PyParser C API produced "node*" objects, like PyParser_SimpleParseFile(FILE*) => node*. This C API was removed in Python 3.10: bpo-40939 "Remove the old parser".

• The AST C API takes "node*" as input: PyAST_FromNode(node*) => mod_ty. I removed these APIs.

• symtable takes "mod_ty" as input.

• ASDL and PyArena API were only useful for the removed AST C APIs.

I searched for removed functions in the top PyPI 4000 projects, there is a single project on 4000: typed_ast. This typed_ast project is special, it copies directly code from CPython. Even after the C API removed, typed_ast can continue to use the internal C API, it only has to define the Py_BUILD_CORE_MODULE macro.

[vstinner]

New changeset d36d6a9 by Victor Stinner in branch 'master':
bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)
d36d6a9

[vstinner]

typed-ast was updated for these changes and to avoid including explicitly pydebug.h (it's included by Python.h: see bpo-35134 change): python/typed_ast#158

[vstinner]

New changeset d27f8d2 by Victor Stinner in branch 'master':
bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() (GH-25252)
d27f8d2

[vstinner]

"Remove Yield macro from pycore_ast.h"

I'm happy that this one is gone, it was a common and annoying compiler warning on Windows!

[vstinner]

I marked bpo-35197 as a duplicate of this issue.