Document functions safe to be called before Py_Initialize()

[vstinner]

BPO	32124
Nosy	@ncoghlan, @vstinner, @ericsnowcurrently, @serhiy-storchaka
PRs	bpo-32124: Document C functions safe before init #4540 bpo-32124: C API preinit doc, change note formatting #4621

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-15.15:09:10.756>
created_at = <Date 2017-11-24.13:19:15.972>
labels = ['3.7', 'docs']
title = 'Document functions safe to be called before Py_Initialize()'
updated_at = <Date 2017-12-15.15:09:10.756>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-12-15.15:09:10.756>
actor = 'vstinner'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-12-15.15:09:10.756>
closer = 'vstinner'
components = ['Documentation']
creation = <Date 2017-11-24.13:19:15.972>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 32124
keywords = ['patch']
message_count = 10.0
messages = ['306894', '306898', '306902', '306912', '306916', '306917', '306921', '306922', '306925', '306937']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'vstinner', 'docs@python', 'eric.snow', 'serhiy.storchaka']
pr_nums = ['4540', '4621']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue32124'
versions = ['Python 3.7']

[vstinner]

Follow-up of bpo-32086, bpo-32096 and "[Python-Dev] Python initialization and embedded Python" thread:
https://mail.python.org/pipermail/python-dev/2017-November/150605.html

I propose to explicitly list functions that can be safetely called before Py_Initialize(). This safety warranty must be part of the C API.

Maybe we should even test all tests function in test_capi using Programs/_testembed, as we did for Py_DecodeLocale() and Py_SetProgramName() in commit 9e87e77 ("pre_initialization_api" test).

Attached PR adds proposed documentation. It also documents "global configuration variables" like Py_DebugFlag.

[serhiy-storchaka]

Are you sure about PyMem_Malloc() and PyObject_Malloc()? What functions require them? I thought only PyMem_RawMalloc() can be called before Py_Initialize().

I think that for all functions that *can* or *should* be called before Py_Initialize() this should be explicitly documented in the documentation of this function, like for Py_SetProgramName() and PyImport_AppendInittab().

[vstinner]

Are you sure about PyMem_Malloc() and PyObject_Malloc()?

Technically, the pymalloc memory allocator is initialized statically by the compiler, from the first instruction of the process.

But maybe we should not suggest users to call them, especially because the allocator can be modified by the PYTHONMALLOC environment variable.

What functions require them?

No function to initalize Python require PyMem or PyObject allocators. Only PyMem_Raw allocator is needed.

Py_EncodeLocale() uses it, but this function also uses Python objects (str, bytes), and so Py_EncodeLocale() must no be called before Py_Initialize().

[serhiy-storchaka]

Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()? There are other functions and macros that can be safely used before Py_Initialize().

[vstinner]

Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

I'm quite sure that calling PyMem_SetAllocator() or PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()?

We don't have to support them.

Ok, I remove them from the pre-init documentation.

[serhiy-storchaka]

I'm quite sure that calling PyMem_SetAllocator() or
PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

Then document this explicitly like for other functions that *should* be called
before Py_Initialize() if called at all.

[vstinner]

New changeset 84c4b19 by Victor Stinner in branch 'master':
bpo-32124: Document C functions safe before init (bpo-4540)
84c4b19

[vstinner]

Then document this explicitly like for other functions that *should* be called before Py_Initialize() if called at all.

I agree that it would be even better to document if a function must not be called after Py_Initialize().

*But* I'm not sure of what I wrote, I have to check the code, and maybe even test manually to "see what happens" (ensure that it works) :-)

So I decided to push my first PR, and will work on a second PR later.

[ericsnowcurrently]

I've left a review (writing it as you merged the PR).

My main concern is that we not promise more than we must. Every pre-init function or variable we promise to embedders represents global state that is hard to get rid of. It also entrenches pre-init API and state that we're aiming to deprecate (via PEP-432).

[ncoghlan]

Key point to note regarding PEP-432: at least personally, I'm not actually aiming to deprecate the legacy embedding API.

Instead, I'm just aiming to eventually stop *adding* to it, with new config structs replacing the current ad hoc mix of pre-init function calls, C globals, environment variables, and filesystem state.

That means I'm quite willing to accept maintaining compatibility for applications using the current single phase initialisation approach as a design constraint for the PEP.

We have a similar constraint in place for extension modules:
even though any *new* features we introduce are likely to be dependent on switching over to PEP-489's multi-phase initialisation APIs, we still ensure that single-phase initialisation continues working for existing modules.