Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode are not available with limited API

[serhiy-storchaka]

BPO	41986
Nosy	@vstinner, @serhiy-storchaka, @miss-islington
PRs	bpo-41986: Add Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode back to limited API #22621 [3.9] bpo-41986: Add Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode back to limited API (GH-22621) #22637 [3.8] bpo-41986: Add Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode back to limited API (GH-22621) #22638

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-10.19:28:11.336>
created_at = <Date 2020-10-09.19:48:03.658>
labels = ['expert-C-API', 'type-bug', '3.8', '3.9', '3.10']
title = 'Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode are not available with limited API'
updated_at = <Date 2020-10-10.19:28:11.335>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-10-10.19:28:11.335>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-10.19:28:11.336>
closer = 'serhiy.storchaka'
components = ['C API']
creation = <Date 2020-10-09.19:48:03.658>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 41986
keywords = ['patch']
message_count = 7.0
messages = ['378340', '378399', '378401', '378404', '378405', '378407', '378408']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'miss-islington']
pr_nums = ['22621', '22637', '22638']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41986'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[serhiy-storchaka]

Py_FileSystemDefaultEncodeErrors was added to limited API in 3.7 and Py_UTF8Mode -- in 3.7. But in 3.8 their declarations are not available if Py_LIMITED_API is defined because they were moved to different header file included only if Py_LIMITED_API is not defined.

[vstinner]

I consider the global configuration variables as deprecated. I would prefer to slowly move towards PyConfig, since almost all Python functions now use PyInterpreterState.config.

The problem is that my idea to provide a stable ABI for PyConfig was rejected:
https://mail.python.org/archives/list/python-dev@python.org/thread/C7Z2NA2DTM3DLOZCFQAK5A2WFYO3PHHX/#2JAJQA5OANFPXAJ3327RRPHPQLKVP2EW

Which problem are you trying to solve your PR 22621?

[serhiy-storchaka]

It solves the breaking of the C API. Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode were not deprecated, they were just suddenly excluded from the limited API. And seems this change was not intentional, otherwise surrounding #ifdef/#endif (which currently do not have effect) would be not preserved.

[vstinner]

Python 3.7 defines it in fileobject.h as:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000
PyAPI_DATA(const char *) Py_FileSystemDefaultEncodeErrors;
#endif

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_DATA(int) Py_UTF8Mode;
#endif

Python 3.8 defines them the same way, but in Include/cpython/fileobject.h:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000
PyAPI_DATA(const char *) Py_FileSystemDefaultEncodeErrors;
#endif

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_DATA(int) Py_UTF8Mode;
#endif

I am likely the one who moved these definitions. It was a mistake to move them inside Include/cpython/, since "Py_LIMITED_API+0 >= 0x03070000" became useless: cpython/fileobject.h is only included if Py_LIMITED_API is not defined.

In Include/cpython/, it seems like only 2 definitions are defined the wrong way, Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode.

[serhiy-storchaka]

New changeset 637a09b by Serhiy Storchaka in branch 'master':
bpo-41986: Add Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode back to limited API (GH-22621)
637a09b