Isolate the _io extension module

[erlend-aasland]

Isolate the _io extension module by moving all global variables to module state, porting static types to heap types, and implementing multi-phase init. All global variables in the _io module are static types:

• Modules/_io/bufferedio.c: PyBufferedIOBase_Type
• Modules/_io/bufferedio.c: PyBufferedRWPair_Type
• Modules/_io/bufferedio.c: PyBufferedRandom_Type
• Modules/_io/bufferedio.c: PyBufferedReader_Type
• Modules/_io/bufferedio.c: PyBufferedWriter_Type
• Modules/_io/bytesio.c: PyBytesIO_Type
• Modules/_io/bytesio.c: _PyBytesIOBuffer_Type
• Modules/_io/fileio.c: PyFileIO_Type
• Modules/_io/iobase.c: PyIOBase_Type
• Modules/_io/iobase.c: PyRawIOBase_Type
• Modules/_io/textio.c: PyIncrementalNewlineDecoder_Type
• Modules/_io/textio.c: PyTextIOBase_Type
• Modules/_io/textio.c: PyTextIOWrapper_Type
• Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type

Converting the static types to heap types involves applying PEP-687 to _io.
Adapting multi-phase init involves applying PEP-489 to _io.

Linked PRs

• gh-101819: Remove _PyWindowsConsoleIO_Type from the Windows DLL #101904
• gh-101819: Remove _testcapi dependencies on specific _io symbols #101918
• gh-101819: Isolate _io #101948
• gh-101819: Adapt _io types to heap types, batch 1 #101949
• gh-101819: Prepare to modernize the _io extension #104178
• gh-101819: Adapt _io.PyWindowsConsoleIO_Type to heap type #104197
• gh-101819: Remove unused 'locale_module' from _io state #104246
• gh-101819: Port _io.PyIncrementalNewlineDecoder_Type to heap type #104249
• gh-101819: Port _io.PyBytesIOBuffer_Type to heap type #104264
• gh-101819: Refactor _io in preparation for module isolation #104334
• gh-101819: Harden _io init #104352
• gh-101819: Clean up _io windows console io after gh-104197 #104354
• gh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic #104355
• gh-101819: Adapt _io._Buffered* methods to Argument Clinic #104367
• gh-101819: Refactor _io futher in preparation for module isolation #104369
• gh-101819: Explicitly disallow pickle protocols 0 and 1 in _io #104370
• gh-101819: Adapt _io.TextIOBase methods to Argument Clinic #104383
• gh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic #104384
• gh-101819: Prepare _io._IOBase for module state #104386
• gh-101819: Fix inverted debug preprocessor check in winconsoleio.c #104388
• gh-101819: Fix _io clinic input for unused base class method stubs #104418

[erlend-aasland]

cc. @kumaraditya303

[erlend-aasland]

I see that _PyWindowsConsoleIO_Type is marked with PyAPI_DATA, but I can't find it in Misc/stable_abi.toml. @zooba or @eryksun, do you know why this is so? I did a quick code search on GitHub, but I only got hits in various clones of the CPython code base.

cpython/Modules/_io/_iomodule.h

Lines 24 to 30 in 2db2c4b

	#ifndef Py_LIMITED_API
	#ifdef MS_WINDOWS
	extern PyTypeObject PyWindowsConsoleIO_Type;
	PyAPI_DATA(PyObject *) _PyWindowsConsoleIO_Type;
	#define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), (PyTypeObject*)_PyWindowsConsoleIO_Type))
	#endif /* MS_WINDOWS */
	#endif /* Py_LIMITED_API */

UPDATE: I see it was added by Steve in 312cef7, probably because of the PyWindowsConsoleIO_Check checks in PC/_testconsole.c and Python/pylifecycle.c:

cpython/PC/_testconsole.c

Lines 54 to 57 in 2db2c4b

	if (!PyWindowsConsoleIO_Check(file)) {
	PyErr_SetString(PyExc_TypeError, "expected raw console object");
	return NULL;
	}

cpython/Python/pylifecycle.c

Lines 58 to 61 in 2db2c4b

	extern PyTypeObject PyWindowsConsoleIO_Type;
	# define PyWindowsConsoleIO_Check(op) \
	(PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type))
	#endif

cpython/Python/pylifecycle.c

Lines 2359 to 2363 in 2db2c4b

	#ifdef MS_WINDOWS
	/* Windows console IO is always UTF-8 encoded */
	if (PyWindowsConsoleIO_Check(raw))
	encoding = L"utf-8";
	#endif

Perhaps exposing _PyWindowsConsoleIO_Type through a CPython internal capsulated C API could be an option?

[erlend-aasland]

Other issues that must be resolved before continuing: _PyBytesIOBuffer_Type is used by _testcapimodule. Since that type is not exposed in _io, we cannot simply use PyImport_Import and look it up. One possibility is to expose it through a CPython internal capsule C API.

[erlend-aasland]

... or we could simply expose the types mentioned in #101819 (comment) and #101819 (comment) in the _io module.

[zooba]

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

[encukou]

The type is explicitly not part of the limited API/stable ABI, see #ifndef Py_LIMITED_API.

[erlend-aasland]

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

IIUC, we can remove this from the DLL and instead export it via _io and look it up from the module using the C API (PyImport_Import, etc.).

[zooba]

Yeah, if it's got a leading underscore then we can do whatever we need to make our tests work.

[vstinner]

@erlend-aasland erlend-aasland closed this as completed

Great achievement!

[serhiy-storchaka]

It causes a crash during garbage collection. See #111049, #111211.