__Pyx_PyType_Ready() can garble memory

[pitrou]

After some debugging, we found out that __Pyx_PyType_Ready can garble arbitrary memory.
Indeed, it has the following lines:

#if PY_VERSION_HEX >= 0x03050000
    t->tp_flags |= Py_TPFLAGS_HEAPTYPE;
#endif
    r = PyType_Ready(t);
#if PY_VERSION_HEX >= 0x03050000
    t->tp_flags &= ~Py_TPFLAGS_HEAPTYPE;
#endif

The problem is if PyType_Ready indirectly triggers the garbage collector. The static type t will be incorrectly considered as a heap type, and therefore as a GC-enabled object. When traversing this object, the GC will lookup the GC header by subtracting 16 bytes from t's beginning, which may point to any other C static variable. Then it will temporarily "decref" the GC header, modifying other data.
(in our case, that other data is a Cython function's ml_flags, rendering it unusable, but that might be something else, depending what your Cython module contains)

[pitrou]

@scoder

[pitrou]

Perhaps a possible solution is for Cython to allocate static types like this:

typedef struct {
    PyGC_Head gc_head;
    PyTypeObject typeobj;
} SafeStaticType;

static SafeStaticType __pyx_safetype_7pyarrow_3lib_KeyValueMetadata = {
    .gc = {
        .gc_refs = -2  /* _PyGC_REFS_UNTRACKED */
    },
    .typeobj = {
        PyVarObject_HEAD_INIT(0, 0)
        /* etc. */
    }
};

static PyTypeObject* __pyx_type_7pyarrow_3lib_KeyValueMetadata = \
    &__pyx_safetype_7pyarrow_3lib_KeyValueMetadata.typeobj;

[pitrou]

In the meantime, we're probably going to disable the GC on module initialization:
https://github.com/apache/arrow/pull/7160/files

[scoder]

Lovely. Thanks for the investigations.
Here is the full code. Note the comment.

cython/Cython/Utility/ExtensionTypes.c

Lines 57 to 72 in 94ea7d0

	#if PY_VERSION_HEX >= 0x03050000
	// As of https://bugs.python.org/issue22079
	// PyType_Ready enforces that all bases of a non-heap type are
	// non-heap. We know that this is the case for the solid base but
	// other bases are heap allocated and are kept alive through the
	// tp_bases reference.
	// Other than this check, the Py_TPFLAGS_HEAPTYPE flag is unused
	// in PyType_Ready().
	t->tp_flags |= Py_TPFLAGS_HEAPTYPE;
	#endif
	r = PyType_Ready(t);
	#if PY_VERSION_HEX >= 0x03050000
	t->tp_flags &= ~Py_TPFLAGS_HEAPTYPE;
	#endif

[pitrou]

Yeah, I saw the comment. I think the BPO issue would deserve a proper fix, but I'm not involved very much nowadays. Perhaps @jdemeyer feels brave enough? :-)

[pitrou]

Perhaps Cython should always use PyType_FromSpec, not always in limited API builds?

[scoder]

Perhaps Cython should always use PyType_FromSpec

Yes, I was thinking that anyway. The main reason for keeping the current static types is Py2.7 support. That'll be ripped out in Cython 3.1, shortly after the 3.0 release. But we could already start by switching to heap types for Py3.4+ in Cython 3.0. (No, this will not change in 0.29.x any more.)

We could also mitigate the issue by only setting the heaptype flag if there actually are heap types involved, I think. That could also be done in 0.29.x.
Edit: turns out, this code is really only run if there are heap type bases.

[scoder]

And regarding always using PyType_FromSpec() in Py3, then we run into bugs like https://bugs.python.org/issue38140
So, I guess the answer is no, we can't currently use PyType_FromSpec(), at least not for Py<3.9.

[scoder]

Anyway, I started splitting the current Limited-API special-casing into separate feature flags in #3611. Let's see where that gets us. We might be able to work around the limitations of PyType_FromSpec() when we're not limited to the limited API.

[scoder]

… and, obviously, PyType_FromSpec() calls PyType_Ready() internally, so that's even worse in the end. :-/

Edit: I keep getting confused by all of those different APIs and their impact. When using PyType_FromSpec(), then the resulting extension type is also a heap type, and we won't have this problem. Back to #3611.

[da-woods]

It's quite likely I'm missing something, but is there a reason not to turn off the GC within __Pyx_PyTypeReady? It might be possible to do in C but even if not then it's only a module import and Python function call.