issue with py3.14 Py_REFCNT in Limited API

[da-woods]

Bug report

Bug description:

cpython/Include/refcount.h

Lines 95 to 118 in 46f11b3

	// Py_REFCNT() implementation for the stable ABI
	PyAPI_FUNC(Py_ssize_t) Py_REFCNT(PyObject *ob);
	#if defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030e0000
	// Stable ABI implements Py_REFCNT() as a function call
	// on limited C API version 3.14 and newer.
	#else
	static inline Py_ssize_t _Py_REFCNT(PyObject *ob) {
	#if !defined(Py_GIL_DISABLED)
	return ob->ob_refcnt;
	#else
	uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
	if (local == _Py_IMMORTAL_REFCNT_LOCAL) {
	return _Py_IMMORTAL_INITIAL_REFCNT;
	}
	Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
	return _Py_STATIC_CAST(Py_ssize_t, local) +
	Py_ARITHMETIC_RIGHT_SHIFT(Py_ssize_t, shared, _Py_REF_SHARED_SHIFT);
	#endif
	}
	#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
	# define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))
	#endif
	#endif

In Python 3.14 Py_REFCNT is moved to become a function in the stable ABI. This works fine. If you use a Py_LIMITED_API version less that 3.14 then it generates _Py_REFCNT as an inline function.

On Limited API versions less than 3.11 then it defined Py_REFCNT to be _Py_REFCNT plus a cast. However on Py_LIMITED_API versions between 3.11 and 3.14 then it doesn't work - Py_REFCNT is defined as a function but isn't in the stable ABI.

See wjakob/nanobind#1183 for this in the wild.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-140153: Fix Py_REFCNT() definition on limited C API 3.11-3.13 #140158
• [3.14] gh-140153: Fix Py_REFCNT() definition on limited C API 3.11-3.13 (GH-140158) #140164

[sergey-miryanov]

I'm not sure it is safe to work with object that has refcnt == 0. It should be already destroyed for this moment.

[colesbury]

I'm not sure it is safe to work with object that has refcnt == 0

This is fine. There are a number of legitimate uses of objects with refcnt == 0, such as in tp_dealloc handlers.

[colesbury]

Here's the 3.13 definition where Py_REFCNT is always available as an inline function in the header. The macro enables the implicit cast of the argument to PyObject* for Py_LIMITED_API versions <3.11:

cpython/Include/object.h

Lines 316 to 331 in 7797676

	static inline Py_ssize_t Py_REFCNT(PyObject *ob) {
	#if !defined(Py_GIL_DISABLED)
	return ob->ob_refcnt;
	#else
	uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
	if (local == _Py_IMMORTAL_REFCNT_LOCAL) {
	return _Py_IMMORTAL_REFCNT;
	}
	Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
	return _Py_STATIC_CAST(Py_ssize_t, local) +
	Py_ARITHMETIC_RIGHT_SHIFT(Py_ssize_t, shared, _Py_REF_SHARED_SHIFT);
	#endif
	}
	#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
	# define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob))
	#endif

For 3.14, we need the macro for renaming to so we can either do something like:

     #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030e0000
     #  define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob)) 
     #endif 

Or if we care about enforcing the type cast:

     #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
     #  define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob)) 
     #elif defined(Py_LIMITED_API) && Py_LIMITED_API+0 < 0x030e0000
     #  define Py_REFCNT(ob) _Py_REFCNT(ob) 
     #endif 

cc @vstinner @encukou

[vstinner]

Oh no, sorry it's my fault! I wrote #140158 to fix this issue.

As a workaround, you can try to add these lines to your code:

#if (defined(Py_LIMITED_API) \
     && 0x030b0000 <= Py_LIMITED_API+0 && Py_LIMITED_API+0 < 0x030e0000 \
     && !defined(Py_REFCNT))
#  define Py_REFCNT(ob) _Py_REFCNT(ob)
#endif

[vstinner]

The regression was caused by:

In the limited C API version 3.14 and newer, Py_TYPE() and Py_REFCNT() are now implemented as an opaque function call to hide implementation details. (Contributed by Victor Stinner in gh-120600 and gh-124127.)

Hopefully, Py_TYPE() doesn't have the bug:

#if defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030e0000
    // Stable ABI implements Py_TYPE() as a function call
    // on limited C API version 3.14 and newer.
#else
    static inline PyTypeObject* _Py_TYPE(PyObject *ob)
    {
        return ob->ob_type;
    }
    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
    #   define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST(ob))
    #else
    #   define Py_TYPE(ob) _Py_TYPE(ob)
    #endif
#endif