Add a C API function to detect temporaries

[ngoldbaum]

Feature or enhancement

Proposal:

NumPy has an optimization to detect temporaries created via the NumPy C API (e.g. in NumPy internals) and elide them. This can lead to a significant performance improvement for some operations.

In numpy/numpy#28681, @colesbury proposed adding some code to handle the change to use stackrefs internally in CPython, which broke the NumPy temporary elision heuristics in 3.14.

We later added that code more or less verbatim to the NumPy main branch:

https://github.com/numpy/numpy/blob/d692fbccd98cb880812b32936e5f94fcfe55053f/numpy/_core/src/multiarray/temp_elide.c#L119-L152

This unblocks testing NumPy on the 3.14 beta but we should really have at least an unstable C API function we can call here rather than relying on CPython internals.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

#133140 (comment)

Linked PRs

• gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API #133170

[colesbury]

To add some more context, in NumPy, if you write:

x = np.array([ ... ])
foo(x)

In 3.14, x will have a reference count of 1 within the call to foo() (from the local variable x), whereas before 3.14, it would be at least 2. The reason is that interpreter now safely borrows references when it can, such as in function calls, if it knows that it already holds a strong reference. It's an optimization in the generated bytecode that replaces some LOAD_FAST bytecodes with the new LOAD_FAST_BORROW bytecode.

This means that in some contexts, it's not safe to assume that Py_REFCNT(x) == 1 means that you can modify x in-place. They weren't 100% true before because of the C API (see the explanation in numpy/numpy@e6c397b), but the 3.14 change definitely breaks some code.

We want to provide some C API that NumPy and other packages can uses to safely replace their Py_REFCNT(x) == 1 checks. Ideally, it should also avoid the need for backtrace() too.

The workaround for NumPy, which I think we should encapsulate in a new C API function, is to check that:

1. The reference count is 1
2. There exists a single temporary on the interpreter's stack that is a strong (not borrowed) reference to the relevant object

We might want to call this something like: PyUnstable_Object_IsUniqueTemporary(ob)

also cc @pablogsal

[ZeroIntensity]

Should this (and probably #133140) be a 3.14 blocker? That beta freeze is creeping up on us quickly.