C API: Add PyObject_AsObjectArray() function: get tuple/list items as PyObject** array

[vstinner]

The Python C API has an efficient API to access tuple/list items:

• seq = PySequence_Fast(obj)
• size = PySequence_Fast_GET_SIZE(seq);
• item = PySequence_Fast_GET_ITEM(seq, index);
• items = PySequence_Fast_ITEMS(seq); -- then you can use items[0], items[1], ...
• Py_DECREF(seq); -- release the "view" on the tuple/list

Problem: If obj is not a tuple or a list, the function is inefficient: it creates a temporary list. It's not possible to implement an "object array view" protocol in *3rd party C extension types.

The other problem is that the &PyTuple_GET_ITEM(tuple, 0) and &PyList_GET_ITEM(tuple, 0) code to get a direct access to an object array doesn't give a clear control on when the array remains valid. The returning pointer becomes a dangling pointer if the tuple/list is removed in the meanwhile.

I propose designing a new more generic API:

PyAPI_FUNC(int) PySequence_AsObjectArray(
    PyObject *,
    PyResource *res,
    PyObject ***parray,
    Py_ssize_t *psize);

The API gives a PyObject** array and it's Py_ssize_t size and rely on a new PyResource API to "release the resource" (view on this sequence).

The PyResource API is proposed separately: see issue #106592.

Example of usage:

void func(PyObject *seq)
{
    PyResource res;
    PyObject **items;
    Py_ssize_t nitem;
    if (PySequence_AsObjectArray(seq, &res, &items, &nitem) < 0) {
        if (PyErr_ExceptionMatches(PyExc_TypeError)) {
            PyErr_SetString(PyExc_TypeError, "items() returned non-iterable");
        }
        goto error;
    }
    if (nitem != 2) {
        PyErr_SetString(PyExc_TypeError,
                        "items() returned item which size is not 2");
        PyResource_Release(&res);
        goto error;
    }

    // items or it may be cleared while accessing __abstractmethod__
    // So we need to keep strong reference for key
    PyObject *key = Py_NewRef(items[0]);
    PyObject *value = Py_NewRef(items[1]);
    PyResource_Release(&res);

    // ... use key and value ...
    
    Py_DECREF(key);
    Py_DECREF(value);
}

This design is more generic: later, we can add a protocol to let a custom type to implement its own "object array view" and implement the "release function" with arbitrary code: it doesn't have to rely on PyObject reference counting. For example, a view can be a memory block allocated on the heap, the release function just would release the memory.

Providing such protocol is out of the scope of this issue. Maybe we can reuse the Py_buffer protocol for that.

Linked PRs

• WIP: gh-106593: PyResource API [DO NOT MERGE] #106596

[vstinner]

Should we prevent using PySequence_AsObjectArray() to modify a tuple items? A tuple must be immutable: modifying a tuple is bad: see issue capi-workgroup/problems#56

Should we copy tuple items if a "write only" (or "read+write") view is requested? It's fine to give a direct access if a "read only" view is requested.

[vstinner]

The other problem is that the &PyTuple_GET_ITEM(tuple, 0) and &PyList_GET_ITEM(tuple, 0) code to get a direct access to an object array doesn't give a clear control on when the array remains valid.

These API were left unchanged by rejected PEP 674 – Disallow using macros as l-values: PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged.

[vstinner]

In 2020, I created issue #85250: [C API] Convert PyTuple_GET_ITEM() macro to a static inline function. See related Cython issue: cython/cython#3701 (still open)

[vstinner]

In June 2020, I created PEP 620: C API for efficient loop iterating on a sequence of PyObject** or other C types discussion about such hypothetical API.

[vstinner]

Proof-of-Concept PR to show how this API can be implemented and used: PR #106596.

[encukou]

A more robust alternative would be API for iterating by chunks, discussed in https://discuss.python.org/t/15993/20 and further.

[steve-s]

I think that ideally the API should consider possible future optimizations, like a compact representation of a list of integers, i.e., we'd be "inflating" the individual values to PyObject* on the fly. Materializing such list to one big PyObject* array is obviously not ideal.

To be more concrete re the "chunks" approach. It can be something along these lines:

PyAPI_FUNC(int) PySequence_AsObjectArray(
    PyObject *,
    Py_ssize_t start_index,
    Py_ssize_t requested_size,
    PyResource *res,    
    PyObject ***parray,
    Py_ssize_t *actual_size);

To iterate the sequence, you'd write two nested loops. The outer calling the API and the inner iterating the items in the buffer.

There can be an inline helper:

PyAPI_FUNC(int) PySequence_AllAsObjectArray(
    PyObject *o,
    PyResource *res,
    PyObject ***parray,
    Py_ssize_t *actual_size) {
    return PySequence_AllAsObjectArray(o, 0, MAX_LONG, ...);
}

However:

• the helper would encourage non-buffered usage, which is not ideal
• the object itself may know better what chunk size is ideal to use (or maybe simply: whether to use buffer or getting the whole contents as one big array is better).

R has similar API for its "alternative" vectors: https://github.com/wch/r-source/blob/688f12cc9627c38ae10c4f597010da3f7142a487/src/include/R_ext/Itermacros.h#L242

They first query the vector for "one big array", which is an optional operation, and if that's not supported, they iterate using fixed size small-ish buffer. All is wrapped in a convenience C macro.

[encukou]

Yeah, this should probably be a PEP. Adding a function for a subset of use cases will only make the C-API bigger, as we add new ones for the remaining cases.

[ronaldoussoren]

I'm pretty sure this has been discussed before, possibly on discuss.python.org or one of the mail lists.

An alternative interface is to provide an interface to iterate in chunks, similar to fast enumeration in Objective-C/Cocoa. That way buffer allocation can be avoided, although the semantics will we different in a way that can affect code that currently uses PySequence_Fast with objects that can be modified during iteration.