ENH: Provide a way to overwrite the standard PEP 3118 way on how objects are converted to numpy arrays

[hannorein]

Proposed new feature or change:

Numpy supports various ways to convert an instance of a class to a numpy array. For example, using the __array__ syntax:

import numpy as np
class Vec3d:
    x = 1
    y = 2
    z = 3
    def __array__(self, dtype):
        return np.array([self.x, self.y, self.z])

v = Vec3d()
n = np.zeros((2,3))
n[0] = v  # v can be used in assignments like this

However, when the class supports PEP 3118, then this overwrites all other ways. For example, the following Vec3dS class is a subclass of ctypes' Structure (which implements PEP 3118):

from ctypes import Structure, c_double
class Vec3dS(Structure):
    _fields_ = [("x", c_double), ("y", c_double), ("z", c_double)]
    def __array__(self, dtype):
        return np.array([self.x, self.y, self.z])

When converting an instance of Vec3dS to a numpy array, the resulting array has dtype([('x', '<f8'), ('y', '<f8'), ('z', '<f8')]). The __array__ function (as well as __array_interface__) is ignored. This prevents any subclass of ctypes.Structure to be used in assignments like this:

vs = Vec3dS(1,2,3)
n[1] = vs # this will fail

Of course, there are several other ways to create a numpy array from a ctypes Structure. But unless I'm missing something, none of them allow a user to use the simple n[1] = vs syntax in the above example.

It would be nice to have a way to explicitly tell numpy how to convert an instance of a class to a numpy array rather than always prioritizing PEP 3118. Correct me if I'm wrong, but I don't think I can overwrite the PEP 3118 interface from the python side.

[seberg]

xref gh-17134 which this is a duplicate of. As I said there, my gut feeling is that we could reorder it if we want to, although the buffer protocol is by far the fastest to test for unfortunately.

[eric-wieser]

This feels like something that should be handled in Python itself; such as being able to set __buffer__ = None to entirely discard the buffer interface on a subtype. I think PyPy already behaves in this way.

[eric-wieser]

See also my SO answer

[seberg]

I quite sure I saw Python discussing implementing __buffer__ recently. But I don't want to search for it right now. That would indeed at least solve the issue by allowing to override the superclass (and some other niceties).

[mattip]

Here is the discussion https://discuss.python.org/t/pep-688-making-the-buffer-protocol-accessible-in-python/15265

which refers to PEP 688 https://peps.python.org/pep-0688/