Unexpected behavior for `**=` when `__numpy_ufunc__` is present.

[lkilcher]

I've been working on getting Pint ready for the new __numpy_ufunc__ functionality, but I've run into a snag.

I've done my best to document the issue in this gist. In particular, the issue is that when creating classes that are not ndarray, I can reproduce expected behavior using standard python special operators (e.g. __add__) when __numpy_ufunc__ is present. For example, myobj.__radd__ is called when I do an_ndarray += myobj. However, myobj.__rpow__ is not called when I do an_ndarray **= myobj. Can anyone explain to me why this is? Thanks.

This issue has arisen from this one: hgrecco/pint#326.

[njsmith]

The __numpy_ufunc__ in master is somewhat broken wrt its interaction with regular special operators, and the way it works in master is not how it will work when it is released. How it will work when it is released is somewhat up in the air :-). There was a giant thread about this about 6 months ago (#5844) that didn't come to any conclusion, and the functionality is on hold until we have time to get back to it and get consensus. (Also, it won't be called __numpy_ufunc__ -- see #5986.) So there's currently no schedule for it actually being included in any release.

FWIW in the particular case of in-place operators, I feel strongly that ndarray op= something should never delegate to something.__rop__, for any value of op, and that if __numpy_ufunc__ (or whatever it ends up being called) is not present then your examples should just raise an error. The reason is that when working with arrays, the distinction between in-place and out-of-place operations is very important -- it affects memory usage, view semantics, all kinds of things. So in regular python with immutable scalars, it's fine that x += obj expands to x = x + obj, but for ndarrays, we should enforce that in-place operators really are in-place (ndarray += obj expands to ndarray[...] = ndarray + obj), and as soon as we allow fallback to the __radd__-style methods then we lose this property. (See also #6020.)

(NB that it will of course be possible to override these via the __numpy_ufunc__-style interface, because ndarray += obj will expand to np.add(ndarray, obj, out=ndarray), and then that expands to something like obj.__numpy_ufunc__(np.add, (ndarray, obj), out=ndarray). But this still doesn't give you a way to convert an in-place operation into an out-of-place operation; it just gives you a way to let obj control how the in-place operation is implemented.)

[lkilcher]

Thanks for this. I definitely agree with you that in-place operators should not fall back on out-of-place. I'll be interested to see how this develops. Given the instability of this I'll abandon my efforts to use it for now.