DOC: update NEP 31 for vendoring and being complementary to NEP-13/18

[hameerabbasi]

cc @rgommers, a nicer interface to discuss the NEP before it's submitted. 😄

[hameerabbasi]

I don't know why the first two switched to having a base of master, I explicitly selected uarray.

[hameerabbasi]

cc @peterbell10 would you like to be co-author? Also, any suggestions here?

[hameerabbasi]

Suggested change

	mechanism similar to NEP-18 `[2]`_. First experiences with ``__array_function`` show that it is
	mechanism similar to NEP-18 `[2]`_. First experiences with ``__array_function__`` show that it is

[rgommers]

yep typo

[hameerabbasi]

How about a soft dependency, like in scipy.fft?

[rgommers]

That will probably be fine, just thinking about if we should put that in here. The key thing is vendoring. Whether to optionally override the vendored version with an installed version to be able to ship bug fixes quicker is a small detail.

Note also that it's not a normal soft dependency - that typically means that not installed == no functionality.

I'd actually leave it out, since it's not important at this point.

[peterbell10]

Note also that it's not a normal soft dependency - that typically means that not installed == no functionality.

Is there an accepted term for this? I was just looking for a word between hard and optional.

Either way, the "optionally overriding" behaviour is very important in order for the vendored version to play nicely with other libraries. For example, if I have a decorator @implements(unumpy.sum) such as in NEP 18, then my backend won't work at all if my unumpy.sum is not the same as the user's unumpy.sum. This is exactly the problem that we get through normal vendoring.

[rgommers]

Is there an accepted term for this?

Not that I know of - I think it requires a one sentence description.

then my backend won't work at all if my unumpy.sum is not the same as the user's unumpy.sum

that's a good point. isn't an issue for scipy.fft (right?) but is tricky here. I think the soft dependency solves the issue, as long as other libraries don't do the same. If they either import from the numpy vendored package, or from an installed unumpy directly that should both work.

[peterbell10]

isn't an issue for scipy.fft (right?)

Yes, this is only an issue for numpy because unumpy is the package defining the multimethods. In scipy.fft we own the multimethods ourself so these cannot be mismatched, even if the uarray libraries were.

I think the soft dependency solves the issue

Completely agree, I just wanted to point out that shipping bug fixes faster is not the only motivation.

[peterbell10]

I'm curious what the motivation is for numpy to vendor unumpy? I personally don't think it's explained here.

I also think that it would be good to mention overrides for numpy arrays such as pyFFTW and another one I've come across was opt_einsum. It's definitely a good example of something that __array_function__ just cannot support.

[peterbell10]

Suggested change

	The motivation behind ``uarray`` is manifold: First, there have been several attempts to allow
	The motivation behind ``uarray`` is manyfold: First, there have been several attempts to allow

[rgommers]

IIRC PyFFTW just monkeypatches NumPy, so yes that's good to mention. And opt_einsum too, didn't realize it had its own backend system with the build_expression stuff. @saulshanabrook that may actually be of interest to you, the transformations seem nontrivial enough that metadsl could be a nice fit.

I'm curious what the motivation is for numpy to vendor unumpy? I personally don't think it's explained here.

We really dislike dependencies. NumPy doesn't have any at the moment. Proposing a hard dependency on a brand new and fast moving package is a showstopper in itself, and will definitely derail the discussion on the merits of the dispatching mechanism.

[peterbell10]

As far as I can tell it should be viable for unumpy to be a completely optional dependency for numpy. I have done the same thing (albeit on a smaller scale) for my cupyx FFT backend in cupy/cupy#2355.

If scipy.fft is not available, I use a mock version instead:

try:
    import scipy.fft as _scipy_fft
except ImportError:
    class _DummyModule:
        def __getattr__(self, name):
            return None

    _scipy_fft = _DummyModule()

Then I create the multimethod overrides using _scipy_fft. e.g.

@_implements(_scipy_fft.fft)
def fft(x, n=None, axis=-1, norm=None, overwrite_x=False):

[rgommers]

Yeah, but then the feature isn't available:) That then propagates to Dask, CuPy, etc. - that doesn't really work well. We need the always-available without a hard dependency. So it's either move into NumPy wholesale, or the vendor + "soft dependency" option.

[hameerabbasi]

I believe "manifold" is what I meant: https://www.merriam-webster.com/dictionary/manifold

[rgommers]

Okay, I pushed some TODOs. The whole thing isn't consistent yet. We need the usage examples so it can be compared to NEP 30 - just reference the unumpy docs won't cut it.

I'm going back and forth on the vendoring. The problem is that if it's opt-in, the imports become messed up - you cannot do import ... as np anymore, because both numpy and unumpy want to be called np.

The key problem is: wholesale adoption is going to be a very hard sell, because it effectively gets rid of __array_function__ and __array_ufunc__. A partial adoption for things that those protocols don't support makes sense from a social acceptance point of view, but seems to run into the np problem I pointed out above.

[hameerabbasi]

A partial adoption for things that those protocols don't support makes sense from a social acceptance point of view, but seems to run into the np problem I pointed out above.

If we set NumPy as the backend (via set_global_backend) for unumpy, those overrides will be preserved.

[hameerabbasi]

Merging since I don't have commit rights to this PR... And I wanted to push some changes. Thanks @rgommers and @peterbell10!

[hameerabbasi]

@rgommers Can you expand on this a little bit?

obviates the need to add an ever-growing list of new protocols for each new type of function or object that needs to become overridable.

It assumes that there are parts of the API that need to be overridable, and that these will grow over time. It provides a general framework and a mechanism to overridable, and that these will grow over time.

So it's consistent, even though it explains other use-cases. Would you like for this section to be removed? Personally, I'd at least like to mention the possibility of having such a system, and for unumpy as an alternative to other overrides.

[rgommers]

Personally, I'd at least like to mention the possibility of having such a system, and for unumpy as an alternative to other overrides.

Yes, I think it's good to mention that.

Can you expand on this a little bit?

If the whole thing is opt-in and kept in a separate namespace, as in the abstract and motivation sections, then a lot of this reads strangely. The examples don't work as np.<funcname>, because that assumes you've already completely taken over the whole namespace. It's clearer when you add the full example, so I'll comment on your new PR.