MNT co-vendor array-api-{compat, extra}

[lucascolley]

closes gh-30367

• Vendors array-api-compat and array-api-extra. array-api-compat no longer needs to be installed by users. Some behaviour that previously was enabled/disabled based on array-api-compat being installed is now enabled/disabled based on SCIPY_ARRAY_API being set
• uses array_api_compat.numpy instead of _NumPyAPIWrapper
• replaces the private helper here with xpx.setdiff1d

TODO:

• regenerate lock files

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 3356566. Link to the linter CI: here}

[lucascolley]

thanks for the script example @thomasjpfan, that seems to have worked! @betatim could I get a responsible adult to add the GPU CI label please 😁

[lucascolley]

oops, converting to draft

[OmarManzoor]

@lucascolley I have a questions about integrating with array-api-extra. If we are implementing the array API for a particular function or class and in the process there seems to be some function that is not implemented in the array API specification and we need to override it, if we implement it in array-api-extra instead of how we are doing it currently in the utils, won't this be a blocker until the next version of array-api-extra is released?

[lucascolley]

if we implement it in array-api-extra instead of how we are doing it currently in the utils, won't this be a blocker until the next version of array-api-extra is released?

It doesn't have to be instead of utils. I would suggest you keep your current workflow the exact same - you can still put private helpers in utils to start off with. But in addition, every now and then, you can offload them to array-api-extra so that others can use them. That's what we're doing in SciPy.

[OmarManzoor]

It doesn't have to be instead of utils. I would suggest you keep your current workflow the exact same - you can still put private helpers in utils to start off with. But in addition, every now and then, you can offload them to array-api-extra so that others can use them. That's what we're doing in SciPy.

Okay I understand. Thank you.

[lucascolley]

Thanks all, I count +4! Is anyone else wanting to check this over before it acquires a merge conflict?

[OmarManzoor]

LGTM. Thank you @lucascolley

Just a few points:

• Is diff1d the only function that we can utilize from array-api-extra right now as we have a bunch of them in the array api utils?
• We might need to take an additional responsibility to either open an issue on array-api-extra or add the functionality ourselves whenever we add a new function in the scikit-learn array api utils?

[lucascolley]

Is diff1d the only function that we can utilize from array-api-extra right now as we have a bunch of them in the array api utils?

It is worth checking http://data-apis.org/array-api-extra/api-reference.html. We should be able to transfer a lot over from sklearn, isin should be pretty simple since we have in1d private already.

We might need to take an additional responsibility to either open an issue on array-api-extra or add the functionality ourselves whenever we add a new function in the scikit-learn array api utils?

That would be appreciated!

[OmarManzoor]

@lucascolley Here is a list of functions I observed that could be moved. However I think if we want to integrate them it should be done in a follow up PR after this is merged. Let me know what you think about these:

• _expit
• _fill_or_add_to_diagonal
• _nanmin
• _nanmax
• _nanmean
• _ravel
• _isin

[lucascolley]

Yep, feel free to open issues (or PRs) over at array-api-extra and we can discuss!

[ogrisel]

Maybe @thomasjpfan has an opinion as the original author of many of the array API helper tooling of scikit-learn.

[betatim]

@lucascolley can you re-run the lock file generation script? The weekly update happened today so we now have conflicts :-/

Should we wait a few more days and then merge? Technically we have enough 👍 already, but maybe worth waiting a moment longer? But not too long so we don't make Lucas sort out conflicts repeatedly

[lucascolley]

🫡

[thomasjpfan]

We did not initially vendor array-api-compat because they were releasing bug fixes quickly and scikit-learn has a longer release cycle.

I am +1 on vendoring both array-api-{compat, extra}.

[lucascolley]

thanks all!