Add initial cupyx.spatial.distance support from pylibraft

[cjnolet]

This is an initial PR to establish integration of pylibraft within cupy. For now, the intention is to use RAFT for supported distances and types and cupy kernels for other distances and types until we support them in RAFT. This provides us a path to make use of what's available and then continue optimizing RAFT underneath, allowing cupy to reap the benefits immediately.

This is really just a start in the direction of backing the entire cupyx.scipy.spatial package with RAFT, as well as the cupyx.scipy.cluster and cupyx.scipy.optimize packages (and potentially cupyx.scipy.stats / cupyx.scipy.qmc in the future).

Tasks:

• Add cdist
• Add scipy.spatial.distance_matrix, minkowski_distance
• Add tests

[emcastillo]

I think the current functionality is enough for this PR.
Can we split the other APIs in subsequent ones?
Thanks!!!

[cjnolet]

@emcastillo sounds great. I'll do that

[cjnolet]

This PR depends upon the changes in rapidsai/raft#643 (which should be merged shortly).

[cjnolet]

@emcastillo, one thing I've noticed is that I don't believe pylibcugraph is being tested in CuPy's CI. At least I don't think it is, because I don't see it listed in any dependencies in the build so I'm assuming those tests are getting skipped.

It would be nice if we could run the tests for pylibraft and I think we might be safe installing it since I don't believe any of the RAFT python artifacts have any dependency on cupy. I suppose we could either just install it as a test or runtime dependency but I think that will depend on whether the sizes of dependent artifacts are a concern. What are your thoughts here?

[emcastillo]

We will try to add pylibcugraph to the CI!

[emcastillo]

seems like it is installed in CI rapids configuration?

https://ci.preferred.jp/cupy.linux.cuda-rapids/101331/
This should test with libcugraph enabled

Need to add
tests/cupyx_tests/scipy_tests/spatial_tests
to https://github.com/cupy/cupy/blob/master/.pfnci/linux/tests/cuda-rapids.sh#L15

[kmaehashi]

We intend to test CuPy code depending on RAPIDS in cuda-rapids test variant.

• Test environment: https://github.com/cupy/cupy/blob/master/.pfnci/linux/tests/cuda-rapids.Dockerfile
• Test script: https://github.com/cupy/cupy/blob/master/.pfnci/linux/tests/cuda-rapids.sh
• Output for the recent master branch: https://ci.preferred.jp/cupy.linux.cuda-rapids/101331/#L7175

These are separated from regular unit tests (which are based on pip) as RAPIDS installation requires conda.

CI can be triggered in pull-requests using /test rapids (run the RAPIDS test variant only) or /test mini,rapids (run regular unit tests + RAPIDS test variant).

[cjnolet]

/test mini, rapids

[cjnolet]

/test mini

[cjnolet]

/test mini

[cjnolet]

/test rapids

[cjnolet]

/test mini,rapids

[cjnolet]

/test mini, rapids

[emcastillo]

/test mini, rapids

[cjnolet]

@emcastillo we're working on the fix for the RAPIDS core-devel container build and then this should be ready.

[jakirkham]

Depends on PR ( rapidsai/raft#714 )

[cjnolet]

/test rapids

[cjnolet]

/test rapids

[cjnolet]

@emcastillo @kmaehashi looks like the RAPIDS tests are finally running (and passing!).

[emcastillo]

LGTM!

[jakirkham]

Thanks all! 🙏 Looking forward to using these 😄

[cjnolet]

Yes, thanks again @emcastillo @kmaehashi and @leofang. As requested by @emcastillo, We'll continue opening smaller PRs to continue to fill in the API support now that we've got the integration path working end-to-end.