Update CUDA Array Interface to v3 - Part 1

[leofang]

This PR is blocked by #4322 because we need to know how to handle per-thread default streams (PTDS).

UPDATE: This PR excludes PTDS from consideration as it's currently not supported in the codebase, see #4357 (comment). In the coming "Part 2" PR it'll be properly addressed.

I am propagating the upstream change in the CAI protocol numba/numba#5162 to CuPy. The most notable change in the update to v3 is the requirement of stream synchronization. CAI v3 specifies that

• Producer should include the stream pointer (whenever applicable) on which the data pointed by CAI can be safely operated
• Consumer by default must synchronize the stream, if given, at the interoperating point
• Either Producer or Consumer can provide options to Users to overwrite this behavior (the protocol does not specify how this option should be implemented)

For the detailed definitions (Producer, Consumer, User, etc), see the CAI v3 documentation; note the capitalized nouns follow the CAI's definitions.

Accompanying numba/numba#5162 Numba introduces a new environment variable NUMBA_CUDA_ARRAY_INTERFACE_SYNC to avoid syncing when acting as a Consumer. Following this path, I introduce two environment variables for advanced users to overwrite the sync behaviors, both of which are defaulted to 1 to make it compliant with CAI v3. However, due to the mismatched concepts on "default streams" between Numba and CuPy, the effect is slightly different.

In a nutshell, they allows us to fully restore the old (status quo) behavior, as if the v3 update does not exist. Specifically,

• As a Producer, CuPy would not export any stream if CUPY_CUDA_ARRAY_INTERFACE_EXPORT_STREAM is set to 0
• As a Consumer, CuPy would not synchronize over any external streams provided through CAI if CUPY_CUDA_ARRAY_INTERFACE_SYNC is set to 0

This should make it as performant and give Users the full control as before (if so desired), while taking care the need of certain libraries (notably, mpi4py) in which none of CUDA API is accessible and thus the required synchronization cannot be performed.

The v3 protocol also made it clear about Users' responsibility of maintaining the lifetime of GPU arrays and streams for the purpose of utilizing CAI, so we can safely assume any given external stream is valid.

UPDATE 2: To make it easier to review, note the 3 test files touched in this PR examine different (though arguably overlapping) aspects of the CAI according to my interpretation:

• tests/cupy_tests/core_tests/test_ndarray.py: Check CuPy's behavior as a Producer
• tests/cupy_tests/creation_tests/test_from_data.py: Check CuPy's behavior as a Consumer
• tests/cupy_tests/core_tests/test_ndarray_cuda_array_interface.py: Ensure various operations are correctly done when CAI is in play

cc: @jakirkham @pentschev

[kmaehashi]

We're thinking of backporting this to v8 to allow consuming CAI v3 safely in CuPy v8, but excluding PTDS support.

[pentschev]

We're thinking of backporting this to v8 to allow consuming CAI v3 safely in CuPy v8, but excluding PTDS support.

Sounds reasonable to me, @leofang I'm fine if you want to push this forward before we merge PTDS.

[leofang]

Sounds good. So perhaps this PR should split into two: the first one handles CAI v3 without PTDS (as @kmaehashi suggested) and will be backported (hopefully it's straightforward), and the second handles PTDS once #4322 is settled and will not be backported.

btw, note that the earliest Numba that supports CAI v3 is expected to be released early next year: numba/numba#5162 (review).

[leofang]

@kmaehashi @asi1024 @pentschev @jakirkham @kkraus14 @gmarkall I think this is ready. The PR description is updated. PTAL.

[leofang]

Jenkins, test this please

[chainer-ci]

Jenkins CI test (for commit 5f12098, target branch master) succeeded!

[leofang]

Jenkins, test this please

[chainer-ci]

Jenkins CI test (for commit fc797e9, target branch master) succeeded!

[asi1024]

Jenkins, test this please.

[chainer-ci]

Jenkins CI test (for commit 76bfcc3, target branch master) succeeded!

[asi1024]

LGTM! Thanks!