[RFC] Memory allocator interoperability with PyTorch

[kmaehashi]

Description

It is one of the common use cases for CuPy to be used in conjunction with PyTorch. One of the difficulties when using two CUDA-powered libraries (including but not limited to CuPy/PyTorch combination) is sharing memory pools and streams between them. Currently, utility functions to share CuPy/PyTorch memory pools/streams are provided in pytorch-pfn-extras (pytorch_pfn_extras.cuda.*), however, it makes sense to host them in CuPy if there is a demand.

Specific features in mind are:

• Use PyTorch memory allocator in CuPy
• Use CuPy memory allocator in PyTorch
  • This was made possible starting in PyTorch 2.0, as we (@emcastillo) contributed a pluggable memory allocator feature to PyTorch: Add Pluggable CUDA allocator backend pytorch/pytorch#86786
• Share the same CUDA stream between CuPy and PyTorch

This RFC issue intends to gather interest for this feature from the community, and discuss where to put these methods (cupyx.???) once we decide to do that.

Additional Information

No response