Vulkan interoperability

[cajoek]

🚀 The feature, motivation and pitch

Vulkan-PyTorch interoperability including zero-copy data transfer on GPU (Cuda <-> Vulkan).

I am working on a real-time machine learning graphics project which uses Vulkan (or OpenGL) both as an intermediate processing step in the model and to visualize the output. But the problem is transferring data between PyTorch/Cuda and Vulkan. I can copy data from PyTorch/cuda to CPU and then back to Vulkan on GPU but this is prohibitively slow.

This feature would simplify a lot of interesting research into machine learning graphics.

Alternatives

No response

Additional context

• Vulkan can be used from Python with e.g.: https://github.com/realitix/vulkan
• As Vulkan is a backend to ExecuTorch maybe this feature wouldn't be to far from what is currently being worked on

[malfet]

Why aren't dlpack sufficient to you?

[cajoek]

Does Vulkan have support for dlpack or would I have to implement that myself?

[malfet]

Does Vulkan have support for dlpack or would I have to implement that myself?

It does, see https://dmlc.github.io/dlpack/latest/

[cajoek]

I looked at it but I am a bit lost to how this is done. Do you have any examples of how to use dlpack to transfer data from/to Vulkan?

[cajoek]

@malfet That single mention in DLPack’s documentation is the only proper mention of DLPack together with Vulkan that i can find on the internet. May I ask for some help here?