Enable Inference Directly By Image Already In GPU memory.

[joshuafc]

In some case, we have image directly in gpu memory, such as call NVDEC to decode H264 stream, or call nvJPEG for JPEG decode.

But now, the C++ inference only can call Apply with an Mat object which cannot specify where the image data is in( CPU, or GPU 0, or GPU1 ?)

In mmdeploy/csrc/mmdeploy/apis/c/mmdeploy/common.cpp:mmdeploy_common_create_input, I found an mmdeploy::Mat is created with device parameter hardcode to 'cpu'.

Is there any plan to export the device parameter to high level API such as Detector::Apply, so when inference, we can specify which memory space hold the image?

[RunningLeon]

cc @lzhangzz

[lzhangzz]

@joshuafc we have plans to support passing external device memory to the API, but it's likely to be ready in v0.9 (the next next release)

[joshuafc]

@lzhangzz thanks for your reply

in v0.9, mmdeploy_common_create_input still create mmdeploy::Mat from mmdeploy_mat with device hardcode to "cpu", does this feature still in plan?

https://github.com/open-mmlab/mmdeploy/blob/v0.9.0/csrc/mmdeploy/apis/c/mmdeploy/common.cpp#L53

[lzhangzz]

Sorry for the late reply, is #1176 what you need?

[joshuafc]

Yes,mmdeploy_device_t device; is what I need.

Additional questions, is there some document about memory layout of mmdeploy_mat_t::data? for example, whether it is need to padding at end of every row, to make next row start at a 4-byte alignment address?(This will be more friendly for writing code that use the SSE instruction set. I remember that Opencv will do this, as a result, some image's cols * channel != step[0].)?

Maybe we need some like cv::Mat::step to let user specify the memory layout of device memory. Or declare that it is the user's responsibility to reform the device memory into the mmdeploy documented memory layout.

[lzhangzz]

Yes, something like step or stride would be very useful. However it will impact every usage of mmdeploy::framework::Mat, as extra care must be taken to handle step correctly.

We are likely to add support for step together with hardware codecs support. But I don't know when it will be ready.

For #1176, you need to re-format the data into contiguous layout. It would waste little GPU time, and will be much faster than traversing the host memory.

[joshuafc]

Thank you for your explanation, I know how to use the new features #1176 now.

And waiting expectantly for hardware codecs support. I work in the field of real-time video structure, and have used NVIDIA GPU and Ascend NPU. I believe that with the hardware codecs support, this great project can also achieve great success in this field.

Thank you and your partners for the contributions to the open source community.

[sisi-au]

@lzhangzz @RunningLeon I know this post is 3 years old. I wonder if you still have a plan to add support passing external GPU device memory to the API.