Feedback on the video decoding extension(s)

[cyanreg]

Hi,

I'm writing hardware decoding code for FFmpeg (code), and in the process I've found some issues with the extensions.
The first one is simple, VkVideoDecodeH264ProfileEXT.pictureLayout is required to not be 0, but VkVideoDecodeH264PictureLayoutFlagBitsEXT has the value for VK_VIDEO_DECODE_H264_PICTURE_LAYOUT_PROGRESSIVE_EXT set as 0 (spec).

In the blog post you asked for feedback on sub-picture level decoding:

Currently only picture-level decode commands are supported (as specified by the appropriate codec-specific EXT extension structures for decode operations, for example VkVideoDecodeH264PictureInfoEXT). We are interested to hear of use cases that need to request more fine grained operations!

The main use-case is for reduced latency. After all, at 60fps, having to wait for 16 whole milliseconds while you receive a frame from a realtime stream, and only then submit a queue to decode it, and then spend 16 more milliseconds while you schedule it for presentation will double the latency. At the cost of screen tearing, you can partially redraw parts of the screen, thus if you do a row of slices at a time, you can cut down on a lot of latency.
FFmpeg supports sub-frame decoding, and can output a row at a time. Would be nice if we could support that using Vulkan hardware decoding.

But moreover, the biggest issue with the decoding API is how you're supposed to have all slices in a single VkBuffer. Sure, you can solve that by using sparse buffers, but a lot of devices do not support such, so your only choice is to copy each slice into a large enough RAM memory allocation, and then upload that to the GPU. Some bitstreams reach bitrates of hundreds, if not thousands of megabits per second, which can take a significant amount of resources. It would be much better if you could decode a slice at a time as you're receiving the slices, which would hugely reduce allocations and copies needed.
Or even if still decoding an entire picture at a time, you could let the GPU work on decoding quicker, rather than wasting a whole 16 milliseconds or so while waiting to receive each slice the frame has.

So, my recommendation would be to introduce a new vkCmdDecodeSlice command, which takes a new VkVideoDecodeSliceInfo structure, which allows for users to feed in a slice inside a single VkBuffer at a time. Additionally, a vkCmdStartFrame and vkCmdEndFrame commands to signal when a decoding starts and ends. All of this must still take place in a single submission, of course, so calling vkCmdStartFrame without calling vkCmdEndFrame would be an error.
If slice-level decoding is found to be within scope, the user can signal VkEvents inside the submission, which would inform the user whether a slice has been decoded and the contents in the VkImage corresponding to the slice are valid.
Additionally, the VkVideoDecodeInfoKHR structure ought to be modified to permit having each slice in a separate VkBuffer object.

[zlatinski]

Hi @cyanreg, first of all - thank you very much for working on adding Vulkan Video support to FFMPEG! We really appreciate your contribution. We highly value your input on the Vulkan Video speechification and the API.

Let me start with the first question about the latency on decoding complete frames vs. slice mode.

After all, at 60fps, having to wait for 16 whole milliseconds while you receive a frame from a real-time stream, and only then submit a queue to decode it, and then spend 16 more milliseconds while you schedule it for presentation will double the latency.

Please help me understand this. Is this a limitation with real-time streaming that it only delivers the data at a specific rate and can not deliver more data?
The way the decode should work on Vulkan is by working with the parser/decoder/render/display ahead of time:

• Parsing bitstream and schedule HW decode of multiple frames ahead of time. Using multithreaded parse and/or buffer recording should be happening at > 300 FPS even for 4k content.

• There should be multiple decode frames queued in the Vulkan decode queue and those should each take much, much less than 16 ms time (for 4k content) to decode on contemporary decode HW. Hopefully, the application is not using fences where is waiting on each frame at the CPU side for the decoded frames to be processed. As soon as the frames are submitted to the decoded queue, those should be submitted right away with a Vulkan semaphore to the graphics and the display. This is the best way to reduce latency - having a lot of work queued at the HW decoder/GPU and display ahead of time.

• The (Vulkan WSI) display should have at least 2-4 frames queued with present timing information, so as soon as the video present time has been reached, those frames should be flipped to the display (by the display HW).

For example, the current Vulkan Video sample app that we are providing is only limited by the display rate (we do not support the display timing yet) and when using a display with a refresh rate of 240 FPS, we do not have trouble maintaining the 240 FPS with a 4k@60 h.264/5 video content. If we were to use one of the display timing extensions (we are planning to add it soon) then the latency should be almost "0" -. i.e. would only depend on the display's flip latency.

[zlatinski]

On more note about slice mode:
Decoding a VCL slice would not allow for it to be presented, right away, with most of the content. Furthermore, not all frames are presentable and the display order is not necessarily the same as the decode order. I know there are some of the VR-targeting low-latency encode setups that allow for that, but if one were to follow the above setup, latency shouldn't be an issue.

But moreover, the biggest issue with the decoding API is how you're supposed to have all slices in a single VkBuffer. Sure, you can solve that by using sparse buffers, but a lot of devices do not support such, so your only choice is to copy each slice into a large enough RAM memory allocation, and then upload that to the GPU.

For this second point, why do we need the bitstream to be put in RAM? Because we do not know the size of the bitstream ahead of time? If sparse memory is not supported, the application can allocate multiple chunks of VkBuffers as a circular buffer with a size that can accommodate bitstream data worth of one frame each. I would guess the limitation you are seeing is related to the HW buffers alignment requirement., so when data gets received from a video stream, it can be pre-parsed and placed as a cache in VkBuffers from get go without doing extra copies. One can also work with the data offset in the bitstream, that does not necessarily has to be HW buffer aligned (srcBufferOffset).

[zlatinski]

All that being said, the Khronos Video TSG has plans to add support for slice mode, but after the first Vulkan Video release. In the meantime, we still want to ensure that there aren't any major Vulkan Video API limitations on supporting low-latency video streaming applications in the absence of this feature.

[cyanreg]

Please help me understand this. Is this a limitation with real-time streaming that it only delivers the data at a specific rate and can not deliver more data?
The way the decode should work on Vulkan is by working with the parser/decoder/render/display ahead of time:

All you described here is caching. Caching is how you ruin latency, not fix it.
While the actual decoding process takes only a few milliseconds to finish, this skyrockets when you have more streams that you have to decode at the same time. NVIDIA hardware has an artificial limit of IIRC 4 streams on consumer hardware, but other vendors do not, and instead, schedule their tile decoder units around all streams' tiles that have to be decoded.
What happens with picture-level decoding and multiple streams is that a single high profile high bitrate stream can bottleneck the entire decoder, preventing the other streams from being decoded in realtime.
This is the whole point of Vulkan - offloading as much work off the driver as possible, only letting it do scheduling such that all API users get a fair slice, and have limited opportunity to interfere with other work scheduled by other API users.

All that being said, the Khronos Video TSG has plans to add support for slice mode, but after the first Vulkan Video release. In the meantime, we still want to ensure that there aren't any major Vulkan Video API limitations on supporting low-latency video streaming applications in the absence of this feature.

What about doing some preprocessing as slices are being received? For example, into a new opaque VkVideoDecodedSlice object, created from a vkCmdDecodeSlice command, which vkCmdDecodeVideoKHR could accept a list of in order to decode?
That way the GPU is still able to do something in the dead time of receiving slices.

[aabdelkh]

Thanks again for this very valuable feedback @cyanreg !

As @zlatinski mentioned, Video TSG is definitely considering low latency slice decoding. Please see description for VkVideoEncodeH264InputModeFlagBitsEXT for some ideas in progress for encode (although that still needs updates to be consistent with the rest of the spec and finalize).

We'll definitely consider how to make use of your feedback. Our main concern is not to delay the work for the first release. Once the main CTS etc. infrastructure and reference implementations are in place, making extension updates to add more features would be a lot easier.

[cyanreg]

Keep in mind the current decode API cannot be extended. You must have all data in a single VkBuffer. So you will need a new command-level API. I think it would be better if you spent time now to make sure it's easily extendable, rather than make hacks later (like specifying each slice's data inside a separate pNext structure) or spend longer debating how a new API would look like, and how it would interfere with the picture-level API.

Plus, even for picture-level decoding, requiring to have all your data inside a single VkBuffer makes the API high CPU overhead. Could you at least consider changing the API to allow for one-slice-per-VkBuffer? Managing a pool of sparse binding/residency buffers without knowing upfront how large your packet is quite heavy. Not everyone wants or thinks it's acceptable to copy megabytes on the CPU per-frame.

[aabdelkh]

Our initial thinking was to later add a codec-specific extension structure to enable per-slice cmds (e.g. VkVideoDecodeH264SliceInfoEXT chained to VkVideoDecodeInfoKHR), or to simply expose a way to allow VkVideoDecodeH264PictureInfoEXT.sliceCount to be 1 for a multi-slice picture if per-slice cmds are supported.

Yes we'll think about this more, and see what should be updated for the first release (even if it's only picture-level) vs. later based on all your valuable feedback. Please keep it coming! :)

[zlatinski]

NVIDIA hardware has an artificial limit of IIRC 4 streams on consumer hardware, but other vendors do not, and instead, schedule their tile decoder units around all streams' tiles that have to be decoded.

@cyanreg, NVIDIA's Vulkan Video driver should not have a limitation on the number of decoder instances (Video Queues and Video Sessions) created. If this is what you are experiencing, it is most likely an unintentional driver bug. Can you please confirm the version of the NVIDIA driver you are using, where you see the limitation with the maximum of 4 decoder streams, and also double-check with the latest NVIDIA BETA driver?

[zlatinski]

What happens with picture-level decoding and multiple streams is that a single high profile high bitrate stream can bottleneck the entire decoder, preventing the other streams from being decoded in realtime.
This is the whole point of Vulkan - offloading as much work off the driver as possible, only letting it do scheduling such that all API users get a fair slice, and have limited opportunity to interfere with other work scheduled by other API users.

Please try this with multiple decoder HW instances (video decode queue family instances), not multiple Video Sessions against the same queue family. I'd be surprised if you can still see what you are describing above even with high-profile 4k content.

Keep in mind the current decode API cannot be extended. You must have all data in a single VkBuffer.

For the slice decode Vulkan API, it should look very similar to the vkCmdDecodeVideoKHR(), but will allow for processing of one or more slices, instead of the entire frame. In this case, the VkBuffer will only need to have one or more slices worth of data.
This would imply a smaller buffer size, right?

Could you at least consider changing the API to allow for one-slice-per-VkBuffer?

Not all HW codecs can support that.

Plus, even for picture-level decoding, requiring to have all your data inside a single VkBuffer makes the API high CPU overhead.

Regardless of if the data is being copied slice-by-slice or for the entire frame, the entire data must be copied. The only difference is - for the per slice copy one is distributing the copy pieces in time and for the frame, it is done as a single burst.

Managing a pool of sparse binding/residency buffers without knowing upfront how large your packet is quite heavy.

I do not see how managing sparse buffers is complicated? Where is the complexity coming from?

Even when using a regular VkBuffer, please consider the following. Do not think of those buffers as individual pieces but rather as a big circular buffer. Instead of copying the stream data to a CPU-allocated buffer, try copying the data directly over this circular buffer. Then when submitting the buffer to the decoder, the offset should reflect the beginning of the VCL frame data in this circular buffer. The circular buffer should be reset to the beginning after the data left to the end is not sufficient to accommodate the worst-case scenario size of bitstream data.

[cyanreg]

Keep in mind the current decode API cannot be extended. You must have all data in a single VkBuffer.

For the slice decode Vulkan API, it should look very similar to the vkCmdDecodeVideoKHR(), but will allow for processing of one or more slices, instead of the entire frame. In this case, the VkBuffer will only need to have one or more slices worth of data. This would imply a smaller buffer size, right?

If the new API supports multiple slices in a single VkBuffer, then why would the current API need to exist and still be maintained? I think a new API shouldn't outright replace the old one, which still isn't finalized, but rather to complement it.

Could you at least consider changing the API to allow for one-slice-per-VkBuffer?

Not all HW codecs can support that.

The driver could internally represent multiple VkBuffers as a sparse buffer. The spec doesn't say anything about sparse buffers not being supported for decoding. And if the device doesn't support that, then accordingly, the spec should be updated to explicitly mention sparse buffers are not supported for video decoding.

Plus, even for picture-level decoding, requiring to have all your data inside a single VkBuffer makes the API high CPU overhead.

Regardless of if the data is being copied slice-by-slice or for the entire frame, the entire data must be copied. The only difference is - for the per slice copy one is distributing the copy pieces in time and for the frame, it is done as a single burst.

However, if you had multiple VkBuffers, you could upload slices as you process them, rather than let the GPU do nothing until it's got data. And better yet, you could host map them, regardless of whether they appear sequentially in memory (since the slice data offset can not be negative, IIUC), which would also remove the alignment limitation, as you could simply adjust your data pointer, and then use the offset field to zero out where the slice data starts.

Managing a pool of sparse binding/residency buffers without knowing upfront how large your packet is quite heavy.

I do not see how managing sparse buffers is complicated? Where is the complexity coming from?

The complexity comes from the fact that you do not know the size of each slice at the start of decoding. Hence, the VkBuffer would need to be sparsely bound and sparsely resident (which also limits device support even more). You would also be submitting one vkcmd per slice in order to bind the vkdevicememory to the buffer, which isn't a low-overhead operation, and would require you to wait on the previous bind operation to finish before you can start a new one.
If you could support a slice per vkbuffer, then you could approach this in multiple ways, either via host mapping, or via explicit uploading to a device_local-only buffer per slice, or copying them via the CPU and invalidating them before you begin decoding.

Even when using a regular VkBuffer, please consider the following. Do not think of those buffers as individual pieces but rather as a big circular buffer. Instead of copying the stream data to a CPU-allocated buffer, try copying the data directly over this circular buffer. Then when submitting the buffer to the decoder, the offset should reflect the beginning of the VCL frame data in this circular buffer. The circular buffer should be reset to the beginning after the data left to the end is not sufficient to accommodate the worst-case scenario size of bitstream data.

The issue with this approach is that packet sizes can wildly change during a stream, and since you cannot resize VkBuffers, you'll often need to tear it down and allocate a new one, or waste memory.

[airlied]

@ceyusa maybe gstreamer needs to be involved also.

[zlatinski]

If the new API supports multiple slices in a single VkBuffer, then why would the current API need to exist and still be maintained? I think a new API shouldn't outright replace the old one, which still isn't finalized, but rather to complement it.

Since not all HW codecs will be able to support slice mode, this feature will be added later on to the spec as an extension and/or codec capability. In order to add this to the Vulkan Video specification, however, we need to have CTS tests written with at least two implementations passing it. Our first release does not have plans for slice mode.
The slice more will be added in the form of an extension structure to the current decode/encode command or as an entirely separate video command.