WIP: Add the VK_EXT_present_timing extension #1364
Hi I have created an WIP/RFC proposal for a Wayland protocol which is modeled after this extension (here). It aims to provide enhanced presentation timing requests and events for Wayland, and to provide support for this VK extension in particular. Given the early stages of the public discussion both at the Wayland protocol and the Vulkan extension level, (possibly significant) changes should be expected. Thanks!
I believe this extension would be useful in developing tools to measure video latency of sink devices, such as displays connected via HDMI, DisplayPort, etc. This purpose has similar requirements to the neuroscience/vision-science testing requirements presented by @kleinerm.
In order to develop such tools, timing information for when specific pixels of an image are transported is required and this timing information will be dependent on blanking interval durations. I believe this extension would be appropriate for reporting blanking interval durations, which would allow calculation of exactly when each pixel of an image is transported.
In terms of how this information could be presented in this extension, two fields would be relevant: exactly when an image begins to be presented and when image presentation has completed. It can be assumed that the remaining time is a blanking interval between images. Currently, I believe only the time when an image begins to be presented is available in these structs. I don't see any way to determine when an image completes presentation and between-image (vertical) blanking begins. With modes such as HDMI Quick Frame Transport, the blanking region can become quite large.
Video latency is relevant for audio/video synchronization (lip sync) between different sink devices, such as a video display and audio amplifier. It is also relevant for a user's selection of a display/mode for purposes such as esports, virtual reality, and neuroscience/vision-science testing. "Video latency" of a sink device, for these purposes, should be measured as the average time between when a pixel’s complete data arrives at a device and when that pixel is presented to the user as light.
This definition is effectively what is stated in the HDMI Specification 1.3 and higher, which is used in the HDMI "Auto Lipsync" feature (Spec 1.3, Section 8.9.1 “EDID Latency Info”):
Video Latency Measurement Tool:
An easy trick to measure the average video latency across a display's surface is to place a photodiode at the center of a display, which will account for different display scannout methods such as black frame insertion/low persistence, presenting the entire image at once rather than raster scanning, HDMI "Quick Frame Transport", variable refresh rate mode, etc. The time between when pixel information for the center of the image arrives at the display's input and when the photodiode detects this change in light is the average video latency for the display.
Thanks for taking the time to consider this purpose! I'd love to hear your thoughts.
I've come to believe that the idealPresentTime/optimalPresentTime mechanism doesn't work properly when the buffer readiness deadline (or even the commit deadline) is earlier than the presentation. This can happen when the presentation engine is compositing but also in other cases.
Consider the follow scenario: the display is refreshing at a 16ms interval, the PE requires the buffer to be ready 8ms before the presentation. A client uses VK_KHR_present_wait to start drawing at a present, the GPU finishes with the buffer 10ms later and the targetPresentTime is set to T+32. The client notices that it only takes 10ms so it sets the idealPresentTime to T+10 and the PE correctly answers with optimalPresentTime T+16. The next frame the targetPresentTime is set to T+16, the client again finishes the buffer at T+10 but it won't reach the present at T+16 because that present has a buffer readiness deadline of T+8.
Adding to that: when VK_KHR_present_wait gets called and the display is refreshing at a 16ms interval one would expect to have a budget of 16ms to finish the frame. If the PE requires the buffer to be ready earlier than the present this is not the case. In the example above the frame budget is 8ms. So while the problem above results in frames with an actualPresentTime != targetPresentTime the VK_KHR_present_wait extension is also flawed and reduces the frame budget. Waiting for the buffer readiness deadline of a present instead would solve this issue.
I'm currently working on a latency reduction solution and I would like use the present timing extension to predict the next present time as accurately as possible (in particular on VRR displays, on FRR the prediction can be done mostly just with an initial timing).
One annoyance currently is that both the EXT and GOOGLE present timings extension requires external synchronization on swapchains, and many of the swapchain functions can block for an extended period. This makes it hard to acquire timing information "as soon as possible".
Please consider adding some synchronization-free way to obtain the timing, just like how timestamp queries can be accessed without synchronization.
This extension allows an application that uses the VK_KHR_swapchain extension to obtain information about the presentation engine's display, to obtain timing information about each present operation, and to schedule a present to happen at a specific time. Applications can use this to minimize various visual anomalies (e.g., stuttering).
Fix some trivial spec build and validation issues: - Update some includes from .txt to .adoc. - Reserve different bits for VkSwapchainCreateFlagsKHR. - Remove disallowed contractions. - Mark pNext members in present timing structs as optional. - Typo fixes. - Update copyright years. - Remove conflicting VUIDs.
This change removes the 'presentID' member from VkAbsolutePresentTimeEXT and VkRelativePresentTimeEXT to rely on VkPresentIdKHR instead. If no present id is supplied, timing collection will reference id 0. Other minor changes: - Rename VkPastPresentationTimingEXT::presentID to 'presentId' for consistency with VK_KHR_present_id. - Change VkPastPresentationTimingEXT::presentId to uint64_t to match VkPresentIdKHR. - Add dependency on VK_KHR_swapchain and VK_KHR_present_id in the spec xml and appendix.
Add a new VkPhysicalDevicePresentTimingFeaturesEXT struct that exposes 3 features: - presentTiming: corresponds to the ability to collect present timing information via vkGetPastPresentationTimingEXT. This feature is required when VK_EXT_present_timing is supported - presentAtAbsoluteTime / presentAtRelativeTime: these are optional and correspond to the "present-at" capabilities of the extension, i.e. adding a VkPresentTimesInfoEXT in the VkPresentInfoKHR pNext chain.
@kleinerm Thank you for the feedback. Your comments have not been overlooked, this is definitely still a work-in-progress. I'm going at the low hanging fruits first to reduce the noise when going after the bigger comments / issues raised in this PR later on. This also leaves a bit of time for people to react to the smaller changes. I will reply to all the existing comments as I address them.
I do not have the expertise to comment on the specific Wayland issues you raised, but adding more queries such as the one you describe is planned.
- Remove remaining reference to VK_PRESENT_MODE_FIFO_LATEST_READY_EXT. - Add missing headings in appendix - Add missing description for VkPresentTimesInfoEXT - Reword VkSwapchainTimingInfoEXT::timeDomain xml comment - Change pNext wording to match current style - Specify order of elements in vkGetPastPresentationTimingEXT