WIP: Add the VK_EXT_present_timing extension

[cubanismo]

This extension allows an application that uses the VK_KHR_swapchain extension to obtain information about the presentation engine's display, to obtain timing information about each present operation, and to schedule a present to happen at a specific
time. Applications can use this to minimize various visual anomalies (e.g., stuttering).

See the proposal doc for more details.

Status

Because of the volume of comments, it has become difficult to follow the current status of the proposal.

Here is a list of issues that still need to be addressed, in no particular order:

• The spec assumes invariant VkSurfaceKHR capabilities in multiple places, e.g. at swapchain creation.
• VK_PRESENT_STAGE_IMAGE_HANDOFF_BIT_EXT is too controversial. Need to rename or re-spec it.
• VkPastPresentationTimingEXT's timingPropertiesChanged and timeDomainChanged are not the right
  way to handle properties updates. For example, time domain could just be reported as part of the timing result instead of invalidating results, with some fallback mechanism.
• VkPastPresentationTimingEXT does not provide enough details for dropped present requests.
• Some present parameters (vblank rounding, time domain, target present stage) are set as swapchain state via vkSetSwapchainTimingEXT or createInfo, rather than per-present in VkPresentTimesInfoEXT.
• Some platforms may perform the various present stages on separate hardware, requiring some kind of per-stage time domain.
• Spec does not provide any forward progress guarantee for timing results to be available.
• No way to synchronously wait for results.

This list is kept updated to follow current discussions.

Contributing

Because of the nature of the problem space this specification covers, the Vulkan System Integration working group has decided to move development of this specification to a public forum to make it easier to gather feedback and collaborate with developers across the various graphics ecosystems. However, to preserve the option of promoting the resulting extension to a Khronos ratified extension in the future, some guidelines (In addition to the usual contributing guidelines) must be followed when providing feedback or making contributions to the specification language.

• If you have high-level feedback that does not include sensitive IP or implementation details, feel free to comment on the merge requested directly.

• If you have specific wording changes or functionality you wish to add to the specification, please provide it as a pull request targeting the VK_EXT_present_timing branch of the KhronosGroup/Vulkan-Docs repository. This will require accepting the Khronos CLA before the pull request can be merged.

This is a relatively new process for Khronos, so feel free to provide meta-suggestions on how best to facilitate dialogue and foster engagement as well.

[CLAassistant]

All committers have signed the CLA.

[afrantzis]

Hi I have created an WIP/RFC proposal for a Wayland protocol which is modeled after this extension (here). It aims to provide enhanced presentation timing requests and events for Wayland, and to provide support for this VK extension in particular. Given the early stages of the public discussion both at the Wayland protocol and the Vulkan extension level, (possibly significant) changes should be expected. Thanks!

[emersion]

Maybe it would be a good idea to state explicitly the motivation for a separate relative timing target.

Applications submitting a single frame at a time wouldn't have any use for this struct I think (apart from maybe a slightly simpler interface), because they could create a suitable absolute target instead (last actualPresentTime + minPresentDuration).

However, applications queueing multiple frames at once don't know in advance when the presentation engine will choose to display their image. VkRelativePresentTimeEXT allows the application to specify a target relative to the previous actual present time, without having to manually adjust the absolute timing (n+1) depending on feedback n.

[hrydgard]

This would be great for stuff like presenting at a stable 30fps on a 60hz display.

[nvlduc]

I think spec language should refrain from making recommendations or suggestions about usage and intent. I know this PR has quite a few of these, which I might clean up, but I'd rather not add more here. The appendix, which could probably be fleshed out a bit, is a better place for these remarks. Alternatively, one of the large existing .NOTE sections may also be appropriate; I'll look into adding a comment to that effect somewhere.

[HansKristian-Work]

What does same mean here? Exactly the same value, or +/- some delta?

[HansKristian-Work]

I'm missing the "headroom" field from GOOGLE_display_timing. That feedback is useful for low-latency situations where you'd be able to time the main loop such that there is little time before GPU is done rendering and display unit scans out.

I suppose calibrated timestamps might work around that niche, but that would rely on swapchain and GPU working on the same timebase, and it wouldn't be able to take into account any compositing work happening outside the knowledge of Vulkan.

[timo-suoranta-varjo]

This language may be valid for other presentation modes, but is not valid for immediate present mode.

[timo-suoranta-varjo]

This language may be valid for other presentation modes, but not for immediate present mode.

[timo-suoranta-varjo]

I think this language is not what is intended.

[timo-suoranta-varjo]

How this extension interacts with immediate present mode?

Can we also make it possible to query vertical blanking duration, which is a fraction of refresh cycle duration?

[kleinerm]

Please reconsider this. I would like to ask for providing (at least as an optionally reported value) both minimum and maximum value for refreshDuration for VRR. One could always have special values like 0 to signal that specific info is not available/not yet available/not applicable to a given display setup, e.g., if a driver vendor doesn't want to implement additional info fields in initial implementations of the extension. Ideally it would be great if even more detailed info would be provided, e.g.,

• minimum duration (== maximum refresh rate), corresponding to what the display can do at current video mode/display link bandwidth/other limitations.

• maximum duration (== minimum refresh rate) of what the display can physically do without tricks like low framerate compensation (lfc). Typical values for current G-Sync or FreeSync/DP-adaptive sync displays are around 30 Hz - 48 Hz as far as i know.

• maximum duration (== minimum refresh rate) of what can be done on the display + gpu combo + driver via tricks like low framerate compensation, panel self-refresh etc., e.g., the solutions implemented by NVidia and AMD atm. for going below ~ 30 Hz refresh rate / above present intervals of 33 msecs.

• Ideally for VRR modes we'd even have some reporting of the "good quality range" in which duration can be varied quickly without causing strong perceptible visual artifacts like significant flicker of the image.

In my experience, some (many? most?) VRR panels do show a perceptible dimming of the image if the time-gap between successive presents is substantially increased beyond minimum duration, e.g., if min duration would be 6.944 msecs (on a max 144 Hz display), and max duration supported by the display itself would be 33.333 msecs (on a min 30 Hz display), then switching between presents with a spacing of 7 msecs and 17 msecs and back to 7 msecs may work without any user-perceptible flicker, but making bigger jumps from 7 msecs to 33 msecs and back will cause very perceptible and irritating flicker.

Similar, going beyond 33.333 msecs interval between presents would work as gpu's or panels would trigger an automatic refresh cycle (once maximum front-porch duration of the vblank is exceeded), but then block the app from presenting for a period of time (ie. at least one active scanout duration). Similar exceeding the physical maximum duration will trigger low framerate compensation which changes the timing characteristics wrt. flicker and blocking the application from presenting.

I am the core developer and maintainer of the popular open-source toolkit Psychtoolbox-3 which is used for basic research in neuroscience, vision-science, cognitive science, psychology and related basic medical research applications related to visual perception, eye care etc. The toolkit is also used for basic perception research in industry for future VR/AR applications and display technologies. Cfe. this VESA press release for how the software will be used for research into new HDR displays, display stream compression methods, VR and AR.. Without going into too many details here, be assured that precise visual presentation timing and precise/trustworthy visual presentation timestamping matters critically for many research applications. VRR is one of the most useful tools developed in the last decade for these paradigms, to allow fine-grained timing and framerate control. But we need presentation to be high quality, e.g., minimizing flicker when transitioning between different presentation intervals between presents. So having as much info about the timing properties as possible is very valuable to achieve that goal.

Since January 2020 Psychtoolbox implements a special VRR based presentation mode for research applications, which tries to make as good use as possible of AMD FreeSync with Displayport adaptive sync etc. to give fine-grained presentation timing control. This is currently done under Linux with classic native X11/GLX + OpenGL on top of AMD's open-source display drivers. A demo and test of VRR scheduling is part of Psychtoolbox under this link VRRTest.m.

I contributed some bug fixes and enhancements to AMD's Linux kernel driver to make this more robust and trustworthy for apps like mine. It works reasonably well, given the constraint of doing all scheduling of bufferswaps/flips in the user-space application, but one annoyance of the current approach is that my algorithm needs to know both minimum and maximum physical refresh duration of the display at a minimum. Ideally also the other properties mentioned above. Lacking api support mean users have to figure out minimum refresh rates themselves, e.g., by finding their displays specs online or in printed manuals, or by manually decoding EDID data or accessing root-only-readable privileged files, deep into the Linux debug filesystem, just to tell the application what these values are. Other useful properties like the ones mentioned above need to be guessed via heuristics. That's a quite tedious and error prone user experience that could be easily avoided in future presentation timing extensions like this one, by providing rather more details about display system timing characteristics rather than less.

Since recently we also support HDR display for research, using Vulkan's HDR extensions. However, reasonable timing can only be achieved under Vulkan atm. via rather gross hacks - and often not at all. Therefore my hope is that this extension can contribute to great progress for applications like mine wrt. research use of VRR + HDR, as well as for VR/AR.

Some of the issues requiring more detailed info about presentation engine properties could be solved in the Vulkan drivers or display drivers themselves, and i intend to contribute to the Linux FOSS drivers related to this extension where it is helpful, once the extension is finalized. But assuming not all problems can be solved on all target platforms perfectly, it will be neccessary for applications like mine to adapt or compensate for shortcomings or limitations of drivers, so more information and control is always good for use cases like research.

[kleinerm]

One thing that i would like to see - not sure if this would be the right place, but given it is related to presentation timing - would be a way for the application to optionally request or forbid the use of VRR mode, similar to the fullscreen_exclusive extension, where there could be three settings "Enable VRR", "Disable VRR" and "Let driver decide between VRR or FRR". Again, research applications would appreciate control about VRR vs. FRR on VRR capable systems, as VRR is very well suited and beneficial for some research paradigms, but detrimental to others. My toolkit currently implements such a switch on Linux+X11+Mesa via some hacks to manipulate X-Atoms "behind the back" of the Mesa graphics library, based on detailed knowledge about how this works on XOrg's X11 + Mesa DRI3/Present implementation. It works, but it is ugly and only works on FOSS drivers on Linux, not on other drivers or operating systems. A proper on/off/auto switch as part of this Vulkan extension could make this so much more sane, future-proof and portable.

[kleinerm]

Would this be a good place to also report if the swapchain's associated presentation engine is in principle capable of VRR, via a VkBool variableRefreshSupported? I'd like to have a way to find out if VRR is in principle supported on a swapchain's display (ie. driver + gpu + display support it and it is enabled by compositor/user). "variableRefresh" describes if VRR mode is active at time of query.

E.g., if connected display or gpu or display server/driver is incapable of VRR or if VRR is disabled by the user in some display setting, then both variableRefreshSupported and variableRefresh would be VK_FALSE. But if the setup is capable of VRR and just at this specific time of query VRR mode is not active, maybe because the window is not a fullscreen window or partially obscured, then variableRefreshSuppported would be VK_TRUE, but variableRefresh would be VK_FALSE.

This would be very useful for research applications but presumably also games and other apps to provide users with more accurate feedback/warnings or some troubleshooting instructions if the given use case requires VRR and VRR is not active, e.g., to figure out if it is because of obscured window / desktop composition kicking in, or instead because of unsuitable display/gpu or user error (=user disabled the required VRR feature in some display setting GUI or config file). My toolkit does this on Linux/X11, involving lots of non-portable hacks, but a proper and portable reporting mechanism in this extension would be splendid.

[versalinyaa]

swick said:

The only stage that makes sense to target is the LATCHED stage which we should also make required.

I agree with ishitatsuyuki below. I believe there exist VRR cases in which it makes sense to target a present stage other than LATCHED.

ishitatsuyuki said:

Two edge cases come to mind:

1. When the PE uses a variable latching target
2. With VRR where vblank can actually be delayed independent from latching.

Do we expose enough information in present_timing right now for 1.? For 2., I think the current design of specifying explicit target stages make more sense.

I agree with (2).

Can you clarify (1)? I want to avoid inferring the wrong understanding of what you mean by "variable latching target".

swick said:

So, the PE will simply get the time targetPresentTime - (PIXEL_VISIBLE - LATCHED) to calculate the time at which the presentation should be latched to hit the targetPresentTime at the PIXEL_VISIBLE stage.

The app can do the exact same thing itself. There is no reason to push this off to the PE.

I believe there will exist cases where the PE, if given a target PIXEL_VISIBLE time by the app, would be able to choose a target LATCHED time significantly more accurately than the app. Daniel already referred to the tuning that Mutter devs have done. I'm considering a layered PE stack whose layering and/or internal latency is deep enough that the LATCHED decision is divorced in time from PIXEL_VISIBLE due to unfortunate architecture quirks. In this case, especially when app's goal is low-latency VRR, it makes more sense for the app to target PIXEL_VISIBLE rather than LATCHED. To be more clear, I'm considering a VM with multiple proxy compositors between the app and display scanout, or a networked environment.

[ishitatsuyuki]

Can you clarify (1)? I want to avoid inferring the wrong understanding of what you mean by "variable latching target".

I meant that a compositor may dynamically adjust their latch-to-flip slop over time. For example, a compositor may be able to use a shorter slop when composition is not required.

[ianelliottus]

As stated above: "applications need to correctly position their geometry for when the presentable image will be presented to the user." From experienced people, I learned that anytime that anytime there is a difference between those 2 times, there can be visual artifacts.

[ianelliottus]

@nvlduc @cubanismo and @versalinyaa FYI ... Android has 2 other types of displays, at least one of which we'll want to add support for (I've started a dialog with the correct people): dVRR and MRR.

• dVRR (discrete variable refresh rate) involves panels that have many discrete refresh rates, and can switch between them without a mode switch. For example, there's a 120 Hz panel that can change the refresh duration in 1/240 (i.e. 4.168 msec) increments, from 8.33 msec up to some maximum.
• MRR (multi-refresh rate) involves a panel that has a few refresh rates, and can switch between them with a mode switch (e.g. 10 Hz, 60 Hz, 90 Hz, 120 Hz).

We'll definitely want dVRR addressed by this extension. Again, I'm learning more and will get back with you.

[krOoze]

What's the diff? 4 ms increments is like 120, 80, 60, 48 Hz. That's basically the same few refresh rates. Feels like a distinction with very little difference.

PS: isn't MRR just FRR?

[AMouri]

MRR sounds distinct from FRR, in that I don't see language in FRR that the refresh rate can change.

In Android, the operating system, can change display modes on the fly with MRR, without involvement from the application. For instance, a display normally running at 120Hz might instantly drop to 60Hz because of low battery conditions, or other contraints.

I think the point is that both dVRR and MRR are not quite VRR, because there are constraints not described in this extension, but they are not FRR either.

[DanOscarsson]

I have now followed this extension for a long time.
Is there a plan for it to be added soon?
For video display I have used VDPAU as it has presentation scheduling and feed back. Not so Vulkan.
So to be able to switch to Vulkan I need a presentation extension.
Vulkan now have video decoding. Is it not time for at least a basic video presentation extension this year?

[ianelliottus]

Does this correlate to when rendering completes?

[chpi7]

In version 5, the user can request results for specific present ids in vkGetPastPresentationTimingEXT.
How should

The VkPastPresentationTimingEXT::presentId of the pPresentationTimings entries can be set to ids referencing outstanding presentation request results. When doing so, the function will only return the results for those presentation requests

be interpreted in the following scenario?

• The internal queue contains results for the present ids [1, 2, 3] where 1 is the oldest and 3 the most recent result.
• The user calls vkGetPastPresentationTimingEXT with one instance of VkPastPresentationTimingEXT with VkPastPresentationTimingEXT::presentId = 2.

Should the internal queue be [3] or [1,3] after the call to vkGetPastPresentationTimingEXT?

Or more general, should results older than the ones requested by the user be dequeued and discarded or should they be able to be returned on a subsequent call? "Queue" could imply that elements are removed starting from the oldest, but it would make sense for the application to be able to query the results in random order.

[jjulianoatnv]

I think the voice should be changed as I suggest in these proposed edits.

[jjulianoatnv]

Would it be advantageous to add a tri-state data member to VkPastPresentationTimingEXT that indicates one of:

• The image was known to display without tearing (was ready in time).
• The image was known to display with tearing (was ready late).
• Unknown whether tearing did or did not occur (the implementation cannot determine this).

[nvlduc]

That seems like it implies having VK_PRESENT_STAGE_IMAGE_FIRST_PIXEL_VISIBLE_BIT timing information and a vblank time to compare it to. If the implementation has access to those, it should just expose them: otherwise, the next question is "how late was the image". There's nothing in the spec currently regarding vblank specifically though, and I'm not sure we'd want to go down that way.

[jjulianoatnv]

Perhaps add the following questions to the issues section:

• Should it be allowed for VkPastPresentationTimingEXT::presentId values to be specified out of order in the pPresentationTimings array?
  • My preference is to add valid usage that disallows this behavior.
• Should it be allowed for VkPastPresentationTimingEXT::presentId values to skip IDs? If so, then do skipped IDs which would have been marked completed release their slit in the internal swapchain present timing queue? Or do they stick around to be queried by a later call to vkGetPastPresentationTimingEXT? Note that not retiring the slots means that the internal present timing queue is not technically a simple queue data structure.
  • My preference is to allow skips, and for skips to retire skipped entries prior to the last-returned completed entry. If there are retired entries between the last-returned completed entry and the first-returned skipped entry, then these completed entries can remain in the queue. (I.e., the implementation deques everything prior to the last-returned completed entry.)

Also, I'm failing to be convinced that the existing spec language enforces that an implementation must: return values which were used in prior calls to vkQueuePresentKHR within pNext-chained VkPresentTimingsInfoExt structs that have not already been retired by returning a completed VkPastPresentationTimingEXT. This is phrased to imply:

• Don't make up presentId values beyond what the app is expecting.
• Don't return present stages beyond what the app has requested.
• No unexpected duplicate VkPastPresentationTimingEXT structs.
• Fail CTS if the implementation violates these expectations.

Or, maybe we do want the implementation to be able make up things the app did not request, and apps have to be tolerant of made up things. Either way, the spec language needs to make this explicit via requirement on the Vulkan implementation or requirement on the app, and I'm not seeing spec language that accomplishes this.

[nvlduc]

Tagging @chpi7 here as this is also relevant for their feedback.

This aspect of the proposal was discussed in the last meeting and it seems like not everyone can implement it as proposed. More specifically, the synchronous behavior of the vkGetPastPresentationTiming isn't achievable by everyone. As this wouldn't make sense as a feature bit, the consensus was to just remove this entirely and keep the simple "fire hose" original design.

[AMouri]

should there be language that the VkPresentStageFlagsEXT must be "valid", or one that the PE supports queries for?

[nvlduc]

Unfortunately, no. Due to the dynamic nature of VkSurfaceKHR properties, implementations can't guarantee that the present stage specified at present time will be valid by the time the PE processes the presentation request. If a presentation request targets a present stage that suddenly becomes unavailable, the corresponding VkPastPresentationTimingEXT struct will not contain any data for the requested stage. I'll try to make that more explicit, thank you.

[cubanismo]

This seems very unfortunate. That's exactly the type of situation VK_ERROR_OUT_OF_DATE_KHR was designed to avoid. The presentation is supposed to guarantee the presentation engine state doesn't change between the presentation request and actual presentation unless it returns that error.

[nvlduc]

Seeing how this extension allows you to schedule presents really far into the future, relatively speaking, I don't know that such a guarantee can be upheld as strongly as before.

[jjulianoatnv]

Maybe reword:

There is no requirement for any precise timing relationship between completion of a present stage and the availability of timing information associated with completion of the present stage. However, the associated timing information must: be made available in finite time after completion of the present stage.

[jjulianoatnv]

Or maybe:

Timing information associated with completion of a present stage must be made available to vkGetPastPresentationTimingEXT within finite time after the present stage completes.

I like this proposed grammatical structure.
I'm remain not-happy about "must be made available". Perhaps there's a better way to reword that phrase.