Reviewing the meaning of update for multi-window apps across displays with varying refresh rates
#817
Comments
mitchmindtree
added a commit
to mitchmindtree/nannou
that referenced
this issue
Dec 14, 2021
This begins work on improving the latency between user input and when the window's surface texture is presented to the display for wgpu's FIFO present mode. Note that you can already improve latency by providing a `SurfaceConfigurationBuilder` with `wgpu::PresentMode::Mailbox` specified which is available on *most* platforms. This PR however aims specifically at improving the `Fifo` present mode as it is the default present mode and only mode guaranteed by WGPU to work across *all* platforms. **The Problem** The primary issue is that under the `Fifo` present mode, requesting the surface texture to draw to can block for up to a frame (~16ms at 60fps) depending on how recently the last frame's surface texture was submitted. Once the surface texture is acquired, we then draw to it and must wait up to another frame before that texture is presented to the window on the user's display. This means we can frequently get up to 2 frames (~33ms at 60fps) worth of total latency between user input and visualisation of that input. This is particularly noticable when controlling a camera with a mouse, or interacting with any kind of GUI. **Potential Solutions** Ideally, it would be nice if `wgpu::Surface::get_current_texture` provided a non-blocking alternative, or some way to query whether or not requesting a new surface texture would block. I imagine this would allow us to continue to collect new user input in the mean-time and perform an `update` right at the moment the new texture is acquired. This would allow us to cut down on up to a frame worth of input latency. In lieu of this we can get close to the same behaviour by attempting to predict the moment we expect a new surface texture to be available and avoid calling `get_current_texture` until we believe it would no longer block. We can do so by keeping track of the moment at which we acquire each texture, and then returning `ControlFlow::WaitUntil(next_frame)` where `next_frame` is the moment we acquired the last texture *plus* the duration of a frame interval. This is the approach currently taken in this PR, and does result in a noticable improvement in input latency. **Caveats & TODO** The duration of a frame differs between displays based on their refresh rates. E.g. the minimum frame interval of a 60hz display is ~16ms, for a 144hz display it is ~7ms. Currently, this PR just assumes 16ms as a proof-of-concept, however ideally we'd retrieve the actual refresh rate from somehwere. `winit` does provide a way to query the *supported* video modes (and in turn, refresh rates) of the monitor upon which a window is currently placed, however it does not appear possible to retrieve the *active* refresh rate. One option might be to simply use the interval duration of the highest rate to avoid missing any frames, at the risk of retaining some latency in the case that a lower refresh rate is active. This solution gets much fuzzier when we start to think about apps with multiple windows across multiple displays with varying refresh rates (not uncommon for installations where you hav a GUI/control window and one or more visualisation windows across different displays/projectors). Resolving nannou-org#817 would help to make this a little clearer.
|
@mitchmindtree On a side note, I am looking for a high-level doc that talks about the order in which functions are called by the framework. For example |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While working on a PR to improve latency under the default
Fifopresent mode I was reminded of a semantic issue we have with our current concept of anupdate.Currently, we have one type of
updateevent. Only oneupdateevent function can be provided to the app builder, regardless of the number of windows or displays available.We also currently make the guarantee to provide a single
updateevent prior to each window'sviewcall per frame. This works if all windows are on the same display, but no longer make sense in the case that the app has multiple windows across different displays with different refresh rates.Potential Solution
One option might be to remove the concept of a single
updateevent, and instead allow more granularity over how we specify update events. Here are a few potentials that come to mind:updates - These could be specified under theWindowBuilder(rather than theAppBuilder) and would be specific to eachWindow. The behaviour would be similar to the current behaviour, where we would provide a singleupdateevent prior to eachviewevent.update- This could act as the defaultWindow::updatefunction, used for each window that has not specified its own update function. This would result in the same behaviour as is currently the case for apps with only a single window. For multi-window apps it would differ in thatupdatewould be called prior to each call toviewfor each window, not only prior to all calls toviewfor each window.It may also be worth reviewing what the
LoopModemeans w.r.t. multi-window applications. E.g. I can imagine a situation where it might be useful to set aWaitmode for a GUI control window and a regularRefreshSyncmode for a visualisation. Further, aRateloop mode may not make sense for a window-specific update seeing as we can't force the display to a particular rate, however it may make sense to allow for specifying one or moreRateupdate functions in theAppallowing a user to specify functions that get called at a particular interval, totally agnostic of the window refresh rate.The text was updated successfully, but these errors were encountered: