Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Behavior of window.requestAnimationFrame and window.requestIdleCallback when presenting #225

Open
toji opened this issue May 2, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@toji
Copy link
Member

commented May 2, 2017

Short and sweet: When we're in WebVR presentation mode on devices without an external display, should the page continue pumping the window rAF and rIC loops?

For context: In Chrome at least when we kick into VR mode on mobile we stop the main page from running it's normal compositing process for performance reasons. This has a natural side effect of preventing rAF and rIC from running unless we take explicit (and slightly hacky) steps to avoid that. It may make sense, though, to treat this like any other time we hide the page (switching tabs, etc.) and suspend rAF simply because the page isn't actually meant to be updating and rAF is meant to be tied to page animation. (I'm not as sure about the intended behavior for rIC.)

That would create a behavioral disparity with desktop, though, which WILL have the page visible while presenting and does want to keep rAF running. That could create a situation where developers test code primarily on one platform and are surprised by the behavior of the others.

Would love opinions on this from other vendors!

@ddorwin

This comment has been minimized.

Copy link
Contributor

commented May 2, 2017

As noted in this Chromium commit, there may be a race condition between the window rAF and the beginning of the WebVR mechanism. Whatever the outcome of this issue, we should ensure the behavior is clearly defined and that there is a reliable path for authors.

@toji

This comment has been minimized.

Copy link
Member Author

commented May 2, 2017

Quick update: This was discussed on the WebVR implementors call today and the general consensus is that if the page isn't visible it shouldn't be pumping rAF. (Didn't discuss rIC as much, because not everyone implements it.)

In the future exceptions would probably need to be made for DOM content that is displayed in VR via a layer or similar mechanism, but at that point it's reasonable to expect that the DOM compositor will have to be running to generate that content anyway, and rAF will naturally run as a result. Some questions came up about what framerate the rAF should run at in that scenario (monitor vs. HMD?) but if the in-VR layer was able to be updated asynchronously from headtracked rendering on the quad it wouldn't be a user comfort issue.

@kearwood

This comment has been minimized.

Copy link
Collaborator

commented May 3, 2017

Also, as a follow-up from the WebVR implementers call, we wish to avoid any artifacts related to video framerate if parts of the page are mirrored into the headset.

For example, if 24fps video was rendered in an HTMLVideoElement it is temporally upsampled to the 2d monitor's 60hz framerate. This is commonly done by repeating some frames, causing a variability in frame duration as the 24hz video catches up to the 60hz output rate. Effectively, half of the video frames are presented for 2 vsync cycles and the other half of the video frames are presented for 3 vsync cycles:

vsync: 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
       30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

video: 00 00 01 01 01 02 02 03 03 03 04 04 05 05 05 06 06 07 07 07 08 08 09 09 09 10 10 11 11 11
       12 12 13 13 13 14 14 15 15 15 16 16 17 17 17 18 18 19 19 19 20 20 21 21 21 22 22 23 23 23

video frame uniformity: 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3

If a piece of the page, containing such a video is then sampled into a VR layer at 60hz, it will be again upsampled. A VR display could be running at 90hz typically:

vr vsync:     00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
              30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
              60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

source frame: 00 01 01 02 03 03 04 05 05 06 07 07 08 09 09 10 11 11 12 13 13 14 15 15 16 17 17 18 19 19
              20 21 21 22 23 23 24 25 25 26 27 27 28 29 29 30 31 31 32 33 33 34 35 35 36 37 37 38 39 39
              40 41 41 42 43 43 44 45 45 46 47 47 48 49 49 50 51 51 52 53 53 54 55 55 56 57 57 58 59 59

video frame:  00 00 00 01 01 01 01 02 02 02 03 03 03 03 03 04 04 04 05 05 05 05 06 06 06 07 07 07 07 07
              08 08 08 09 09 09 09 10 10 10 11 11 11 11 11 12 12 12 13 13 13 13 14 14 14 15 15 15 15 15
              16 16 16 17 17 17 17 18 18 18 19 19 19 19 19 20 20 20 21 21 21 21 22 22 22 23 23 23 23 23

video uniformity: 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 (Janky!)


If we were upsampling the video directly from 24hz to 90hz:

vr vsync:     00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
              30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
              60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

video frame:  00 00 00 01 01 01 01 02 02 02 02 03 03 03 03 04 04 04 05 05 05 05 06 06 06 06 07 07 07 07
              08 08 08 09 09 09 09 10 10 10 10 11 11 11 11 12 12 12 13 13 13 13 14 14 14 14 15 15 15 15
              16 16 16 17 17 17 17 18 18 18 18 19 19 19 19 20 20 20 21 21 21 21 22 22 22 22 23 23 23 23

video uniformity: 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4

@kearwood

This comment has been minimized.

Copy link
Collaborator

commented May 3, 2017

The video frame uniformity issues would also apply to CSS animations and WebGL canvases that will be projected onto "floating quad in space" layers or onto textures within the VR presentation.

I believe we should always prioritize the VR experience when a VR presentation is active.

The key question should be if the effective framerate of the 2d page should bump up to the HMD's framerate in the event that parts of it are captured for VR presentation, or if it should remain at 60hz.

This might not be a simple task for browser implementers, and could slow adoption for the WebVR 2.0 API if required.

If the video frame uniformity is the issue affecting users the most, perhaps we should first implement a specialized video VR layer or implement optimized functions for decoding video directly to WebGL textures.

@kearwood

This comment has been minimized.

Copy link
Collaborator

commented Jan 30, 2019

Regarding the "requestIdleCallback", there may be issues with running that out-of-sync from the normal XR framerate, as effectively there would be "idle" work happening during the period of a XR frame that would not be ideal, such as when the VR compositor is performing the hard-realtime task of compositing asynchronously transformed / warped layers at vblank.

For many (most?) tracked XR headsets, the "requestIdleCallback" would ideally occur immediately after this vblank interval occurs (often signaled by lower level api's blocking until this ideal time).

One reason that XR sites themselves would want to use "requestIdleCallback" is to move non-rendering tasks outside of "requestAnimationFrame" so that the XR runtime environment can optimize the start of "requestAnimationFrame" based on prior frame times. The goal of such systems include moving the rendering as late as possible to ensure the least work done by the reprojection (less delta time between actual frame display and the predicted pose). If idle work, such as managing game state, updating physics, and playing audio also occurs within "requestAnimationFrame" and there is no longer a "submit" call like in WebVR, the precise time needed for the rendering phase of the frame can not be inferred.

One solution is to have a "requestIdleCallback" added to the XRFrame, triggered some time in the frame after the "requestAnimationFrame". Another would be to re-introduce the "submit" call on the XRWebGLLayer and have it return a promise that resolves when the rest of the per-frame processing can continue.

@kearwood

This comment has been minimized.

Copy link
Collaborator

commented Jan 30, 2019

Caveats:

Synchronizing window raf to XRSession raf would preclude any possibility of multiple simultaneous XRSessions for separate devices with un-aligned frames or varying frame rates.

Alternate solution:

Sync window.requestIdleCallback to XRSession but not alter window.requestAnimationFrame

@toji toji self-assigned this Feb 13, 2019

@toji toji modified the milestones: Next Working Draft, CR for 1.0 Feb 13, 2019

@NellWaliczek NellWaliczek removed the agenda label Feb 13, 2019

@cwilso cwilso modified the milestones: July 2019, August 2019 Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.