F2F Followup: Render Loop Proposal #188

Open
toji opened this Issue Feb 8, 2017 · 10 comments

Projects

None yet

4 participants

@toji
Collaborator
toji commented Feb 8, 2017 edited

A persistent issue that has come up with WebVR as it exists today is the details of how the render loop works. Things like when the frame data is allowed to update and what should happen if submitFrame is not called have remained somewhat stubbornly unresolved. Additionally, there's calls from the web platform side to tie the canvas presentation mechanism more closely to the commit() or transferToImageBitmap() patterns being established by OffscreenCanvas. This proposal attempts to give us a basic frame flow that we can work off of, and usefully extend in the future.

Issues being addressed:

  • Define when VRFrameData is updated (#81, #112)
  • Clarify behavior around submitFrame (#69, #138, #170)
  • Restructure how canvas rendering is presented to the display (#141, #170, #174)
  • Resolve potential confusion with multiple requestAnimationFrame variants. (#171)

Overview:

The first big change proposed here is to get rid of VRDisplay.requestAnimationFrame and VRDisplay.submitFrame and combine them into a single, promise-returning call: VRSession.commit. commmit would function like submitFrame does today, taking any dirty canvases from the layer list and pushing their contents to the VR compositor. It's important that we can continue doing this explicitly so that the render loop can dispatch it's rendering work as soon as possible to reduce latency, then continue doing other work (rendering a mirrored view, simulation logic, etc.) in the same loop. The promise returned by commit resolves when the next frame is ready to be rendered, effectively allowing it to take the place of requestAnimationFrame as well.

(For the record this isn't too far off from what's being discussed for OffscreenCanvas)

To pre-emptively answer the question of why we wouldn't just want to use gl.commit() for this, as suggested by #174: We do plan on supporting multiple layers, each potentially with their own canvas eventually. In that scenario how would you update them all in sync? Which commit do you use to drive the render loop? When does the context commit switch from running at the monitor framerate to the VRDisplay framerate? Having the commit reside with the VR system that's using it rather than one of it's sources appears to be a more consistent solution.

As for frame lifetime the WebVR spec currently defines that the VRFrameData returned is the same between calls to submitFrame. This is so that apps composed of multiple libraries that all query their own frame data are in sync. This was problematic because submitFrame wasn't always called, such as in the case of magic window uses. This proposal changes that to say that the frame data is update with each commit resolve. So in essence the app must pump the render loop in order to continue receiving new frame data. This works better than it would have previously due to the proposal in #179 that makes even magic window mode into an explicit session that must be started and ended. So now any use of VRFrameData can be considered to occur while "presenting" (with different platforms able to make different choices about when that's allowed.)

I would expect that on many systems (like Chrome) the implementation will actually resolve the commit promise when new frame data has been made available, possibly over an IPC channel, but other platforms may worry that retrieving the frame data at the beginning of the callback like that may introduce unwanted latency. For those platforms they can simply defer fetching the frame data until the first call to getFrameData, and only reset the dirty bit on that data when the next requestVRFrame callback is ready.

Side note: While it's not part of this proposal future API revisions could also introduce an 'updateFramePrediciton' function as has been suggested by Microsoft in the past to force a mid-frame frame data refresh if we felt it was necessary.

Combined these changes create a more tightly controlled render loop with more predictable behavior that's harder to use "wrong". Hopefully that outweighs a couple of minor oddities listed below.

Rough IDL:

partial interface VRSession {
  Promise<DOMHighResTimeStamp> commit();
};

Example usage:

let vrSession = null;

function RenderLoop() {
  if (vrSession) {
    // VR path
    let frameData = vrSession.getFrameData(frameOfReference);
    DrawVRFrame(frameData);
    vrSession.commit().then(RenderLoop);
  } else {
    // Non-VR path
    DrawFrame();
    window.requestAnimationFrame(RenderLoop);
  }
  UpdatePhysicsOrSomething();
}

// Kick off non-VR rendering
window.requestAnimationFrame(RenderLoop);

function onEnterVRClick() {
  vrDisplay.requestSession().then(session => {
    vrSession = session;
    SetUpLayers();
    RenderLoop();
  });
}

Also, one really nice side effect of using promises is that with async/await support we can get something that looks an awful lot like a more traditional native game loop, complete with blocking swap!

async function RenderLoop() {
  do {
    // Render render render
  } while (await vrSession.commit())
}

Oustanding questions:

  • If we use a promise for the rAF mechanism it doesn't leave us with an analog for cancelAnimationFrame. Do we care?
  • What happens if you call commit multiple times? Reasonable options are to reject the promise or silently discard the rendering and resolve with the previously scheduled frame.
  • Starting up a render loop that's been paused looks a little weird in this model. You'd call commit() without necessarily having rendered anything. Is that a problem?
@RafaelCintron
Collaborator

One of the essential aspects about RAF is that it allows the user agent to throttle queuing of unnecessary work. For instance, the traditional RAF associated with the window is not raised when the document is on a hidden tab, minimized, or otherwise invisible. In Edge, if the GPU is bogged down with work, we throttle the RAF so that the GPU can keep up with the work the CPU is providing it. Failure to do this can lead to poor performance as frames are dropped on the floor and prevent the CPU/GPU from rendering other windows in the system in a timely manner. The nice thing about RAF is that it is a "pit of success" when it comes to expectations around frames. One callback == one frame; no more, no less. You can't get into a situation where you queue more than one frame of work because the API simply won't let you.

Given this, it is essential that we prevent commit from queuing more than one frame of work during JavaScript execution. Rejecting the promise is one avenue. I would prefer that we throw an exception to make it clear to the developer that they're doing it wrong. Dropping frames means wasting CPU/GPU cycles for frames the user never sees, a problem we do not have with today's approach.

Another disadvantage of the new approach is there is no way to signal the last frame or an "I am done now". Committing the last frame means you're forced to sign up for another one. I suppose you can call exitPresent right after you call Commit, but we still have to run the Commit's promise rejection code unnecessarily. The current approach also does not have this problem.

I agree having a similar look to the traditional game loop is attractive but the new approach seems to create more problems than it solves. What am I missing?

@toji
Collaborator
toji commented Feb 9, 2017

Thanks for the speedy feedback! I figured this one would spur some good discussion. :)

First off, I'd like to point out that I consider there to be effectively two different but related proposals here: The frame data lifetime and the render loop shape. I totally understand having concerns about the commit proposal, but I'd also like to know if you have any concerns about tying the frame data's lifetime to the render loop, whether that be done via rAF or a promise?

Putting that aside, though, I'm not sure if I'm clear on the potential issues with a promise-based loop.

In Edge, if the GPU is bogged down with work, we throttle the RAF so that the GPU can keep up with the work the CPU is providing it.

Is there a reason I'm not seeing that this can't be done with a promise? We can delay resolving it indefinitely.

You can't get into a situation where you queue more than one frame of work because the API simply won't let you.

With rAF today you can't queue up multiple frames but you can queue up multiple callbacks per frame by calling rAF repeatedly. What I was suggesting was that repeated calls to commit() could act the same way. Just queue up multiple promises to be executed on the next display vsync. So:

vrSession.commit().then(funcA);
vrSession.commit().then(funcB);
vrSession.commit().then(funcC);

Is equivalent to:

requestAnimationFrame(funcA);
requestAnimationFrame(funcB);
requestAnimationFrame(funcC);

Actually we could probably have the commit variant even return the same promise object for each call (for the same frame, anyway). Not sure if that provides a concrete benefit, but it's architecturally nice.

I guess the bigger question, and what you may have been getting at, is what do if the developer does something like this:

DrawSomeStuffToTheLayer();
vrSession.commit();
DrawSomeMoreStuff();
vrSession.commit().then(renderLoop);

But we've already established that to be a problem with the current rAF/submitFrame pattern anyway. That's the scenario that I suggested silently dropping the rendering work done for (but still returning the existing outstanding promise so that nobody misses a frame). Console warnings are probably a more appropriate mechanism for communicating the undesired behavior at that point.

Another disadvantage of the new approach is there is no way to signal the last frame or an "I am done now". Committing the last frame means you're forced to sign up for another one. I suppose you can call exitPresent right after you call Commit, but we still have to run the Commit's promise rejection code unnecessarily. The current approach also does not have this problem.

Actually, now that you bring that up I'm suddenly concerned about a flaw in the current system. If we call vrDisplay.requestAnimationFrame and then vrDisplay.exitPresent immediately after we've effectively dropped that callback. That seems bad. I'd actually prefer and explicit reject to happen.

The concept of "last frame" is a funny one anyway, because I have a hard time imagining a scenario where commit(); exit(); makes sense. I guess maybe if you had a video and wanted to boot the user from VR mode on the last frame? (That sounds unpleasant.) I think as a developer I would consider this to be a benign race, with no guarantee that the frame would actually be shown before presentation ended. But most of the time the exit signal is going to be triggered by some event or other async input, which means you'll probably discover that you want to end the session while a frame is outstanding, and I don't expect anyone to catch that signal, wait for the next render loop cycle, draw a full frame, and THEN end the session.

Anyway, I'm not actually trying to argue that this is unquestionably better than a traditional rAF, I'd just like to understand your points a bit better. I think we could evaluate a few different options here as well. For example, we could keep the submit and rAF as a single call but do it callback-based if the use of promises prove to be a sticking point. We could also discuss having an implicit submit at the end of the animation loop callback, WebGL style, but that making mirroring trickier AND makes controlling latency harder, so I've been assuming the other UAs would not be in favor of it.

@AlbertoElias

I like the idea of using Promises and I think it could emulate a the benefits offered by raf currently. Also, I believe cancellable promises have been worked on for some time, and we could also standardised on some boolean check that the developer could set to false when they want to stop the promise based render loop for the meantime

@toji
Collaborator
toji commented Feb 9, 2017

FYI: It's my understanding that there were significant technical issues with cancellable promises and that they are no longer being considered for addition to the spec.

I don't object to a boolean indicating that you don't need the next frame, but I also don't see much harm in simply letting the developer ignore the returned promise when they're done?

@toji toji modified the milestone: 1.2 Feb 9, 2017
@AlbertoElias

Yup, I agree that the promise based implementation doesn't seem to have any major drawbacks and I wasn't a big fan of re implementing raf and calling it the same

@RafaelCintron
Collaborator

@toji , The main thing I do not want to have happen is for the following case:

DrawSomeStuffToTheLayer();
vrSession.commit();
DrawSomeMoreStuff();
vrSession.commit().then(renderLoop);

to render DrawSomeMoreStuff on some hardware but force user agents to drop frames on other hardware if the GPU cannot keep up with the work. Instead, I think we should be clear to developers that following this pattern means DrawSomeMoreStuff gets completely dropped on the floor in all instances. Returning the previous promise on commit and warning in the console for this scenario seems reasonable to me.

Good point about the last frame being normally triggered as a result of some other async event. I agree this is a relatively minor drawback of the promise based approach.

I agree we should tie updates to the FrameData with resolution of the promise.

One thing I thought about after my initial reply. Suppose you have the case of the Three.js application with disparate pieces of code. Does each piece of code hook requestAnimationFrame on the VRDisplay itself and some primary code hooks it last and calls submitFrame? If so, the new approach will mean that the master code will need to do the moral equivalent of requestAnimationFrame to get everyone started. It will also have to pass around the subsequent promise so that the subordinate pieces of code can call then(). So, potentially more bookkeeping if that's how Three.js behaves, albeit relatively minor.

@toji
Collaborator
toji commented Feb 16, 2017

Talking with @bfgeek about this particular issue, there's still a desire to merge this functionality with the CanvasRenderingContext.commit() concept which is looking more and more concrete. This would slightly complicate some things but make others more elegant, so I think it's worth discussing.

It's useful to consider where we want things to go in the future with other types of layers. Let's presume that we have a VRSkyboxLayer that takes in some images to present as a skybox and a VRVideoLayer that takes a video source and defines a quad in space where it should appear. A simple 3DoF video player could be built with zero WebGL by setting a theater environment to the skybox and positioning the video quad where the screen should be. In this scenario, having javascript pump the render loop is nonsensical. The video plays at it's own rate and the UA will be able to handle updating it with lower latency than JS+WebGL ever will, and the skybox doesn't update at all, it just needs to be re-rendered with the right pose. (And some APIs have special mechanisms to handle that case.) So this should clearly be a "set and forget" situation.

Coming back to the present, we can look at the VRCanvasLayer and see that there are occasions that should work the same way. For example: If the app just wants to present a splash screen it should be able to render it once and have the UA reproject it repeatedly while the page loads other resources. When we support multiple canvas layers we should be able to render the main scene to one canvas and some headlocked UI to another (Gaze reticle, etc.) The headlocked layer probably needs to be updated infrequently, so we shouldn't require it to produce a fresh image every frame in order to continue displaying, even if the main scene is updating each frame.

This implies two things about the design:

  1. Layers should be able to commit imagery independently.
  2. The UA will need to keep track of it's own VR render loop that is not dependent on the page activity. The page can observe the VR render loop, but does not drive it.

These two items probably require a formal concept of a VR compositor (no JS interface, just a spec description.)

With that in mind, I'll propose a slightly different flow: Use either CanvasRenderingContext.commit() when the canvas is a layer source or, if that doesn't seem workable, a new VRCanvasLayer.commit() to push the content of a Canvas to the VR compositor. The returned promise resolves when the current VR compositor frame has been presented. You can render frames and call commit() at any time, and the newly produced layer imagery will be associated with whatever frame the compositor is currently on. (This alleviates the need to call `commit() on something just to "get into" the render loop.) If multiple layers all call commit within the same frame they all get back the same promise. If a single layer calls commit multiple times in a single frame then only the most recently committed imagery is used, but both commits would get the same promise back. This should easily allow things like debug overlay layers to run independently of other content.

Everything committed in the space of a single frame is guaranteed to show up simultaneously on the display. Frame boundaries are defined by the VR compositor notifying the page when a frame has been completed. Since pages are single threaded they won't process that until javascript releases control back to the page, giving the app a simple mechanism for sync. (If you want multiple layers to display at the same time commit them all within a single callback. Note that committing and rendering don't necessarily have to happen at the same time.) Pose updates are available on the frame barriers.

Mirroring continues to work effectively the same way it does now. Anything you draw to a normal canvas without calling commit gets presented to the screen at the end of the callback. For offscreen canvases you can mirror with transferTo/FromImageBitmap.

It's a pretty big shift, but this feels to me like a pretty forward looking pattern. I'd love everyone's thoughts on it!

@toji
Collaborator
toji commented Feb 17, 2017

One random follow up thought from my musing on the subject last night: One concrete benefit I could see to having commit() on the layer rather than the canvas context is that you could have multiple layers driven by a single context (enabling shared resources between layers). In other words, you could have multiple layers who's source is the same canvas, but you could draw separate content for each layer and commit individually. If the commit from the context is used then multiple layers pointing at the same canvas source would all get the same imagery on commit, which is pretty useless and we'd probably want to jump through some hoops to disallow it.

@mkeblx
Contributor
mkeblx commented Feb 17, 2017

@toji If you commit from the session can you not do everything you want, or do really need explicit control per layer for commits?

// 3-DoF video player, no WebGL
var skyboxLayer = new VRSkyboxLayer(imgEl);
session.layers.push(skyboxLayer);
var videoLayer = new VRVideoLayer(videoEl, w, h, frameOfReference, position, /*etc*/); // 2D
session.layers.push(videoLayer);
session.commit(); // set and forget no?

// every frame of VR compositor uses latest state of videlLayer.videoEl, skyboxLayer.imgEl, etc.

Come to think of it scenarios like this (no WebGL canvas) what is (any) explicit commit doing? Do different layer types have different commit behavior, where some don't need commit per frame but others do?

--
Also, session commit of multiple layers with same canvas source would have same content but different end display due to layerBounds differences, or other layer properties.

var l1 = new VRLayer(canvas, mainBounds);
var l2 = new VRLayer(canvas, gazeCursorBounds);
session.layers.push(l1, l2);
renderLoop() {
  drawMainScene(mainBounds);
  drawCursor(gazeCursorBounds);
  canvas.commit().then(renderLoop);
}
@toji
Collaborator
toji commented Feb 19, 2017

In your first code snippet I really don't think there should be a commit, unless you simply wanted to track the frame timing. Also, thanks for pointing out the possibility for using layer bounds to draw multiple layers with the same canvas! I hadn't considered that before!

There's pros and cons for both per-layer commits and session level commits. I think we could plausibly go either way and have solid reasoning for the choice. As far as I can see it's mostly a question of more explicit layer synchronization control vs. more flexible layer composibility.

(Of course none of this matters until we support multiple layers. Up to that point the two methods are identical, but it's worth considering the multi-layer case now so we don't have to change or complicate things down the road.)

I think it's probably a given that we want a way to guarantee that you can update N layers and have the new imagery for each layer be displayed on the HMD simultaneously. Relatedly, a way to indicate that all layers have been updated and can be submitted early is desirable, so we can have the imagery in flight while the page is doing physics simulations and such. A session-level commit handles this really cleanly:

var sceneLayer = new VRCanvasLayer(canvas1);
var cursorLayer = new VRCanvasLayer(canvas2, { headlocked: true });

function renderLoop() {
  drawScene();
  drawCursor();
  session.commit().then(renderLoop);
  updatePhysics();
}

It can also handle sparse layer updating by speccing that we only update imagery from canvases that are currently dirty (which is a well established concept already). Where it gets tricky is if we want to allow layers to be updated outside the main rendering loop. Picture a WebVR performance HUD that runs in an extension or as a drop in utility library.

// main.js
var sceneLayer = new VRCanvasLayer(canvas1);

function renderLoop() {
  drawScene();
  session.commit().then(renderLoop); // Hey, has the hud updated yet?
  updatePhysics();
}

// webvr-perf-hud.js
var hudLayer = new VRCanvasLayer(canvas2);

function renderHudLoop() {
  drawHud();
  // Whoops! Can't call session.commit() here.
  // Needs main.js to be aware of this file, which limits injectability.
  // Could monkeypatch session.commit() to do this drawing at the last second, I suppose?
}

A per layer commit allows this scenario to work out really nicely, though!

// main.js
var sceneLayer = new VRCanvasLayer(canvas1);

function renderLoop() {
  drawScene();
  sceneLayer.commit().then(renderLoop);
  updatePhysics();
}

// webvr-perf-hud.js
var hudLayer = new VRCanvasLayer(canvas2);

function renderHudLoop() {
  drawHud();
  hudLayer.commit().then(renderHudLoop);
}

Both commits would get the same promise back, so they'd stay in sync nicely even though they have effectively no knowledge of each other.

BUT! There's a less obvious issue here which is that we still want everything updated within a single frame to be presented to the HMD at the same time. This means I can't just shove a new image into the VR compositor immediately on calling commit on a layer because I may not know how many other layers are also going to commit that frame. So now we need to rely on an implicit sync point, which is probably the point at which the previous frame's commit promise has finished resolving. That's not terrible, WebGL currently works the same way with rAF, but it also means that the above code snippet is waiting for the physics simulation to complete before it sends the layer images off to the VR compositor. Not ideal!

You can work around that without too much trouble by doing something along the lines of

function renderLoop() {
  drawScene();
  sceneLayer.commit().then(renderLoop);
  setTimeout(updatePhysics, 0); // Force this to happen outside the commit promise resolve.
}

But that is a bit awkward.

Now is this style of loosely coupled, independently updating layers something that we must support? No. I personally like it, and I feel that the developer community will be able to use it to good effect, but I'm also certain that the API can survive and thrive without it. We've also faced questions in the past, however, about how to handle things like multiple calls to submitFrame that will come up again with a session-level commit. The per-layer model handles those types of edge cases more elegantly, IMO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment