Which timestamp should be used for frame callbacks?

[Manishearth]

Some XR APIs, when providing a frame will provide the predicted display timestamp, i.e. the timestamp at the expected time of display. I believe they also provide the predicted frame for that time. For example, openxr does this

When we're working with predicted frames, should the timestamp of the callback be in the future?

[Manishearth]

It seems like we want the timestamp to be the time the frame occurred (like other non-xr rAF callbacks), not any predicted frame display time.

That said, this still leaves open the question of whether we should be providing predicted frames at all.

Currently the spec talks of the "time represented by the XRFrame", which doesn't use XRFrame/time (this is an oversight). XRFrame/time is set to "now" here. We might need to make XRFrame/time be defined as "the time represented by the XRFrame" and continue to pass "now" to the animation frame callbacks, with some non normative text saying "hey, the XRFrame may represent things in the future".

If we do this it may be worth adding something that lets you access the XRFrame/time directly.

[Manishearth]

/agenda we should pick something here

[toji]

Hm. We talked about this in the past and I'm pretty certain that we decided that the timestamp passed to the callback would basically be the same timestamp that would have been passed to a standard rAF that would have happened at the same time. (I don't know the technical way to express this off the top of my head.) This was primarily so that content that switched between window.rAF and session.rAF for their presentation wouldn't experience weird jumps back in time or anything like that. This also tracks with the intent to have all rAF-like methods follow a predictable use pattern.

At the time we also decided that any XR specific timing values would be better communicated through the frame. (XRFrame.timing.predictedDisplayTime, for example). I'll do some digging and see if I can figure out where we made and recorded that decision.

[blairmacintyre]

I don't remember talking about this, but the only timestamp that makes sense, IMNSHO, is the one that corresponds as closely as possible to the time of the frame on the underlying XR platform. Using some purposely wrong timestamp just to satisfy such an odd requirement seems wrong to me; how often do you expect this switch to happen? How often do you expect it to happen without other major changes to the display (going in/out of AR/VR mode)?

[toji]

Apologies for the slow reply, but the issue where we previously discussed this was #347. The proposal I made above was specifically made in theis comment #347 (comment) and it got widespread approval at the time. (Including from one @blairmacintyre who replied "I like this, thanks @toji. I'm especially in favor of having the rAF parameter behave as you describe." 😉)

From that thread it sounds like the while the timestamp behavior was updated in the explainer (you can read the text in the "Main Render Loop" section) we didn't pull the same description over into the spec text. Sorry for that!

Now that thread is two years old, and so opinions may have shifted regarding the conclusions we came to since then, but for the record I'm still in favor of that general plan and would like to see the spec updated with a more rigorous definition for the timestamp that matches the explainer's behavior and then a renewed look at what XR-specific timing values we can usefully expose to developers on the frame.

[Manishearth]

Seems good to me, think we should help-wanted this with a link to that explainer section?

[toji]

Agreed. The primary task is to translate the behavior from the explainer into spec text. Sepcifically:

The timestamp provided is acquired using identical logic to the processing of window.requestAnimationFrame() callbacks. This means that the timestamp is a DOMHighResTimeStamp set to the current time when the frame's callbacks begin processing. Multiple callbacks in a single frame will receive the same timestamp, even though time has elapsed during the processing of previous callbacks.

Then as a follow-up we should have an issue to track what other useful timing values we can expose as attributes of an XRFrame. That conversation was somewhat re-started in #963, but it may be better for ongoing conversation if we created a specific issue for it.

[blairmacintyre]

@toji I think what I was agreeing with was essentially two things:

• using the rAF time as a common value across all rAF's that might be cued that is the same type of data as as normal rAF (not some radically different scale and kind of number, such as micro- vs milli- seconds). Some common value is useful!
• having some other way of getting the frame-level timing info from the platform. I liked the suggestion you had.

Did anything happen similar to the XRFrameTiming struct you proposed back then? (Sorry, there's no XRFrameTiming in the spec and I don't recall something similar).

The frame time != when the rAFs start running, and any non-trivial math based on trying to interpolate or infer things across frames for AR/VR sensing is going to be hopeless if we don't have accurate values for when those poses and other frame data occured.

If the only time value we have is the time the rAF's started executing, that's a problem. Even at the native level, for example, tools like "ARToolkit" (or other application-level AR sensing apps) could never do proper sensor fusion on most platforms because they had no accurate estimate of when the video frames (and, thus, the poses extracted from them) were acquired. Most of these libraries used the application level time that they received the video frame, which was wildly inaccurate when compared to the time stamps from the inertial sensors. The reason ARKit/ARCore/etc work is because all of this stuff is being done inside the OS with accurate timing, and that time value is being passed up into the app with the data; if we throw out that timing info, there are a host of things native apps will be able to do that web apps never will.