Allow session to specify refresh rate

[cabanier]

To elaborate on #963 , did we ever discuss if the author can specify a refresh rate?
The Oculus Quest currently has a non-standard setting during session creation for low, mid and high refresh rate which is not ideal.

Could we make it so the author can state their preferred refresh rate and then the UA can choose to honor that.

For instance, if the author specifies 50 but the UA can only go as low as 60, 60 would be picked.

I'm unsure how we would go about integrating this into WebXR. Should we allow changing the refresh at runtime, or at session creation time?

[toji]

To recap some of the conversation that we had at the end of todays IW call:

Modern hardware can pick and choose from multiple framerates to run at, so "High frame rate" is not particularly meaningful anymore. (On a Quest 2, for example, it could mean 72Hz, 90Hz, or 120Hz) It's hard for content to know what rate is optimal given the range of hardware a page may be run on. It would be ideal if authors could indicate a framerate switch mid-session. Oculus hardware can do this, and it's not expensive, Not sure if other hardware can as well. A critical case is for smoothness of video playback, in that developers want to choose a framerate that is an even multiple of their media's frame rate. Could specify desired rate as "I want a multiple of X" if we're concerned about exposing actual framerates. But Jeff Gilbert at least stated that he'd be okay exposing a list of supported rates once a session is started, given the user consent requirements to get to that point. Multiple people agreed with that sentiment. Then we ran out of time.

[cabanier]

Thanks for the writeup!
So, we will likely need 3 things:

• a method to get the list of supported rates. (Do we need to support devices with dynamic framerates? ie Quest is theoretically able to go from 60 to 120 but we only expose 60, 72 and 90)
• a method to set the desired framerate. The UA is allowed to ignore it, round it or change it on its own (like @thetuvix mentioned)
• a read only attribute that returns the current framerate

These should likely be on an XRSession and only work for immersive ones.

[thetuvix]

We should definitely have some way to know what the duration of the current frame is - that's a simple oversight we should correct no matter what we do here.

I could see us adding explicit framerate enumeration and setting - however, I am concerned about compat in the logic apps write to make these selections. Note that for similar mode negotiations for camera sensors, there is a more at-a-distance request the app makes where it prioritizes what factors matter to the app - that feels more similar to the "prefer multiple of X (e.g. 24)" approach.

I am curious to hear what use cases folks have in mind beyond aligning to a multiple of the video framerate, which makes sense to me. If this is primarily about performance tuning, do we expect apps to smoothly move up and down the chain of framerates when they find themselves missing frames? Do we have enough performance timing APIs that apps can access in WebGL to know if they are well over or under their allotted frame times?

Basically, I'd love to see an example outside of the video-matching example of how someone would use such an API in practice. I suspect in practice we would see:

• Apps locking to a multiple of the video framerate during playback. (good!)
• Apps randomly setting a framerate without live feedback because it seemed to give good performance on the device they tested on. (bad!)

If there are many other such good scenarios, we could decide to accept sites hardcoding device-specific framerates as the cost of doing business - however, if video playback is the only real-world scenario we're lighting up here, we may get a better result here with a more targeted API.

[toji]

For the purposes of media frame rate matching I generally prefer a "prefer multiple of X" approach I think, but the immediate problem that comes to mind is developers saying "I prefer a multiple of 120" in order to provoke higher framerates, and then check the reported or observed framerate to see if it worked. If we allow that then we might as well simply let them pick from a list of available rates.

[cabanier]

Looking at how this is used on the Oculus platform, afaik we do the dynamic scaling to match media framerate.
Most apps release with a predetermined frame rate because that is what they optimized for. For the web, I expect that frameworks will do dynamic optimization since they have to target different hardware configurations. We talked to @RaananW from Babylon.js and he said they would definitely used it if it were available.

Apps randomly setting a framerate without live feedback because it seemed to give good performance on the device they tested on. (bad!)

I agree that that's a risk but we're there today anyway where people just optimize for the most popular device :-(
At least this way, we will get acceptable performance across devices if the framework happens to solve this for the developer.

[cabanier]

If we allow that then we might as well simply let them pick from a list of available rates.

I agree with this. So, let's provide a method to get the list of available refresh rates and then a setter.
I'm unsure if all devices support dynamic switching. If not, I'm unsure how we can expose this before the session is created.

[Maksims]

I'm unsure if all devices support dynamic switching. If not, I'm unsure how we can expose this before the session is created.

If that is the case, it is a fingerprinting concern.

Dynamic (during session) switching definitely would be great, as there are many cases where it could be usefull.

[cabanier]

If that is the case, it is a fingerprinting concern.

Dynamic (during session) switching definitely would be great, as there are many cases where it could be usefull.

Yes, that would be the problem. If these devices exist, we would need another property that can be set at creation time. Maybe that could be the "framerate multiply" for media.

[cabanier]

As a first pass, let's add a frameRate property to the session.
We can define that if the auhor sets it, the UA is free to ignore it or round it up/down to a supported frequency.

In a second pass, we could add support to report what frequencies are supported.

[thetuvix]

What is the primary key scenario we're aiming to enable here? Is it aligning framerate to match video? If so, setting a framerate directly with no enumeration seems like it will result in the wrong behavior:

• App wants to play 24Hz video, so it locks to 72Hz, since that's what their device uses for 24Hz video.
• UA knows that device supports only 60/120Hz or 60/96Hz, and so it chooses 60Hz, since that's the closest match. This causes the video to stutter, since the UA didn't know that the app was setting the framerate for video quality reasons rather than performance reasons.

Introducing a framerate setter without enumeration seems worse than doing nothing before we figure out framerate enumeration, since we will have lots of content start to hardcode framerates for divergent purposes without any means to do the right thing.

To the fingerprinting concern, perhaps there are two separate features here?

• Before session: Request of a framerate multiplier, which sets the initial framerate for hardware supporting dynamic framerate
• During session: Enumeration of framerates and dynamic setting of framerate

We could start with the dynamic version for now if it solves more scenarios folks are actually chasing down and see if there is a need on some hardware for the before-session version.

[cabanier]

What is the primary key scenario we're aiming to enable here? Is it aligning framerate to match video?

Aligning video frame rate is certainly important but the more immediate problem is managing performance.

Some users have very simple scenes and want to be able to show higher frame rates while others have complex scenes and would rather lower (or dynamically lower) the frame rate to avoid falling behind which will cause the compositor to duplicate frames.

• Before session: Request of a framerate multiplier, which sets the initial framerate for hardware supporting dynamic framerate

I'm unsure if we should do provide support for this. Authors can query/set the frame rate after the session is created.

• During session: Enumeration of framerates and dynamic setting of framerate

Do you have a proposal? I guess it could be a simple list of values...

Also, I believe the editors have mentioned that they want to go to CR so don't want changes to the spec anymore. I'm fine to put this in a level 2 version.

[cabanier]

proposal:

partial interface XRSession {
  attribute long? frameRate;
  readonly attribute Uint16Array? supportedFrameRates;
}

frameRate -> set/return the current frame rate
supportedFrameRates -> return a list of all supported rates. I'm assuming that will never need to be larger than 65535 :-)

[Maksims]

session.frameRate - might be interpreted as "actual frame rate", which is not the case here.
Perhaps some alternative such as: session.targetFrameRate could be better?

[cabanier]

session.frameRate - might be interpreted as "actual frame rate", which is not the case here.
Perhaps some alternative such as: session.targetFrameRate could be better?

Do you mean that it would be lower in case the system can't keep up?
I'm unsure if developers would be confused by this but I'm open to change it.

[cabanier]

Also, framerate reflects the refresh rate of the compositor. That one will always make frame rate; even if the experience can't keep up.

[thetuvix]

One note here is that not all systems will be able to synchronously switch from one frame duration to another. Even if the system can switch instantly, it's likely too late for the frame that is current when you might set a frameRate attribute.

The model I'd imagined here is more like XRView.requestViewportScale() - the app makes the request at some point, and then if that request is honored, it takes effect at some future frame, with the app reading out the effective frame rate as a duration on a given XRFrame:

partial interface XRFrame {
  attribute double duration; // displayInterval?
}

partial interface XRSession {
  undefined requestFrameRate(unsigned short? frameRate);
  readonly attribute Uint16Array? supportedFrameRates;
}

Expressing the effective frame interval as a duration or displayInterval in seconds could avoid the notion that this is feeding back to you some performance measurement of your app's frame rate.

Note that the UA is already allowed to adjust the frame interval as needed even without any new app control here. If the app wants to keep its animations correct in the face of those changes, it should already be subtracting the time parameter it gets in each frame's rAF callback from the previous frame's time - the one thing you'd lose without an explicit duration is how long the current frame being rendered will then display. (the next interval vs. the previous interval) To acknowledge the possibility of UA frame rate control, we could retroactively define the default requestFrameRate value of null to mean that the UA is in control of frame rate to achieve the best experience.

[cabanier]

One note here is that not all systems will be able to synchronously switch from one frame duration to another. Even if the system can switch instantly, it's likely too late for the frame that is current when you might set a frameRate attribute.

I didn't write the prose yet but I was envisioning that it'd be similar to fixedFoveation where the author sets it and it is up to the UA to decide when it takes effect.
Do you think it matters that it might take some frames to take effect while the new value is reflected in the attribute?

...

partial interface XRFrame {
  attribute double duration; // displayInterval?
}
partial interface XRSession {
  undefined requestFrameRate(unsigned short? frameRate);
  readonly attribute Uint16Array? supportedFrameRates;
}

Expressing the effective frame interval as a duration or displayInterval in seconds could avoid the notion that this is feeding back to you some performance measurement of your app's frame rate.

Wouldn't there be a danger that authors would assume that they are allowed to take that long to execute the frame?
Also this seems more appropriate for the profiling API we have discussed in the past.

Don't we also need an attribute that returns the current frame rate or do you think that's not needed?

Note that the UA is already allowed to adjust the frame interval as needed even without any new app control here. If the app wants to keep its animations correct in the face of those changes, it should already be subtracting the time parameter it gets in each frame's rAF callback from the previous frame's time - the one thing you'd lose without an explicit duration is how long the current frame being rendered will then display. (the next interval vs. the previous interval) To acknowledge the possibility of UA frame rate control, we could retroactively define the default requestFrameRate value of null to mean that the UA is in control of frame rate to achieve the best experience.

Are there any UA's that dynamically adjust the framerate?
If so, yes we could define a value of 0 that the UA is allowed to dynamically adjust the rate.

[Maksims]

If there are 2 modes of frame rate: specific provided and UA driven, this might lead to a bad experience.
Currently, most browsers implement variable frame rates, and usually target some stable "points" like 120 > 60 > 30, as it is better to keep stable but lower framerate, than less stable jumping from 30 to 60 let's say.
Browsers and platforms have been doing this for a very long, and they have their own ways to properly measure performance and adapt.

Ability to specify "preferred target framerate" - would be instructing platform that let's say 120, but it is up to a platform to decide if it can manage such framerate or not.
If Spec enforces UA to strictly match target framerate, this leads to a couple of problems:

1. Developer might assume that the platform will have to comply, but will not be able to due to unoptimized application performance. Basically, there is no way to always match the preferred framerate.
2. If the platform will be enforced to comply by Specs, then it will lead to variable framerate depending on application performance, without "stopping points", which will lead to reduced battery life and worse visual experience (in VR especially important).

I would assume that platforms currently decide by their means on what target frame rate is best for the browser. This spec can provide a way for developers to specify "preferred target" from a list of available ones. But the way that frame rate is managed - should be still dependant on UA and their platform specifics.

Also, setting targetFrameRate - can be instant, as frame rate changes can depend on underlying systems and the performance of the application. This means that we don't mix targetFrameRate with actual measurements of current frame rates.

EDIT:
Also, there is a preferred pattern, which is used by Depth Sensing Spec. This works well too: preferredFrameRate.

EDIT2:
Layers - how does it affect Layers, do they support separate rendering / refresh rate, such as video layers?

[cabanier]

I'm fine calling it targetFrameRate. It doesn't imply that it's the current one. For instance, during blur events we drop the frame rate and we want to continue to allow that.

Currently, most browsers implement variable frame rates, and usually target some stable "points" like 120 > 60 > 30, as it is better to keep stable but lower framerate, than less stable jumping from 30 to 60 let's say.

I'm unaware of any VR browsers that are doing this. Maybe you are thinking of the regular 2D browser? If so, that is a different use case and likely shouldn't be under the browser's control.

Layers - how does it affect Layers, do they support separate rendering / refresh rate, such as video layers?

Regular WebXR is a single layer that is drawn at the device's refresh rate. Projection layers continue to have this requirement.
The other layer types are allowed to redraw at their own rate, or if they are static, only once.

[thetuvix]

targetFrameRate sounds good to me, as it doesn't imply that you've reached the target yet. It's fine then to immediately get back the target you set even if it's not achieved - perhaps that is enough if apps already diff the rAF frame time to target animations.

Currently, most browsers implement variable frame rates, and usually target some stable "points" like 120 > 60 > 30, as it is better to keep stable but lower framerate, than less stable jumping from 30 to 60 let's say.

I'm unaware of any VR browsers that are doing this. Maybe you are thinking of the regular 2D browser? If so, that is a different use case and likely shouldn't be under the browser's control.

VR platforms such as Windows Mixed Reality can have logic in the compositor itself to fall back to a stable lower frame rate if the current app is not hitting a steady frame cadence. This can happen in any app, including WebXR browser backends, and pages need to accept whatever rAF timing they receive each frame.

[thetuvix]

Layers - how does it affect Layers, do they support separate rendering / refresh rate, such as video layers?

Regular WebXR is a single layer that is drawn at the device's refresh rate. Projection layers continue to have this requirement.
The other layer types are allowed to redraw at their own rate, or if they are static, only once.

That's an interesting point - is this session rate updating the device's actual refresh/composition rate, or just the rate at which the app is asked to produce projection layer frames for all layers? I believe it's the former, as that is what enables smoother video if something like 72Hz is chosen. This would affect the prose we write here.

[cabanier]

I'm unaware of any VR browsers that are doing this. Maybe you are thinking of the regular 2D browser? If so, that is a different use case and likely shouldn't be under the browser's control.

VR platforms such as Windows Mixed Reality can have logic in the compositor itself to fall back to a stable lower frame rate if the current app is not hitting a steady frame cadence. This can happen in any app, including WebXR browser backends, and pages need to accept whatever rAF timing they receive each frame.

Should this happen when the author sets 0 for the frame rate, or is it always allowed to happen?

For Oculus, we'll always apply the requested frame rate but we'll drop to a lower one when the session is blurred.
In our case, the list of supported rates won't contain '0'

[cabanier]

Layers - how does it affect Layers, do they support separate rendering / refresh rate, such as video layers?

Regular WebXR is a single layer that is drawn at the device's refresh rate. Projection layers continue to have this requirement.
The other layer types are allowed to redraw at their own rate, or if they are static, only once.

That's an interesting point - is this session rate updating the device's actual refresh/composition rate, or just the rate at which the app is asked to produce projection layer frames for all layers? I believe it's the former,

It is the rate at which the author should produce frames. In most headsets that is the same as the device's but for example for Magic Leap, it is half of the actual device rate.

as that is what enables smoother video if something like 72Hz is chosen. This would affect the prose we write here.

I'm unsure if we need to call out this special case. For example animations or media won't be smooth if the author can request 72 fps but the device ends up reprojecting to 120fps.

[cabanier]

@thetuvix @Maksims I made a PR to introduce this concept to the spec.
Let me know what you think.

[cabanier]

/agenda discuss the PR to query and control the device's frame rate

[cabanier]

Setting the frame rate to an unsupported value currently doesn't throw or report an error; instead it silently ignores it.
Should it throw?

[cabanier]

Can we change it so the spec guarantees that the framerate is updated at the next Raf?
Right now the UA is free to apply it down the line and we got some feedback from developers that they want to know if/when it is applied.

[cabanier]

Thinking about it some more, maybe it's better to make the framerate read-only and reflect the current frame rate.
Setting the rate could then be done with a setFrameRate call that returns a promise. The targetFrameRate attribute would then update when the promise is fulfilled.

@toji @thetuvix thoughts?

[thetuvix]

Should this happen when the author sets 0 for the frame rate, or is it always allowed to happen?

Device platforms will have various reasons that the frame rate needs to change that are out of the app's control. For example, on HoloLens 2, the frame rate drops from 60 to 30 when the user asks the system to start a video recording. Apps cannot presume that they will always get the frame rate they requested, and so I wouldn't tie any caveat here to whether the target frame rate is still at the default.

Can we change it so the spec guarantees that the framerate is updated at the next Raf?
Right now the UA is free to apply it down the line and we got some feedback from developers that they want to know if/when it is applied.

Some devices may not be able to switch frame rates that quickly - that seems like a heavy requirement for every device.

The idea of a target frame rate seems reasonable on the XRSession. We could then have a separate attribute on a given XRFrame that tells you the effective duration/displayInterval for that frame:

partial interface XRFrame {
  attribute double duration; // displayInterval?
}

It feels weird to talk about the "frame rate" of a given frame, so this may be a better representation that is useful even for apps that don't change the target frame rate, to better understand any changes imposed by the underlying system for a given frame.

[cabanier]

Should this happen when the author sets 0 for the frame rate, or is it always allowed to happen?

Device platforms will have various reasons that the frame rate needs to change that are out of the app's control. For example, on HoloLens 2, the frame rate drops from 60 to 30 when the user asks the system to start a video recording. Apps cannot presume that they will always get the frame rate they requested, and so I wouldn't tie any caveat here to whether the target frame rate is still at the default.

Can we change it so the spec guarantees that the framerate is updated at the next Raf?
Right now the UA is free to apply it down the line and we got some feedback from developers that they want to know if/when it is applied.

Some devices may not be able to switch frame rates that quickly - that seems like a heavy requirement for every device.

Given this, maybe we need to change the proposal:

partial interface XRSession {
    readonly attribute long? targetFrameRate;
    undefined setTargetFrameRate(long targetFrameRate);
    readonly attribute Uint16Array? supportedFrameRates;

    attribute EventHandler ontargetframeratechange;
}

targetFrameRate would then always reflect the rate at which the experience should produce frames.
setTargetFrameRate would set the preferred rate. The UA should try to honor it but is allowed to change it at any time. 0 would leave it completely up to the UA to decide what to do.
supportedFrameRates would return the list of supported rates.
ontargetframeratechange would be a sessionEvent to signal that a rate change is now in effect.

This addresses the feedback I've gotten so far:

• clarifies what a 0 frame rate means
• produces an event so authors know when the frame rate actually updates
• gets rid of the writeable attribute (which @toji mentioned)
• allows out-of-band frame rate updates

A drawback is that it's more complicated

[cabanier]

The idea of a target frame rate seems reasonable on the XRSession. We could then have a separate attribute on a given XRFrame that tells you the effective duration/displayInterval for that frame:

partial interface XRFrame {
  attribute double duration; // displayInterval?
}

It feels weird to talk about the "frame rate" of a given frame, so this may be a better representation that is useful even for apps that don't change the target frame rate, to better understand any changes imposed by the underlying system for a given frame.

Can we do this as a separate issue? Measuring how much headroom there is so authors can tune their experiences should be its own thread. (I thought we already had an issue but I can't seem to find it...)