Allow content to know an upper bound on the number of views?

[asajeffrey]

At the moment, content doesn't get told how many views it needs to render until after a session is started. This means that frameworks which allocate per-view resources (like three.js cameras) have to do that after the session has started. If we added an attribute to XRSystem that gave an upper bound on the number of views, then frameworks could allocate resources ahead of time.

[asajeffrey]

Annoyingly, this is information that could be used for device fingerprinting, so we may not want to reveal it.

[Manishearth]

/agenda To see what folks think about this

[klausw]

Technically, the spec currently also allows changes to the number of views during the session. If a framework allocates resources at session start based on the first frame's views, it would need to be able to cope with additional views being added later.

If there were a single XRSystem attribute for the maximum number of views, before knowing the session mode or chosen configuration, this would need to be a conservative estimate. For example, Chrome on Android currently uses two views for VR, and a single view for AR. I don't know how helpful the attribute would be if it results in the framework setting up two cameras for an AR session that only needs one of them.

[asajeffrey]

I thought the list of views was immutable, it's just whether they're enabled or not that can change? But you're right that new devices can connect or disconnect at any time, so we probably can't do anything sensible, other than ask frameworks not to assume anything about the views until they start a session.

[klausw]

https://immersive-web.github.io/webxr/#xrview-interface says: No guarantee is made about the number of views any XR device uses or their order, nor is the number of views required to be constant for the duration of an XRSession.

The list of views returned by getViewerPose is a readonly FrozenArray, but it's only immutable for the duration of the frame. The next frame could potentially return a different number of views or reorder the views.

In practice, I suspect that many current frameworks wouldn't cope well with drastic changes to the views during a session since they are likely to make assumptions based on current implementations that don't do so.

If I remember right, there is interest in adding a view for a third-person camera, i.e. a monitor view for others to see what's going on in a headset session. This would be useful as a feature that can be enabled or disabled on the fly during a session, resulting in a change to the view count.

However, I'm not entirely convinced if it's a good idea to handle such a third-person camera as an XRView since it seems to complicate other parts of WebXR, i.e. for the proposed viewport scaling (issue #1091) and blend mode (immersive-web/webxr-ar-module#53).

[Manishearth]

@klausw

Technically, the spec currently also allows changes to the number of views during the session.

It actually doesn't! The spec already knows the maximum number of views that a session will support and uses it to allocate the framebuffer.

https://immersive-web.github.io/webxr/#xrview-interface says: No guarantee is made about the number of views any XR device uses or their order, nor is the number of views required to be constant for the duration of an XRSession.

The spec makes no guarantee about the views array being exposed. Internally, it has a maximum cap.

https://immersive-web.github.io/webxr/#xrsession-list-of-views

The list of views is immutable during the XRSession and MUST contain any views that may be surfaced during the session, including secondary views that may not initially be active.

[klausw]

@Manishearth , thank you for clarifying. However, I think there's a bit of a disconnect here between specification internals and application-observable behavior. If I understand it right, the list of views and the view active flag are internal attributes used in the specification, and there's no current way for the application to know about the potential maximum number of views.

Unless I'm misunderstanding it, the application still needs to act as if the number of views it receives via getViewerPose can change at any point during the session?

Relatedly, even if exposing the maximum number of views on XRSystem isn't feasible due to fingerprinting or other reasons, would it be useful to expose the maximum number of views as a XRSession attribute, considering that it already exists as a spec-internal value?

[klausw]

By the way, as far as application compatibility is concerned, the spec now has a secondary-views opt-in feature, and apps opting in to this are expected to be able to deal with view counts changing every frame. It's implied (but as far as I can tell not required) that UAs should avoid doing so if applications haven't opted in.

[Manishearth]

Unless I'm misunderstanding it, the application still needs to act as if the number of views it receives via getViewerPose can change at any point during the session?

Yes. This is basically a hint to the content to preallocate any resources it may need (shaders, cameras, etc)

would it be useful to expose the maximum number of views as a XRSession attribute

Yeah I would be against exposing it on the XRSystem even without fingerprinting concerns, it's very much a per-session thing depending on how the session is initialized.

I meant to propose this on this issue but forgot.

[thetuvix]

By the way, as far as application compatibility is concerned, the spec now has a secondary-views opt-in feature, and apps opting in to this are expected to be able to deal with view counts changing every frame. It's implied (but as far as I can tell not required) that UAs should avoid doing so if applications haven't opted in.

From the call, I believe it's our common working understanding that UAs would not add and remove secondary views from an app if it does not opt into "secondary-views", and that features like optional foveated left/right views would count as secondary views for this purpose. That would make the number of primary views effectively constant.

Does anybody see a gotcha if we just tighten up the core to say that a UA MUST NOT add or remove displays after the session starts? I'll fork a new issue to track that.

[Manishearth]

@avadacatavra had some fingerprinting concerns on the call. Broadly speaking, the three kinds of secondary views we know of so far are:

• first person observer views. A session will support them based on the browser and hardware (typically, "AR hardware in an AR session"), which is not really a new fingerprinting bit.
• quad views: not a new fingerprinting bit because you will immediately know this number on the first frame
• third person observer views

TPO views are interesting, and @kearwood had some good points about them. Namely, the existence of a TPO view is a new fingerprinting bit based on the implementation, because it's possible for the existence of a TPO view to be a matter of "the user has some streaming software installed", and that is a new fingerprinting bit

Does anybody see a gotcha if we just tighten up the core to say that a UA MUST NOT add or remove displays after the session starts? I'll fork a new issue to track that.

For primary views only, yes 😄

[Manishearth]

The rough conclusion in the call was that without knowing the full breadth of possible secondary views we will have fingerprinting concerns here, and it's not clear that this feature is a major benefit: most of these resources can be allocated lazily, and at worse you have a little frame jitter when that happens.

The suggestion was to close the issue as "not now", but to maybe revisit this if/when there's stronger motivation and a better understanding of the breadth of secondary views.

[expenses]

Utgh, I'm now running into this issue when trying to render to multiple with the OVR_multiview2 WebGL 2 extension and it's proving to be a pain point. I want to have this information up-front because I want to select which pipelines to create based on whether I'm doing multiview rendering or not.

The other solutions I'm left with are:

• Build both sets of pipelines (multiview and single view) or potentially 3 sets when 4-view VR headsets become available.
• Only build pipelines after rendering has begun (messy from a code point of view and leads to a first-frame delay)
• Just guess if we're rendering to 2 or 1 views based on whether an immersive VR, AR or inline session was started (this failed for me today when I tried out a holodeck).

or (I thought about this after writing this):

• Request a single animation frame after the session has started, and pass the number of views out from that.