Add 'layout' hint for quad, cylinder, equirect stereo layers

[Artyom17]

Displaying pre-rendered stereo images or videos will require specification of stereo layout, which is currently mentioned for video quad layer. However, for stereo photos, stereo 180/360 panoramas it is also required WHEN texture2D is used. For texture arrays it will be ignored.

[cabanier]

@Artyom17 and I talked offline about this.
His concern was that for certain sizes, the UA won't be able to support a left to right texture because it exceeds the maximum texture size.

However, the current API allows the UA to switch to a different layout or even another texture if the requested size is too big for a left-to-right texture. At best, if the author knows if it's left-to-right or top-to-bottom, they could blit for 2 eyes at the same time.
For now, we don't need the hint but we might revisit this later.

[Artyom17]

For now, we don't need the hint

I disagree. If I use pre-rendered content, such as stereo 360 photos, or regular 3D photos, I want dev to be able to use the texture unmodified. I don't understand why we are going to limit to side-by-side stereo only and not to include top-to-bottom as well. I'd strongly recommend to add the hint to all the layers (well, other than projection layer).

[cabanier]

The only thing this would solve is a blit during rendering (so 2 instead of 1).
It also forces the UA to support all layout types. Currently they can optimize and only offer what is efficient.

If we offer a hint, do we also extend the API to say what type of layout was created by the UA? Can the hint be ignored?

[Artyom17]

What I think:

• Each blit takes at least 0.5 ms on Quest, those are not cheap at all on mobile (of course depends on size and stereo 360 content is usually of big resolution). Especially when you update it every frame (more applicable to video)
• We already have those layouts for video, so supporting layouts for other layers is not a problem at all.
• Hint can't be ignored, otherwise it will defeat its purpose. Projection layer can be different.
• layout can be returned by layer as readonly attribute
  I even think we don't need a 'boolean stereo', we need an enum XRStereoLayout instead, with members like:
  "mono" - mono
  "stereo" - default stereo, UA is selecting which layout is used; also used for texture arrays;
  "stereo-left-to-right"
  "stereo-top-bottom"

[Artyom17]

I am currently prototyping stereo support for Equirect (and same will be applicable to Cylinder and Quad, while Cube might be different). What I also found out:

• XRWebGLLayerFactory.getSubImage should work for stereo layers and viewport must be a combination of both left and right eye; this is required to do proper blit.
• bias and scale must be provided for BOTH eyes (currently Equirect has only one bias and scale which is wrong; it happened because it was copied from OpenXR, but OpenXR has no stereo layers). I'll open a separate issue for this.

[cabanier]

I even think we don't need a 'boolean stereo', we need an enum XRStereoLayout instead, with members like:
"mono" - mono
"stereo" - default stereo, UA is selecting which layout is used; also used for texture arrays;
"stereo-left-to-right"
"stereo-top-bottom"

Do you feel strongly that this should be in the first version of the layers spec or can we postpone it until later?

[cabanier]

I am currently prototyping stereo support for Equirect (and same will be applicable to Cylinder and Quad, while Cube might be different). What I also found out:

• XRWebGLLayerFactory.getSubImage should work for stereo layers and viewport must be a combination of both left and right eye; this is required to do proper blit.
• bias and scale must be provided for BOTH eyes (currently Equirect has only one bias and scale which is wrong; it happened because it was copied from OpenXR, but OpenXR has no stereo layers). I'll open a separate issue for this.

yes, please file separate issues

[Artyom17]

I even think we don't need a 'boolean stereo', we need an enum XRStereoLayout instead, with members like:
"mono" - mono
"stereo" - default stereo, UA is selecting which layout is used; also used for texture arrays;
"stereo-left-to-right"
"stereo-top-bottom"

Do you feel strongly that this should be in the first version of the layers spec or can we postpone it until later?

I feel strongly that this is a basic functionality that should be in the first version.

[cabanier]

I feel strongly that this is a basic functionality that should be in the first version.

I added support for this in the layers readme in my repo.
Please check out https://github.com/cabanier/layers/pull/1/files and let me know what you think.

[Artyom17]

BTW, one more argument to have stereo layout: it will simplify not only usage of the pre-rendered texture, but it also will save a dev time for defining proper texture transforms (bias / scale) (see #48).

[cabanier]

For pre-rendered textures, you'd only draw once so there's no real performance benefit to have the side-by-side support...

[cabanier]

/agenda do we need 'layout' enum for quad, cylinder, equirect stereo layers?