KHR_mesh_quantization feedback

[juj]

1. The proposed spec KHR_mesh_quantization adds snorm8, unorm8, snorm16, unorm16 and float types as possible encodings for vertex attributes. These are all nice since they are types that GPUs can automatically decode to 0...1 and -1...1 range in shaders.

There are other interesting vertex attribute types that GPUs support built-in:

• OpenGL ES 2.0+ and WebGL 1.0+ support fixed16_16 (GL_FIXED, no D3D support afaik),
• Direct3D11+, OpenGL ES 3.0+ and WebGL 2.0 support float16 (GL_HALF_FLOAT, DXGI_FORMAT_R16_FLOAT, DXGI_FORMAT_R16G16_FLOAT, DXGI_FORMAT_R16G16B16A16_FLOAT),
• OpenGL ES 3.0+ and WebGL 2.0 support unorm16 (GL_HALF_FLOAT + normalized==true, DXGI_FORMAT_R16_UNORM, DXGI_FORMAT_R16G16_UNORM, DXGI_FORMAT_R16G16B16A16_UNORM),
• OpenGL ES 3.0+ and WebGL 2.0 support sint10.10.10.2 (GL_INT_2_10_10_10_REV, no support in D3D11?),
• Direct3D11+, OpenGL ES 3.0+ and WebGL 2.0 support snorm10.10.10.2 (GL_INT_2_10_10_10_REV + normalized==true, DXGI_FORMAT_R10G10B10A2_SNORM),
• Direct3D11+, OpenGL ES 3.0+ and WebGL 2.0 support uint10.10.10.2 (GL_UNSIGNED_INT_2_10_10_10_REV, DXGI_FORMAT_R10G10B10A2_UINT),
• Direct3D11+, OpenGL ES 3.0+ and WebGL 2.0 support unorm10.10.10.2 (GL_UNSIGNED_INT_2_10_10_10_REV + normalized==true, DXGI_FORMAT_R10G10B10A2_UNORM)
• Direct3D11+, OpenGL 4.4+, OpenGL 3.0+ with ARB_vertex_type_10f_11f_11f_rev extension support float11.11.10 format (GL_UNSIGNED_INT_10F_11F_11F_REV, DXGI_FORMAT_R11G11B10_FLOAT)

GPUs can readily decode these where available. Should some of these vertex attribute formats also be made available for this extension?

2. There are other interesting ways to encode vertex attributes to save space. Two such ways that are most common are to encode normals by omitting one component and reconstructing it in the shader, and to encode coordinate frames by omitting one component of a quaternion, and reconstructing it in a shader. For better numerical stability, sometimes the index that is largest in absolute magnitude is dropped, and the index of the omitted component is included in the attribute stream.

A very good treatise to this scheme for quaternions is presented by Stephen McAuley at GDC 2015, 6:24 into the video.

Example code in
https://github.com/r-lyeh-archived/quant/blob/bad7fe2958a1304952d1677c7599c460da7c1c18/quant.hpp#L287

Should the spec include support for some of these kind of packings?

[zeux]

For 1, the extension is specified explicitly to only include the WebGL1 baseline and only including formats that are universally supported by native APIs as well. So it's a lowest common denominator if you will.

We discussed some formats you mention, for example 10_10_10_2 is really interesting for normal encoding. However it's unclear how to implement this for WebGL1 decoders.

For 2, see #1702 (specifically filter specification). The intention there is to save transmission size but decode the data at runtime; there are other ways to specify this to keep the data compressed in GPU/CPU memory but that is more complex for decoders to handle, requiring extra shader permutations to deal with various encoding combinations, e.g. normal/tangent.

There are additional complications with tangent encoding. For example, it's tempting to standardize 2-component octahedral encoding for normals (note - it's crucial to not just reconstruct .z, two components + index are a reasonable idea but it's not as easy to decode on the GPU and it's not as efficient in terms of precision/bit), however tangents need an extra bit, and glTF has a 4-byte alignment requirement for attributes - so how to precisely encode normals & tangents in separate attributes without wasting too much space isn't clear.

So, my perspective is that the extension is in the right place - it's easy for loaders to implement, and doesn't require special GPU support. I don't think we should conflate different representations of various attributes with that, and I don't see a good way to allow extra types such as 10_10_10_2 without requiring all existing JS loaders to implement a software decode fallback for WebGL1 (which is still the dominant API on the Web AFAIK?).

[juj]

Thanks, both rationales make sense! Not sure if glTF extensions process collects Questions&Resolves like GL/GLES extensions do, if it does, might be good material to cover there(?)