Basis Universal and IBL with KTX2 images

[lexaknyazev]

Here's a set of work-in-progress extensions that will provide high-quality IBL with GPU-compressed textures.

Glossary

KTX2 - An image container file format optimized for GPU uploads. The spec is available here.

Basis Universal - Texture compression technology developed by Binomial. It enables JPEG-sized textures that can be transcoded on-the-fly to natively supported compressed GPU formats.

ASTC HDR - Block-compressed floating point GPU image format. Supported by many mobile devices and some desktop platforms.

BC6H - Block-compressed floating point GPU image format. Supported by many desktop and console platforms.

Proposed extensions

KHR_image_ktx2

Defined on glTF image, provides JSON form of the KTX header for better workflow on the web.
Does not restrict allowed KTX2 features / formats.

KHR_texture_basisu (requires KHR_image_ktx2)

Defined on glTF texture, similar to WebP extension with optional fallback.
This allows usage of Basis Universal images for IBL (normal quality) and material textures (albedo, roughness, etc).

EXT_texture_bc6h and EXT_texture_astc_hdr (require KHR_image_ktx2)

Defined on glTF texture. The same texture object should have KHR_texture_basisu extension object for better compatibility.

KHR_lights_image_based (requires KHR_image_ktx2 and KHR_texture_basisu)

Defines image-based lights on scene.

[UX3D-nopper]

As you already created a pull request, can you please add the diffuseEnvironmentMap as discussed yesterday? Or do you want me to make the changes in your branch?
You can gather the information from my previous mails.

[lexaknyazev]

Sure. Should I simultaneously remove SH? I don't think we have decided on a final form:

• diffuseMap + SH;
• diffuseMap + optionally SH;
• SH + optionally diffuseMap.

[UX3D-nopper]

Hmm, thought we agreed on supporting both. But it is an either or. So do not kick out SH. But either SH or the diffuseMap has to be present.

[lexaknyazev]

Thanks, I'll update the PR.

[lexaknyazev]

@UX3D-nopper
Done.

A few actionable items that should be addressed before marking this as ready for trial implementation.

• Does rotation affect both maps? Yes.
• Should we disallow KHR_texture_transform on IBL textures? Yes
• Does the IBL coordinate system match glTF default (Y-up)? Yes
• Provide complete sampling math for SH, as the ratified extension cannot refer to external sources.
• Verify that linear transfer function doesn't introduce banding with normal quality diffuse maps.
• Decide whether an asset can provide both SH and a diffuse map so that engines can decide at runtime.

[MiiBond]

I'm assuming that we're suggesting CTTF here so that the texture can remain compressed in memory. However, block-based compression schemes really don't work well with RGBE because they don't take into account that the alpha is being used as an exponent. The result is usually a lot of ugly banding between exponent boundaries. I don't remember how RGBD behaves when block-based compression is used but I expect it would be better. We should do some tests.

[UX3D-nopper]

RGBD, RGBE or RGBM should be decoded on CPU before uploaded to GPU. So, in general, not the perfect format but defined as mostly used.

[lexaknyazev]

I'm assuming that we're suggesting CTTF here so that the texture can remain compressed in memory.

Regardless of an effective GPU format, CTTF should give us better transmission bandwidth than typical web formats.

RGBD, RGBE or RGBM should be decoded on CPU before uploaded to GPU.

CPU decoding of RGBD (RGBE) from CTTF is expected to be supported.

[UX3D-nopper]

If we want to achieve consistency, we should only allow either SH or difusse maps in a scene.
E.g. if in some case SH is sufficient, one can use SH. If Sh is not sufficient, the a diffuse map is provided.
But this decision is made by the content creator and the rendering engine should just use what provided.

[UX3D-nopper]

Agree regarding "Provide complete sampling math for SH".
@bghgary Can your team please gather this information?

[bghgary]

From @sebavan

So our implementation is here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Maths/sphericalPolynomial.ts

Containing all the various references in the header comment.

It is called for each pixel of the cubemap here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Misc/HighDynamicRange/cubemapToSphericalPolynomial.ts

And applied by the shader here:
https://github.com/BabylonJS/Babylon.js/blob/master/src/Shaders/ShadersInclude/harmonicsFunctions.fx

The comments in the code include all the math and relevant references.

[bghgary]

Would it be helpful to call out that the specular radiance cubemap has prefiltered data in its mips and that the diffuse irradiance map has only one mip?

[bghgary]

In the vendor extension, we had a list of images and the expected image size. This allows the possibly to drop a few mip levels at the end of the mip chain. Is this possible with this extension?

[lexaknyazev]

KTX2 allows random access to mip levels, so loaders can do whatever they see fit.

[bghgary]

Sure, but it needs to be specified here how to interpret the mip levels or we will get inconsistent behavior?

[UX3D-nopper]

For simplicity, I would recommend to just use the full mip map chain. What were Babylon.JS reasons, to have this feature?

[bghgary]

See #1377 (comment)

[bghgary]

What is brightnessOffset? Is this supposed to be a color? If so, why do we want this?

[UX3D-nopper]

brightnessOffset is for shifting the color range more into the darker or brighter area. E.g. required if the HDR maps are RGBM encoded and to support negative values.

[bghgary]

Below, this section says:

finalSampledColor = sampledColor * brightnessFactor + brightnessOffset;

brightnessOffset is currently a number. Is it supposed to be an offset per component?

[UX3D-nopper]

finalSampledColor = sampledColor * vec3(brightnessFactor) + vec3(brightnessOffset);
Maybe more obvious like this.

[zeux]

In KHR_image_ktx2, is the JSON image header optional? In terms of ease of integration into existing pipelines and maintaining the simplicity of the baseline format it seems like it's better to follow something like MSFT_texture_dds, that just allows specification of .dds path without requiring producers of glTF files to parse KTX files and emit duplicate header information.

[lexaknyazev]

@zeux
The JSON form of the binary KTX2 header is mandatory. Just like buffer views and accessors, it gives the information required to allocate GPU resources before actual texture data comes in.

[donmccurdy]

@lexaknyazev to my understanding a .basis file contains much of the same header information that the KTX2 header will offer, so with the JSON form we're specifying that in triplicate. I'm still not sure I understand the need for the extra header. In this and other threads, we've offered reasons such as:

• allocate GPU resources before actual texture data comes in.
• check compatibility before loading a texture
• reduce the amount of network requests in case of partial data fetching

At least for the use case of Basis textures referenced by materials, none of these reasons are self-explanatory to me... when might a Basis file be incompatible in a way that can only be known from the header? What implementations preallocate GPU resources this way?

I'm a bit worried this is extra data that implementations will not actually use, and it has the side effect that users cannot easily swap out textures without modifying the JSON. It would be helpful to have a clear and robust example of why the extra data is necessary.

[donmccurdy]

Maybe another way of putting this – why are we using a KTX2 wrapper if we're going to replicate everything that wrapper provides?

[lexaknyazev]

.basis file contains much of the same header information that the KTX2 header will offer, so with the JSON form we're specifying that in triplicate.

We're not going to use .basis file format wrapper, KTX2 will directly embed codebooks and slices. Compression tools will be able to directly output .ktx2 files with Basis payloads.

Basis textures referenced by materials

Mip-levels cannot be generated online for compressed textures, so they have to be provided by an application. Without JSON form, apps would have to make an additional request (or two) to download texture file header to be able to progressively stream mip levels.

I'm a bit worried this is extra data that implementations will not actually use

JSON copy of the header makes glTF clients simpler since they would be able to skip parsing the binary header and pull texture data directly.

users cannot easily swap out textures without modifying the JSON

Putting extra burden on exporters is aligned with other glTF choices. Geometry cannot be easily swapped out either.

[donmccurdy]

We're not going to use .basis file format wrapper, KTX2 will directly embed codebooks and slices. Compression tools will be able to directly output .ktx2 files with Basis payloads.

Ok, that's good thanks!

JSON copy of the header makes glTF clients simpler ...

This seems like a very minor simplification, and in my opinion should be considered a non-argument compared to more important pros (fewer requests) and cons (confusing user workflows, redundant data).

Without JSON form, apps would have to make an additional request (or two) to download texture file header to be able to progressively stream mip levels.

We discussed a number of concerns about this on our call today, namely – the data is redundant, and common workflows today involve unpacking a .glb, editing the textures, and repacking the .glb. Users will certainly be surprised to learn they can't swap out .ktx2 files referenced by an unpacked .gltf like they would with .png or .jpeg; that is a guaranteed cost we'd be paying with this design.

We discussed an alternative – that the header information could be optional, or could be provided by a separate extension, such that it can be omitted until/unless tools exist that are actually progressively streaming mip levels.

I am still open to that alternative, but an even better outcome would be this: can we get at least one common engine to implement support with streaming before ratification? The scenario I strongly want to avoid here is one where we've complicated the format for hypothetical streaming support, but later find that no one has actually implemented the perceived benefit, or that other features are actually necessary for that benefit to be practical. If we knew, for certain, that streaming textures will be implemented without requiring additional extensions, I'd be much more open to accepting the known cost of end-user confusion when externally referenced .ktx2 files are not interchangeable.

[lexaknyazev]

The size of the KTX2 header index fields is:

80 + 24 * levelCount

For 16384*16384 textures (currently the second most common max texture size), the levelCount is less than or equal to 15. This makes the index part of the header less than 500 bytes. It's almost certain that the first partial resource fetch would include more than 500 bytes, so an engine would have all the information needed to access mip-levels independently.

This already unlocks streaming at the cost of one extra request compared to having all indices in JSON.

[atteneder]

I have a working prototype implementing parts of this PR. Still very raw, but a first step:

Unity package: https://github.com/atteneder/glTFast/tree/ktx

I tested it on 4 models DGG3D kindly provided to me.

Now I'd like to encode more glTFs to do more testing/stabilizing, but I could not get glTF-Compressonator to build yet. Asking for support did not yield in a response so far.
Is there a way to encode glTFs with KTX/BasisUniversal textures? maybe @UX3D-nopper can tell?

Thanks!

[UX3D-nopper]

When we took over the Compressonator project, it did not build for us on Linux and/or Mac either.
At the moment, it only builds and runs on Windows.

If you need the IBL images, please use this tool:
https://github.com/KhronosGroup/glTF-IBL-Sampler

[zeux]

Does anybody know if any web glTF viewer implements KHR_image_ktx2 (possibly in some development branch)? I'd like to add support to this to gltfpack but not sure how to test the resulting files.

[atteneder]

Does anybody know if any web glTF viewer implements KHR_image_ktx2 (possibly in some development branch)? I'd like to add support to this to gltfpack but not sure how to test the resulting files.

I could provide a Unity project + WebGL build with an URL input field. Current alpha state can load glTF (non-binary; separate files) with KTX textures (no levels, no mipmaps yet). Should take me a day or two to.

I'm much interested in test files and ktx support in gltfpack on the other hand. Would be awesome if you share your results!

[zeux]

Is this necessary when the extension is marked as required? (I understand that it's possible to replace references to KTX2 images in "texture" object with references through extensions, but I'm not sure I understand why this is better than simply using an image/ktx2 MIME type on the image objects that are referenced directly)

[donmccurdy]

My understanding is that KHR_image_ktx2 may be dropped entirely, and that the other extensions in this PR would simply independently allow KTX2 images. @lexaknyazev @bghgary is that a correct understanding of #1705 (comment)? This was confusing me as well.

[zeux]

(quick clarification, the comment got misplaced and refers to line 65 - implementation note - of the spec, not line 63)

[bghgary]

KHR_image_ktx2 may be dropped entirely

This was my suggestion after attempting to implement it in Babylon. Unless we need some other metadata that goes with the extension, it will likely end up with a no-op implementation.

[zeux]

(just clarifying) I am assuming that in absence of "source" property on the original texture JSON element, the loaders that support the extension must use the image specified through the extension metadata?

[zeux]

I've implemented support for these extensions in gltfpack: zeux/meshoptimizer#81. To build this yourself, make sure you have basisu in your path (you can build & install it from the official repository: https://github.com/BinomialLLC/basis_universal/). To compress textures with Basis & KTX2, command line flag -tc is required; gltfpack will automatically run basisu when this flag is specified.

Here's an example - it's the Corset model from glTF-Sample-Models, converted to .glb with embedded KTX2 images:
corset.glb.zip

Note that the code is somewhat untested; please let me know if there are any issues with this! It's hard to test this without renderer support...

[MarkCallow]

@zeux, what software can I use to view corset.glb? Firefox (on macOS) doesn't seem to understand .glb files. I tried Windows' 3D Viewer but it says "couldn't load model, please try again later."

[zeux]

@MarkCallow This file requires support for KHR_texture_basisu, KHR_image_ktx2 and KHR_mesh_quantization so I would not expect existing renderers to open it, but it could be used to validate support for the extensions from this PR in some work-in-progress renderer...

[mlimper]

I tested it on 4 models DGG3D kindly provided to me.

Now I'd like to encode more glTFs to do more testing/stabilizing, but I could not get glTF-Compressonator to build yet. Asking for support did not yield in a response so far.
Is there a way to encode glTFs with KTX/BasisUniversal textures? maybe @UX3D-nopper can tell?

hehe happy that the models were helpful

we have an experimental KTX2 encoder branch, it was tested with a special (also experimental) BabylonJS branch for our joint demos for the BOF at SIGGRAPH

the main reason it is called "experimental" is because the spec is still in draft state

would it help if our devs run it on a couple of samples?

[atteneder]

Here's an example - it's the Corset model from glTF-Sample-Models, converted to .glb with embedded KTX2 images:
corset.glb.zip

Note that the code is somewhat untested; please let me know if there are any issues with this! It's hard to test this without renderer support...

I tried to load this glb in my prototype but I get an KTX_FILE_UNEXPECTED_EOF (using KTX-Software). I also extracted the first ktx file/glTF buffer view. Running ktxinfo on it works, but loading it (in the glloadertests executable) fails as well.

[MarkCallow]

@atteneder please provide more details and, if it looks like a bug in KTX-Software, open an issue over there. How can I extract the ktx2 file(s) from the .glb? I tried opening with 7-zip but it says it can't open it. I also tried The Unarchiver on macOS. Be aware that ktxinfo just looks at the file header. I'd suggest trying ktx2check but I know it will fail as I haven't yet updated it for the latest KTX2 spec. draft and I know @zeux's files are compliant to that draft. However loading should still work, providing the file doesn't have other issues.

[zeux]

I'll investigate this, since this implementation is written from scratch this is probably a bug on my end.

Also attaching a .zip file where texture files aren't embedded into .glb for easier analysis.
corset-separate.zip

[zeux]

Ah, the mip levels are written from smallest to largest but the mip level index must be written from largest to smallest :-/ ktxcheck doesn't catch this, but loading the entire file does.

After fixing this, the only other change that seems to be needed to get libktx to load these files is to align the last mip by 8 bytes. I'm not sure why this is necessary - the spec seems to explicitly call out that last mip doesn't need to be padded? - but libktx texture2.c has the following code that necessitates this:

    // Calculate size of the image data.
    This->dataSize = 0;
    for (ktx_uint32_t i = 0; i < This->numLevels; i++) {
        This->dataSize += _KTX_PAD8(private->_levelIndex[i].byteLength);
    }

[zeux]

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

[atteneder]

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

glb works smooth now! found some remaining bugs on my side though.
thanks for the super quick answer!

[MarkCallow]

ktxcheck doesn't catch this

Please file an issue at KTX-Software.

[ShantanuSriv]

Here's a new version; for easier testing, it contains the .glb (with embedded textures) as well as .gltf (with separate .bin + texture files)

corset-fixed.zip

how do you generate glb file? I am able to create glTF + bin + basis texture files but unable to pack them into one glb file using gltf-pipeline. Which tool are you using to output glb file?

[zeux]

@ShantanuSriv These files are produced by gltfpack (https://github.com/zeux/meshoptimizer#gltfpack); -tb produces files with Basis container and -tc produces files with KTX2 container format which is what I posted above. -te embeds textures into .bin or .glb. Note that for KTX2 files you should use master version since the last point release contained a bug with mipmap layout.

[donmccurdy]

Per discussion earlier, we are planning to:

1. Discontinue work on the KHR_image_ktx2 extension: it isn't necessary. Other extensions Like KHR_texture_basisu can independently allow use of KTX2 files.
2. Close this PR for now.
3. Open a new PR to resolve remaining questions on KHR_texture_basisu.
4. Revisit the IBL extensions at some point in the future.

@lexaknyazev or @bghgary would you be able to open a new issue for (3)?

[lexaknyazev]

The story continues in #1751.