Add premultiplied alpha setting to KTX format

[alecazam]

I'd like for KTX to formalize this reality into the spec, so that it doesn't have to be stored in some ignored name-value pair.

Simple proposal:
PremultipliedAlpha: true/false

Alternate proposal:
ChannelDescription: "AR, AB, AG, A" (premultiplied) vs. "R, G, B, A" (unpremultiplied)
ChannelDescription: "AL, A" (premultiplied) vs. "L, A" (unpremultiplied)

Premultiplied alpha is the norm for 2D and 3D renderers, and yet only a few image formats (notably EXR) have any indicator for premultiplied alpha support. EXR uses the AR, AG, AB convention vs. R, G, B to indicate a premultiplied channel. PNG only supports unpremultiplied, JPG doesn't support any, etc. DXT2/4 used to be a premultiplied version of DXT3/5, but disappeared with the rename to BC1/2/3.

To get proper sampling on the GPU and especially with sRGB premultiplied formats, you can't just multiply the alpha in the shader after sampling. Bilinear and trilinear and anisotropic sampling need the texture format premultiplied before it reaches the texture cache. sRGB is undone if present, and then all samples combine in premultiplied linear space with adds/averaging in the texture unit.

In an encoder, pixels are premultiplied, cluster fit the point cloud of colors, and finally snapped back into 565/888 BC/ASTC endpoints in the file format. Uncompressed data with an alpha channel (RGBA8, 16f, 32f) would have the rgb channels premultiplied. Premultiplication makes storing constant data (f.e. in rb in BC3nm) less simple, since the alpha channel would affect all channels, but premultiplied data usually applies to albedo channels which use all channels for color data.

Finally sRGBA8 data would do: sRGB -> linear, rgb * a, linear -> sRGB. This is best done on the GPU. Mipmaps must do a similar conversion to average in linear space, and convert back to sRGB.

BC1 with alpha cutout, could use premultiplied form, but with alpha = 0 the color would go to black. Finally, textures with an alpha channel that is all 1, can set the PremultipliedAlpha flag to true without any math to the rgb pixels.

Note that I'm not proposing KTX format do these conversions, but libKTX could help do the right thing and texture compression tools would do the right thing here.

[MarkCallow]

There is already a bit in the flags field of the data format descriptor (DFD) in KTX2. There is nothing that can be done for KTX1 except using metadata.

Two values for the bit are defined in khr_df.h: KHR_DF_FLAG_ALPHA_STRAIGHT (=0) and KHR_DF_FLAG_ALPHA_PREMULTIPLIED (=1). In libktx the bit can be accessed in a ktxTexture2 object called ktex by

bool isPremultiplied = KHR_DFDVAL(ktex->dfd, FLAGS) & KHR_DF_FLAG_ALPHA_PREMULTIPLIED

In libktx, we need to make sure that this bit is preserved along with the transfer function and primaries when the DFD is rewritten during supercompression to Basis. We also need to provide a way to create pre-multiplied textures. currently toktx accepts PNG and Netpbm formats. Neither appears to support premultiplied. So what to do? I will leave this issue open to track resolution of these 2 points.

[alecazam]

I like that header spec a lot, it's very explicit. My only concern is that this flag not be restricted to DXT2/4 type formats. Basically any compressed or uncompressed endpoints with alpha should be valid to have premultiplied alpha applied, and that bit set if I'm understanding it correctly. The spec only stated usage on DXT2/4.

And yes, have libktx hand premultiplication of alpha would be a big help, since if work is done on the gpu, then this is a cheap multiply operation in addition to the required sRGB conversions that are best done on GPU.

[MarkCallow]

My only concern is that this flag not be restricted to DXT2/4 type formats.

It can be set for any descriptor. It only makes sense for the RGBDSA and compressed texture color models and only when an alpha channel is present. See Table 90 in Chapter 11, Example format descriptors.

And yes, have libktx hand premultiplication of alpha would be a big help

I don't understand what you are trying to say here.

[alecazam]

I don't understand what you are trying to say here.

I just meant that doing srgb -> linear, linear * a, then linear * a -> srgb could be done entirely on the gpu in libktx. Otherwise, it's simple done with a 3x 2d table lookups on ra, ga, ba. Just that that ordering should be honored for correctness and to avoid clamping. Bits will be lost on low alpha values being stored in 8-bit components regardless.

[MarkCallow]

Ahh! So s/hand/handle/. Are you suggesting an option to toktx to premultiply the incoming images?

The parts of libktx involved in creating .ktx2 files do not use the GPU. They can run anywhere. As creation doesn't seem particularly time critical I'm loathe to add the complexity of checking for and using a GPU and dealing with the resulting dependencies. A premultiplication option can be implemented on the CPU of course.

The only parts that do use the GPU are the OpenGL and Vulkan loaders. By that point the data should already be premultiplied.

[MarkCallow]

PIng! Are you asking for a pre-multiplication option to toktx? Should I add an isPremultiplied() function to ktxTexture2 or is accessing the DFD sufficient?

[alecazam]

No, the "bool isPremultiplied = ..." example that you provided is probably sufficient.

I do think that premultiplication of the values should happen in the texture encoder, but it's complicated with the low-bit depth of BC1-BC3. With ASTC, the endpoints are 8-bit, but sometimes split into a base + delta. And finally, many of the existing compressors just take the srgb colors and then fit the endpoints to those. That doesn't seem right, since that's not a linear color space.

So ideally for srgb:

1. convert colors from srgb to 10-bit or 8-bit linear. Conversion to 565 isn't enough bits.
2. do rgb * alpha
3. fit endpoints to the premultiplied linear rgb values
4. remap the rgb endpoints back to srgb

The gpu does the reverse of step 4 before storing the colors in the texture cache. That way bi/tri-linear filtering are correct in linear space. On a simpler note, EAC/BC4/5 can also be stored at higher precision in the texture cache, since they're only 1-2 channels.

For rgb, it's easy since it's just step 2 and 3 only.

[MarkCallow]

The only encoder within libktx is the BasisU encoder which is incorporated as a git subrepo. You should probably open an issue at https://github.com/BinomialLLC/basis_universal requesting this. I don't think there is anything to do in the rest of libktx so I'm going to close this. Note that there is now a ktxTexture2_GetPremultipliedAlpha function to find out if the bit is set or not.