Proof-of-concept KTX1 GPU texture format support

[B-Interactive]

I want to premise this not so much as a feature request, but as a discussion about how I might approach a potential PR presenting this feature. In large part this is also a Starling discussion, as these features impact it too.

• Khronos Texture website

Adding basic support for KTX1 wasn't all that difficult, and I have a working proof-of-concept parsing the KTX1 container with DXT1 and DXT5 encoded textures. KTX1 as a container type seems like an ideal first step, as it naturally supports all formats ATF currently supports, with the addition of newer formats like BC7 and ASTC.

My goal is three fold:

• Firstly and broadly speaking, efforts to help OpenFL / Starling remain up-to-date with modern standards.
• Implement support for a standard open file format, such as KTX, as support for ATF was dropped more than two years ago by tools such as TexturePacker. The baseline goal here is to at least support the same texture compression options as ATF. In effect, introduce KTX1 parsing, but then use existing GPU texture pipelines.
• Then, open the door to adding support for modern GPU texture compressions, such as BC7, ASTC, KTX2 (Basis-U), etc.

Here's the issue

I've hit a road-block with how to best approach a PR, and I wonder it may be better to provide a proof-of-concept before a PR is even considered (let me know please).

I believe there are two ways of approaching this. The issue stems from OpenFL and Starling codewise assuming only a single GPU texture format exists: ATF. It's not currently designed for the possibility of another (or even many more). I'm mindful Starling is designed to be lean, and I hope to keep it that way.

With that in mind, the options are:

1. Leave existing ATF classes and methods untouched and implement additional classes and handlers for each new format (beginning with KTX1).
2. Refactor to a unified GPU texture pipeline that is no longer ATF focused and has scope to evolve as newer standards become available.
3. Do both. Leave ATF groundwork in place, but all new GPU texture formats and codecs use a unified pipeline.

Option 2 and to a degree 3, would avoid a lot of duplication which would have introduced maintenance overhead. It also avoids turning classes like starling.textures.Texture into the kitchen sink, having to parse and conditionally identify and manage multiple formats.

The changes in an option 2 unified approach would be fairly wide-sweeping. I want to check first if the team has an appetite for such a thing, whether it be KTX1 support at all, or refactoring to a unified GPU texture pipeline.

~ Bink

[joshtynjala]

I would not be against adding support for new ATF-like texture formats.

The main thing to emphasize, though, is that the public APIs for working with ATF and uncompressed textures should not undergo any breaking changes while adding support for new formats. In particular, existing function signatures should not be changed (parameters can support new values as long as the type doesn't change, though, including integer, float, string, enum values), and properties should have the same type in the end. Basically, someone should still be able to translate old AS3 code to Haxe where the only difference is syntax, and their code should work.

For the Flash/AIR targets, new formats obviously can't be supported, and that should be documented. However, we want to continue supporting these targets in OpenFL, so code that was written for OpenFL before adding the new formats that could compile to Flash/AIR should still work without changes. Basically, the same point as above. We want to maintain backwards compatibility with existing APIs from Flash/AIR, so no function signature or changing properties to use different types than before.

As for how it should work internally, I can't really make suggestions from a place of deep experience with GPU programming. However, I have some thoughts. I'm going to focus on OpenFL specifically, and how to add support in Starling can wait until later, I think.

I feel like we could continue to use Texture.uploadCompressedTextureFromByteArray(). It is named in such a way that doesn't strictly tie it to the ATF format specifically, and I'm assuming that all new texture formats can be provided as ByteArray. As long as those formats don't require additional parameters, I think it makes sense to continue to use this method.

Then there's the question of Context3DTextureFormat. It similarly provides the generic names COMPRESSED and COMPRESSED_ALPHA. We could potentially use those for additional formats besides ATF. However, in that case, we'd need to detect the format from the contents of the ByteArray. For instance, I see that an ATF ByteArray literally starts with the string "ATF". If other formats have a similar signature to detect, then that seems pretty straightforward.

However, if additional formats that you'd like to support seem like they'll have bigger differences from ATF, or they are harder to detect quickly, so wouldn't work well with the existing Context3DTextureFormat.COMPRESSED and Context3DTextureFormat.COMPRESSED_ALPHA, then we can consider adding more values to the Context3DTextureFormat enum.

[B-Interactive]

Thank you @joshtynjala, that's helpful direction.

I can pare it back to focus on OpenFL implementation first, at least initially. That seems like a sensible approach.

I'd taken it further into Starling in my proof-of-concept, largely as a simple means of testing without wading into uncharted waters with AGAL or GLSL. I've been mimicking the ATF implementation in many respects, whilst leaving ATF handling alone.

For example, in my proof of concept, I'd created openfl.display3D._internal.KTX1Reader, which is a KTX1 counterpart to the existing openfl.display3D._internal.ATFReader. As you mentioned, an ATF's bytecode begins with "ATF", and in similar fashion a KTX1 starts with a 12 byte signature which contains within it "KTX11". This is being used to identify and differentiate between ATF and KTX1 within openfl.display3D.textures.Texture.__uploadCompressedTextureFromByteArray() method.

Where I've deviated a bit, is with how the KTX1 container identifies quite specifically the contained format (via bytecode), with more precision than ATF appears to (ATFReader.readHeader()). There are many formats that are supported too. This is the list I have been referencing:
https://github.khronos.org/KTX-Software/pyktx/_modules/pyktx/gl_internalformat.html

So where openfl.display3D._internal.ATFFormat has:

package openfl.display3D._internal;

#if !flash
#if (haxe_ver >= 4.0) enum #else @:enum #end abstract ATFFormat(Int)
{
	public var RGB888 = 0;
	public var RGBA8888 = 1;
	public var COMPRESSED = 2; // JPEG-XR+LZMA & Block compression
	public var RAW_COMPRESSED = 3; // Block compression
	public var COMPRESSED_ALPHA = 4; // JPEG-XR+LZMA & Block compression with Alpha
	public var RAW_COMPRESSED_ALPHA = 5; // Block compression with Alpha
}
#end

The more generalised openfl.display3D._internal.GLCompressedTextureFormat I'm working on contains the Khronos bytecode signatures, as well as helper methods to identify those with/without alpha, blocksize, colour format, etc.

package openfl.display3D._internal;

/**
 * Centralized GL compressed texture format constants.
 * See: https://github.khronos.org/KTX-Software/pyktx/_modules/pyktx/gl_internalformat.html *
 */
#if (haxe_ver >= 4.0) enum #else @:enum #end abstract GLCompressedTextureFormat(Int) {
    // S3TC/DXT
    public var COMPRESSED_RGB_S3TC_DXT1_EXT = 0x83F0; // DXT1
    public var COMPRESSED_RGBA_S3TC_DXT1_EXT = 0x83F1; // DXT1 w/ 1-bit alpha
    public var COMPRESSED_RGBA_S3TC_DXT3_EXT = 0x83F2; // DXT3
    public var COMPRESSED_RGBA_S3TC_DXT5_EXT = 0x83F3; // DXT5

    // S3TC sRGB
    public var COMPRESSED_SRGB_S3TC_DXT1_EXT = 0x8C4C;
    public var COMPRESSED_SRGB_ALPHA_S3TC_DXT1_EXT = 0x8C4D;
    public var COMPRESSED_SRGB_ALPHA_S3TC_DXT3_EXT = 0x8C4E;
    public var COMPRESSED_SRGB_ALPHA_S3TC_DXT5_EXT = 0x8C4F;

    // ETC1/2
    public var ETC1_RGB8_OES = 0x8D64;
    public var COMPRESSED_RGB8_ETC2 = 0x9274;
    public var COMPRESSED_RGBA8_ETC2_EAC = 0x9278;

    // PVRTC
    public var COMPRESSED_RGB_PVRTC_4BPPV1_IMG = 0x8C00;
    public var COMPRESSED_RGB_PVRTC_2BPPV1_IMG = 0x8C01;
    public var COMPRESSED_RGBA_PVRTC_4BPPV1_IMG = 0x8C02;
    public var COMPRESSED_RGBA_PVRTC_2BPPV1_IMG = 0x8C03;

    // ASTC
    public var COMPRESSED_RGBA_ASTC_4x4_KHR = 0x93B0;
    public var COMPRESSED_RGBA_ASTC_5x4_KHR = 0x93B1;
    public var COMPRESSED_RGBA_ASTC_5x5_KHR = 0x93B2;
    public var COMPRESSED_RGBA_ASTC_6x5_KHR = 0x93B3;
    public var COMPRESSED_RGBA_ASTC_6x6_KHR = 0x93B4;

    // BC7 (BPTC)
    public var COMPRESSED_RGBA_BPTC_UNORM_ARB = 0x8C7F;

    /**
     * Human-readable string for GL compressed format.
     */
    public static function toString(fmt:GLCompressedTextureFormat):String {
        return switch (fmt) {
            case COMPRESSED_RGB_S3TC_DXT1_EXT: "DXT1";
            case COMPRESSED_RGBA_S3TC_DXT1_EXT: "DXT1A";
            case COMPRESSED_RGBA_S3TC_DXT3_EXT: "DXT3";
            case COMPRESSED_RGBA_S3TC_DXT5_EXT: "DXT5";
            case COMPRESSED_SRGB_S3TC_DXT1_EXT: "DXT1_sRGB";
            case COMPRESSED_SRGB_ALPHA_S3TC_DXT1_EXT: "DXT1A_sRGB";
            case COMPRESSED_SRGB_ALPHA_S3TC_DXT3_EXT: "DXT3_sRGB";
            case COMPRESSED_SRGB_ALPHA_S3TC_DXT5_EXT: "DXT5_sRGB";
            case ETC1_RGB8_OES: "ETC1";
            case COMPRESSED_RGB8_ETC2: "ETC2_RGB";
            case COMPRESSED_RGBA8_ETC2_EAC: "ETC2_RGBA";
            case COMPRESSED_RGB_PVRTC_4BPPV1_IMG: "PVRTC4_RGB";
            case COMPRESSED_RGBA_PVRTC_4BPPV1_IMG: "PVRTC4_RGBA";
            case COMPRESSED_RGBA_BPTC_UNORM_ARB: "BC7";
            // ... add ASTC, etc as needed
            default: "UNKNOWN";
        }
    }

    /**
     * Returns block size in bytes for given format.
     * For PVRTC4, this is not per-block, but total image size = width*height/2.
     */
    public static function blockBytes(fmt:GLCompressedTextureFormat):Int {
        return switch (fmt) {
            case COMPRESSED_RGB_S3TC_DXT1_EXT,
                 COMPRESSED_RGBA_S3TC_DXT1_EXT,
                 COMPRESSED_SRGB_S3TC_DXT1_EXT,
                 ETC1_RGB8_OES,
                 COMPRESSED_RGB8_ETC2:
                8;
            case COMPRESSED_RGBA_S3TC_DXT3_EXT,
                 COMPRESSED_RGBA_S3TC_DXT5_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT3_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT5_EXT,
                 COMPRESSED_RGBA8_ETC2_EAC:
                16;
            // PVRTC4: block size is not used, use image size calculation instead
            case COMPRESSED_RGB_PVRTC_4BPPV1_IMG,
                 COMPRESSED_RGBA_PVRTC_4BPPV1_IMG:
                0; // handled specially
            default: 0;
        }
    }

    /**
     * Returns true if format is sRGB.
     */
    public static function isSRGB(fmt:GLCompressedTextureFormat):Bool {
        return switch (fmt) {
            case COMPRESSED_SRGB_S3TC_DXT1_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT1_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT3_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT5_EXT:
                true;
            default: false;
        }
    }

    /**
     * Returns true if format supports alpha.
     */
    public static function hasAlpha(fmt:GLCompressedTextureFormat):Bool {
        return switch (fmt) {
            case COMPRESSED_RGBA_S3TC_DXT1_EXT,
                 COMPRESSED_RGBA_S3TC_DXT3_EXT,
                 COMPRESSED_RGBA_S3TC_DXT5_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT1_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT3_EXT,
                 COMPRESSED_SRGB_ALPHA_S3TC_DXT5_EXT,
                 COMPRESSED_RGBA8_ETC2_EAC,
                 COMPRESSED_RGBA_PVRTC_4BPPV1_IMG:
                true;
            default: false;
        }
    }
}

[joshtynjala]

It sounds like you have a very good handle on the format, and I like how you've described your code so far. It sounds like it will fit well into OpenFL.

[B-Interactive]

I have a proof of concept here that contains the classes I'd be including in a PR:
https://github.com/B-Interactive/KTX1_proof-of-concept

The last thing I'm working on is mipmaps. I have them implemented, but I'm still testing as I'm getting some unexpected results.

[B-Interactive]

I've now created a pull request with this:

Add KTX v1 container support for existing OpenFL supported GPU texture compression formats #2826