[< 1.12.2] Resource packs with custom banner or shield textures stop displaying properly

[AlexTMjugador]

Describe the bug
Resource packs that target Minecraft versions up to 1.12.2 and modify banner textures, located at assets/minecraft/textures/entity/banner and assets/minecraft/textures/entity/banner_base.png, stop displaying banner patterns if the banner pattern textures were replaced under most circumstances. The same applies for shields, whose textures are located at assets/minecraft/textures/entity/shield, assets/minecraft/textures/entity/shield_base.png and assets/minecraft/textures/entity/shield_base_nopattern.png. In addition, the base textures look speckled and low-quality, even though the textures that PackSquash stores in the ZIP file look much better when viewed in other PNG manipulation programs and using the Java ImageIO class, which these Minecraft versions use to read texture data.

To reproduce

1. Optimize a resource pack that replaces banner or shield textures. Make sure that the replacement textures pixels are stored in 8-bit RGBA format (most PNG images are); textures with other color types will be broken too.
2. Open the pack in Minecraft 1.12.2 or older.
3. Observe that banners and shields that use modified textures stop displaying properly.

Expected behavior
Patterns in banner and shields should work as they did before PackSquash optimized the pack. Base textures should look as expected given the current quantization settings, not so worse.

Screenshot
The three banners shown have different base colors, but they all look like they had no pattern applied to them. Upon closer inspection, it is also easy to see that the wooden part of the banners looks speckled, uglier than usual.

Additional context
The affected Minecraft versions instantiate and load a net.minecraft.client.renderer.texture.LayeredColorMaskTexture (MCP 9.4.0 names for 1.12) to draw banners and shields. Shields share the same relevant code path as banners, and this texture type is only used to render these items in vanilla Minecraft. A LayeredColorMaskTexture is composed by a base texture, which is drawn on a texture canvas of the same pixel format as the base texture, and lists of layer textures and dye colors, each layer texture being drawn on top of the texture canvas in list order, after applying the corresponding dye tint color to it.

When a LayeredColorMaskTexture is loaded via its loadTexture method to be bound for OpenGL draw calls, the previously described texture canvas is generated, and its contents used for the actual banner texture. So, if the texture canvas generation code at loadTexture does something quirky, the banner will look wrong.

In fact, LayeredColorMaskTexture#loadTexture completely skips applying a layer texture if it's not in 8-bit RGBA format, even though the code does not need the layer texture to be in this format at all, because it later only ever uses BufferedImage#getRGB calls, which implicitly convert pixels to RGBA as needed. This quirky check explains why patterns are not applied at all.

if (
    layerMask.getWidth() == textureCanvas.getWidth() &&
    layerMask.getHeight() == textureCanvas.getHeight() &&
    layerMask.getType() == BufferedImage.TYPE_4BYTE_ABGR
) {
    for (int l = 0; l < layerMask.getHeight(); ++l)
    {
        for (int i1 = 0; i1 < layerMask.getWidth(); ++i1)
        {
            int j1 = layerMask.getRGB(i1, l);

            // ...
        }
    }

    textureCanvas.getGraphics().drawImage(layerMask, 0, 0, null);
}

The speckled look of the affected textures may be explained by Java's Graphics#drawImage method not preserving color accuracy when drawing on the canvas texture, or the GPU/render engine/OpenGL not dealing properly with the texture format of the base texture, which usually is encoded with a palette. These hypothesis were not confirmed.

Workarounds

• Set the color_quantization_target option to none for the affected texture, and make sure that they have more than 256 colors. This will force PackSquash to not store them in the problematic paletted color format.
• Migrate to Minecraft 1.13 or newer. I understand that performance is worse in newer versions, but using these old, quirky versions that are no longer supported by Mojang, while missing out on new features and bugfixes, is not the proper way to fix things. The best way to fix any performance issues is to send Mojang well-researched bug reports about performance regressions they can act upon. I want to support older versions of Minecraft too, but there are only so much things I can work around, and this support may become untenable in the future.
• If zip_spec_conformance_level is set to pedantic, you can modify the ZIP file PackSquash generates to replace the affected textures with their original versions.
• If it's not really that important for the banner and shield textures to be replaced, you may just remove them from your resource pack, using the vanilla textures instead.

Fix
v0.3.1 will add a quirk workaround that skips color type reduction for the affected textures. This will come at a substantial compression efficiency cost for such textures.

[AlexTMjugador]

This issue was fixed in the latest unstable builds, which can be downloaded from the actions page, or, if you're using the GitHub action, you can also change the packsquash_version parameter to latest. If everything goes as planned, the next release will ship this fix.

[AlexTMjugador]

Please note that the workaround implemented in the latest builds does not fix Minecraft issues with HD banner and shield textures not showing properly, which was reported in MC-63485.

Working around that would mean automatically resizing the textures to match vanilla's, which is a too lossy optimization for PackSquash and may mess with the vision and intent of the pack author. Maybe it'd be nice to check that the affected textures have the same size in the future, and error out if they don't, as they won't work anyway.