Palettes with rawmode RGBA;15 don't work

[CookiePLMonster]

What did you do?

In an attempt to convert a PlayStation TIM image format to a bitmap, I turned to Pillow to handle image processing for me. This proved to be reasonably straightforward, but when operating on 8-bit palettized TIM files, I seem to be unable to preserve transparency. As per THIS DOCUMENTATION palette entries are RGBA5551 entries, but attempting to use this format with Pillow results in an exception (see the sample code below for more detail).

Since palettes with format 'RGBA;15' (RGBA5551) are reported as supported, while both 'RGBA' (RGBA8888) and 'RGB;15' (RGB555) are accepted, I believe this might be a bug.

What did you expect to happen?

I expected image.putpalette(clut_data, 'RGBA;15') to succeed and produce a correct image with the alpha channel used.

What actually happened?

image.putpalette(clut_data, 'RGBA;15') throws unrecognized raw mode.

What are your OS, Python and Pillow versions?

• OS: Windows 10 21H2
• Python: 3.10.0
• Pillow: 9.0.1, grabbed from pip

For a relatively small reproduction, refer to the attached .zip with a .tim file to convert and run the following example with python code.py unpack title_gtmode.tim. Notice image.putpalette(clut_data, 'RGBA;15') throws unrecognized raw mode, while running it with mode RGB;15 works as expected and produces a correct image, albeit with transparency data discarded.

import sys
import struct
import os
from PIL import Image

if len(sys.argv) < 3:
    exit

mode = sys.argv[1].lower()
if mode == 'unpack':
    clut_data = None
    image_data = None
    image_width = 0
    image_height = 0

    with open(sys.argv[2], 'rb') as tim:
        tag, version = struct.unpack('BB2x', tim.read(4))
        if tag == 0x10:
            if version != 0:
                sys.exit(f'Unknown TIM file version {version}!')
            
            flags = struct.unpack('B3x', tim.read(4))[0]
            bpp = flags & 3
            clp = (flags & 8) != 0

            if clp:
                # Parse CLUT
                length, x, y, width, height = struct.unpack('IHHHH', tim.read(12))
                clut_data = tim.read(length - 12)
            
            # Parse image
            length, x, y, image_width, image_height = struct.unpack('IHHHH', tim.read(12))
            image_data = tim.read(length - 12)
        
    if image_data:
        image = Image.frombytes('P', (image_width * 2, image_height), image_data, 'raw', 'P')
        image.putpalette(clut_data, 'RGBA;15') # 'unrecognized raw mode' exception here

        name = os.path.splitext(sys.argv[2])[0]
        image.save(name + '.png')

title_gtmode.zip

[radarhere]

Hi. You're calling putpalette with a rawmode argument of "RGBA;15".

As noted in https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.putpalette

rawmode – The raw mode of the palette. Either “RGB”, “RGBA”, or a mode that can be transformed to “RGB” (e.g. “R”, “BGR;15”, “RGBA;L”).

I've created PR #6031 to add this missing transformation (in Pillow development terms, an "unpacker") from RGBA;15 to RGB.

[CookiePLMonster]

Hey, thanks! That looks good.

However, does that mean it's impossible to preserve the transparency bit in palettes in my case? TIMs palettes have a single bit signifying if the color is transparent or not, so it should translate to alpha reasonably well when exporting, but I don't know if Pillow lets me do that.

[radarhere]

I expect this code should let you make use of alpha.

image = Image.frombytes('P', (image_width * 2, image_height), image_data, 'raw', 'P')
palette = ImagePalette.raw("RGBA;15", clut_data)
palette.mode = "RGBA"
image.palette = palette

name = os.path.splitext('title_gtmode.tim')[0]
image.save(name + '.png')

Do you have a palette containing transparency that you can test it on?

[CookiePLMonster]

I tried the following example and it throws unrecognized raw mode inside image.save. An example image containing alphas was 4bpp and not 8bpp so I had to slightly update the code to add support for P;4 - I also had to swap the nibbles around as else "right" and "left" pixels were in wrong order, and I don't think I can tell Pillow to swap them:

import sys
import struct
import os
from PIL import Image, ImagePalette

if len(sys.argv) < 3:
    exit

mode = sys.argv[1].lower()
if mode == 'unpack':
    clut_data = None
    image_data = None
    image_width = 0
    image_height = 0

    with open(sys.argv[2], 'rb') as tim:
        tag, version = struct.unpack('BB2x', tim.read(4))
        if tag == 0x10:
            if version != 0:
                sys.exit(f'Unknown TIM file version {version}!')
            
            flags = struct.unpack('B3x', tim.read(4))[0]
            bpp = flags & 3
            clp = (flags & 8) != 0

            if clp:
                # Parse CLUT
                length, x, y, width, height = struct.unpack('IHHHH', tim.read(12))
                clut_data = tim.read(length - 12)
            
            # Parse image
            length, x, y, width, height = struct.unpack('IHHHH', tim.read(12))
            image_data = tim.read(length - 12)

            if bpp == 0:
                # 4bit, groups of 4 pixels
                image_width = width * 4
                rawmode = 'P;4'
                mode = 'P'

                # TODO: Is there a better way to do it in Pillow? Order of nibbles needs to be swapped
                image_data = bytes(map(lambda x: ((x & 0xF) << 4) | ((x >> 4) & 0xF), image_data))
            elif bpp == 1:
                # 8bit, groups of 2 pixels
                image_width = width * 2
                rawmode = 'P'
                mode = 'P'
            elif bpp == 2:
                # 16bit, each pixel separate
                image_width = width
                rawmode = 'RGB;15' # TODO: Alpha?
                mode = 'RGB'
            elif bpp == 3:
                # 24bit, 3-byte groups
                # TODO: Verify this
                image_width = (width * 3) / 2
                rawmode = 'RGB'
                mode = 'RGB'
            image_height = height
        
    if image_data:
        image = Image.frombytes(mode, (image_width, image_height), image_data, 'raw', rawmode)
        palette = ImagePalette.raw('RGBA;15', clut_data)
        palette.mode = 'RGBA'
        image.palette = palette

        name = os.path.splitext(sys.argv[2])[0]
        image.save(name + '.bmp')

GT1_Trial_Mountain.zip

[radarhere]

Ok, I've created PR #6054 to resolve this. With that addition, your code (without the change from my earlier comment) produces this image.

[CookiePLMonster]

Thanks, that looks very good! That is indeed the image you should get when interpreting the transparency bit as alpha.

Slightly offtopic but still on the topic of palettes with alpha, how should I go about retrieving such a palette back from an image, e.g. after a conversion? (my current toolchain works on RGBA images after all due to poor support of indexed PNGs with alpha from GIMP)

My current approach is to use im.palette.getdata()[1] but it feels improper. From my understanding, my use case calls for .tobytes() to be used (as I wish to retrieve a 16-color RGBA palette for serialization), but unlike .getdata() it'll fail if the palette is in "raw mode":
https://pillow.readthedocs.io/en/stable/_modules/PIL/ImagePalette.html#ImagePalette.tobytes

In an ideal scenario, I feel like the best way would be to retrieve the palette the same way I retrieve image bytes, i.e. through a decoder like image.tobytes('raw', 'P;4'). This way I could call im.palette.tobytes('raw', 'RGBA;15') or im.palette.tobytes('raw', 'RGBA') and be 100% sure that the return value is in the format I expect.

That is probably a feature request separate to this ticket, but I'm asking in case I am missing something obvious in Pillow.

EDIT:
An ability to specify the output bytes format for palettes may also be largely useful when dealing with images thay may or may not have alpha - so with an explicit conversion request, the user would not have to guess the bytes format.

[radarhere]

If you're after RGBA values, im.palette.getdata() will return RGBA values from your code after #6054.

If you're after RGBA;15 values, then if we added a "packer" from RGBA to RGBA;15 (almost the inverse of #6031), then you could do image.im.getpalette("RGBA", "RGBA;15") - but that's making use of a private API, so I can't recommend that and understand if you request a public interface for this purpose.

I'm not sure why in your last edit you mentioned guessing the returned format of the palette - I would have thought that was exactly why the first return value of im.palette.getdata() is the mode.

[CookiePLMonster]

I'm not sure why in your last edit you mentioned guessing the returned format of the palette - I would have thought that was exactly why the first return value of im.palette.getdata() is the mode.

You're absolutely right, although arguably it's not the most user friendly way of dealing with the palettes.
This differs slightly from the examples in OP as due to GIMP's issues with indexed PNGs I had to resort to exporting/importing RGBA images after all. However, consider the following operations done by an importer part of my tool at the moment:

with Image.open(os.path.join(dir_name, rid)) as org_im:
    if org_im.getcolors(maxcolors=16) is None:
        print(f'WARNING: {rid} has more than 16 unique colors! The image will be quantized down to 16 colors when packing, but quality may suffer.')

    im = org_im.quantize(colors=16)
	
    imageData = im.tobytes('raw', 'P;4')
    imagePalette = im.palette.getdata()

Now depending on whether org_im has alpha not, imagePalette will contain RGB or RGBA data, and it's up to the user to "fill the blanks" if they expect an RGBA palette and get RGB instead. I currently work it around by opening the image with with Image.open(os.path.join(dir_name, rid)).convert('RGBA') as org_im: instead, but maybe it's not the best approach.

EDIT:
image.im.getpalette("RGBA", "RGBA;15") sounds reasonable but I don't intend to push for it since it won't be of much use for me. However, it does sound like it could help bring feature parity between Image tobytes with an encoder and Palette's tobytes without.