Premultiplied alpha support

[kornelski]

Efficient compression of images with alpha benefits from using premultiplied alpha color space.

In AVIF that is marked by iref box with prem type: AOMediaCodec/av1-avif#96 (comment)

I'm attaching a test case. Both images in the zip file should look the same. If the premultiplied.avif looks darker, it's a bug.

prem-test-case.zip

image attribution: wikipedia

[joedrago]

So from a libavif perspective, am I understanding the support request correctly (after reading the standard):

• In addition to an iref of type auxl, consider an iref of type prem to also be an alpha auxiliary item
• Surface the boolean "the alpha is premultiplied" to avifImage
• If that boolean is set at encode time and alpha exists, use a reference type of prem instead of auxl for the alpha channel

Is this sufficient for libavif support? Obviously downstream projects will have to honor the boolean independently of libavif.

[kornelski]

Hmmm… I understood the spec differently. The alpha association remains exactly as-is, nothing changes there. And there in addition to image<>auxl assocation, there exists a reverse association of type prem that only acts as a boolean.

And if the boolean is true, then multiply by alpha when encoding, divide by alpha when decoding.

[joedrago]

@cconcolato Can you provide guidance on how the actual boxes should be structured for prem?

I'd prefer to not actually offer pixel manipulation in avifenc itself (other than the YUV conversion requirements, of course). I think it is reasonable to let people signal to avifenc if the data they're offering me is premultiplied or not, but I'd rather not get involved with converting their pixel data for them.

[kornelski]

It would be very useful to have an option to accept and return premultiplied alpha data unchanged. Premultiplied alpha is the "native" format on macOS and iOS.

If you don't want to do conversions, that's a bit unfortunate. I don't see this colorspace conversion much different than all the other conversions the library needs to do.

If you don't want to perform the conversion, please keep AVIF_RGB_FORMAT_RGBA to mean uncorrelated alpha and add another identifier for premultiplied. Otherwise if it's just an extra flag somewhere else, I'm afraid it will be too easy to miss it, and applications will ignore the flag and mangle the alpha. It's not a theoretical issue. The first QuickLook AVIF plugin I've tried has exactly this bug.

[joedrago]

In your mind, how would this correctly interact with YUV<->RGB conversion? I'm concerned that this will turn into a huge can of worms, and I haven't fit the issue entirely in my head yet.

@bnason-nf Do you have opinions on these mechanisms?

[kornelski]

AFAIK you can't apply it to YUV, so in avifImage it'd have to be just a flag, something like .alphaPremultiplied set to true.

I would expect avifImageYUVToRGB to be aware of the .alphaPremultiplied flag and act accordingly. If the desired format is AVIF_RGB_FORMAT_RGBA then interpret it as user asking for regular uncorrelated alpha, and undo premultiplication if the premultiplied flag was set.

If the .alphaPremultiplied is set and rgb image format is AVIF_RGB_FORMAT_RGBA_PREMULTIPLIED, then don't perform any alpha multiplication.

If the .alphaPremultiplied is false, and required rgb image format is AVIF_RGB_FORMAT_RGBA_PREMULTIPLIED, then do multiply by alpha.

macOS apps could be expected to always ask for premultiplied. Apps that convert to PNG could be expected to always ask for uncorrelated.

Technically it'd be similar to transferCharacteristics. It is a property of RGB channels, so you can't apply it to YUV image immediately, and need to defer it until conversion to sRGB.

[cconcolato]

The alpha association remains exactly as-is, nothing changes there. And there in addition to image<>auxl assocation, there exists a reverse association of type prem that only acts as a boolean.

I agree with that.

[baumanj]

FYI, there's support for premultiplied alpha now in Firefox nightly and should be part of the next release (86) when AVIF support will be enabled by default

[joedrago]

I think this is worth tracking still, for sure. I can make a new issue if you'd prefer to not be involved, however. Let me know.

[joedrago]

I just accepted PR #477, which should add support for this.