icc_transform vs colourspace for handling of 16-bit images vs 8-bit images

[daniellovera]

Hi,

I'm trying to better understand how icc_transform acts on 16-bit images with regards to using the output profile "srgb". What I'm trying to accomplish is taking whatever image the user supplies and standardizing it into a consistent format to work with in numpy, where the only difference in the form is the bit depth. So considering only 8-bit (uchar) vs 16-bit (ushort) images, I have the following code:

image_vips = vips.Image.new_from_buffer(image_bytes, "", access='sequential')
untagged_profile_default = "srgb"
if image_vips.get_typeof('icc-profile-data') != 0:
    # get and log the profile info using Pillow ImageCms
    image_profile = ImageCms.getProfileInfo(io.BytesIO(image_vips.get('icc-profile-data')))
    logger.info(f'Image profile is {image_profile}')
    elif image_vips.get_typeof('icc-profile-data') == 0 and image_vips.interpretation == 'cmyk':
        untagged_profile_default = "cmyk"
image_2_srgb = image_vips.icc_transform("srgb", embedded=True, input_profile=untagged_profile_default)
image_array = image_2_srgb.numpy()

My understanding is that for 8-bit images, this works correctly as it will transform to the srgb ICC profile in libvips. What I'm not clear on is if icc_transform applies the srgb profile or a rgb16 profile if the image is 16-bit. This

libvips/libvips/colour/icc_transform.c

Line 379 in 3adc62b

? VIPS_INTERPRETATION_sRGB

made me think that it's autoapplying a 16-bit profile if the image is 16-bit.

I read this comment (#580 (comment)) and ultimately that would work for me if I got all the 8-bit images as equivalent to VIPS_INTERPRETATION_sRGB (black to white 0 to 255, luminance encoded with gamma 2.4 plus straight line near 0, and 709 primaries for color) and all the 16-bit images as equivalent to VIPS_INTERPRETATION_RGB16 (same as sRGB except linear encoded [gamma = 1, right?] and 0-65535 for black to white). So I changed the code to the following:

image_vips = vips.Image.new_from_buffer(image_bytes, "", access='sequential')
bit_depth = image_vips.format
image_profile = None

if bit_depth not in ['uchar', 'ushort']:
    raise ValueError(f'Invalid bit depth of {bit_depth}')

if image_vips.get_typeof('icc-profile-data') != 0:
    image_profile = ImageCms.getProfileInfo(io.BytesIO(image_vips.get('icc-profile-data')))
    logger.info(f'Image profile is {image_profile}')

if bit_depth == 'uchar':  # 8-bit
    if image_profile is not None:
        image_2_numpy = image_vips.icc_transform("srgb", embedded=True, intent="absolute")
    else:
        image_2_numpy = image_vips.colourspace("srgb")
elif bit_depth == 'ushort':  # 16-bit
    if image_profile is not None:
        image_2_numpy = image_vips.icc_transform("srgb", embedded=True, intent="absolute")
    elif image_vips.interpretation == 'rgb16':
        image_2_numpy = image_vips
    else:
        image_2_numpy = image_vips.colourspace("srgb", image_vips.interpretation)
    
image_array = image_2_numpy.numpy()

The two things that have me kind of stumped are the difference between icc_transform and colourspace, specifically as I'm not going to be outputting these images to file at all since I just want them in a consistent RGB colorspace with a known gamma, and what's the colourspace that needs to be applied to the 16-bit images to make them match the RGB16 interpretation (or be the same as RGB16 except with a non-linear gamma, that's fine too)?

[jcupitt]

Hello @daniellovera,

image_vips = vips.Image.new_from_buffer(image_bytes, "", access='sequential')
untagged_profile_default = "srgb"
if image_vips.get_typeof('icc-profile-data') != 0:
    # get and log the profile info using Pillow ImageCms
    image_profile = ImageCms.getProfileInfo(io.BytesIO(image_vips.get('icc-profile-data')))
    logger.info(f'Image profile is {image_profile}')
elif image_vips.get_typeof('icc-profile-data') == 0 and image_vips.interpretation == 'cmyk':
        untagged_profile_default = "cmyk"
image_2_srgb = image_vips.icc_transform("srgb", embedded=True, input_profile=untagged_profile_default)
image_array = image_2_srgb.numpy()

I don't think you need to do this, do you? icc_transform() will pick an appropriate input profile for you if the image doesn't have one:

https://github.com/libvips/libvips/blob/master/libvips/colour/icc_transform.c#L670-L750

So just this ought to work:

image_vips = vips.Image.new_from_buffer(image_bytes, "", access='sequential')
if image_vips.get_typeof('icc-profile-data') != 0:
    # get and log the profile info using Pillow ImageCms
    image_profile = ImageCms.getProfileInfo(io.BytesIO(image_vips.get('icc-profile-data')))
    logger.info(f'Image profile is {image_profile}')
image_array = image_vips.icc_transform("srgb").numpy()

The difficulty you might have is that many user images have incorrect profiles, often a cmyk image with an embedded srgb profile :( libvips will try to fix this, but it might not always make the right choice.

What I'm not clear on is if icc_transform applies the srgb profile or a rgb16 profile if the image is 16-bit.

ICC profiles don't care about depth -- the same profile will do 8 or 16 bits. Whatever the input image depth, icc_transform will always write an 8 bit image by default, or 16 with depth=16. So I'm not sure you need the second bit of code, won't just image_array = image_vips.icc_transform("srgb").numpy() do what you want? I've probably misunderstood.

[daniellovera]

Thank you. If there was a misunderstanding, it's mine regarding the subject material. That said, given how difficult images have been to work with prior to learning about libvips, this feels like magic. Essentially, one line of Python image_array = image_vips.icc_transform("srgb").numpy() enables me to take nearly any image that exists and transform it into a ndarray with the sRGB colourspace, 2.4 gamma, and bit-depth/type corresponding black to white (e.g. 0-255 or 0-65535), and the catch is that it doesn't always make the right choice on rectifying user errors?

That's some next level wizardry.

That leaves a few questions that'll help my understanding, if you have a moment:

1. Is the difference between .icc_transform and .colourspace that the former changes the attached ICC profile while the latter does not?
2. Is there a libvips recommended colourspace that's analogous to Adobe RGB? I was looking at AdobeCompat-v4.icc from saucecontrol and I was curious if there's risks to icc_transform success with using a external icc profile and that's why libvips chose to host default profiles for sRGB, p3, and CMYK.
   a. This (Q2) is not related to the initial question, it's related to a future need to transform all images intended for digital display to p3 and all images intended for print to Adobe RBG so as to minimize data loss.
3. .icc_transform("srgb") on a 16-bit (ushort) image converts to a 2.4 gamma, not a linear encoding like VIPS_INTERPRETATION_RGB16, correct?

Thanks again, I've learn a ton about images and image processing from reading your posts here and in many other places.

[jcupitt]

It's the difference between a device space and a standard colourspace.

Stepping back a bit, in 1931 the CIE defined XYZ colourspace, a way of describing colours based on measurements of the human visual system:

https://en.wikipedia.org/wiki/CIE_1931_color_space

The experiments were actually done in 1929 and hilariously were based on 10 male students at Imperial College. Ten people almost 100 years ago all of the same sex, age and ethnicity set the basis of all colour representation for the whole world.

There are a set of other spaces derived from XYZ which are useful in different situations: Lab, Yxy, LCh, CMC, scrgb, etc. libvips supports the major ones.

The colourspace operator moves images between these spaces.

So the CIE spaces are linked to the human visual system and are universal and unchanging.

Device spaces on the other hand are the RGB or CMYK values for a specific bit of hardware. They are the voltages that go into the back of a monitor, or the ink density on a bit of paper. They are NOT universal or unchanging, they are very tied to the details of a bit of hardware.

ICC profiles are descriptions of the behaviour of a piece of hardware. They let you compute the voltages to send to a display to get a certain colour, or conversely the voltages that would be required to get a specific colour from a display.

icc_transform uses a pair of profiles to move from one device to another, icc_import lets you move from a device space to a CIE space (usually XYZ, but it can be Lab ... it's usually called PCS, meaning Profile Connection Space), and icc_export takes a CIE space back to a device space.

So! If you want a convenient and universal colourspace to work in, I would suggest scrgb. This uses three float numbers for RGB, gamma 1.0, with 0 - 1 for black to white. It uses the sRGB primaries, but scrgb values can go over 1.0 and under 0.0, so you can represent ANY colour, not just the ones in the sRGB triangle. Plus float is very easy to work in. It'll do HDR as well, obviously.

The downside is that you now have 12 bytes per pixel, which can be tricky for large images and numpy. libvips handles scrgb well since everything is on demand and memory use stays modest, but numpy can struggle. Maybe you can use libvips instead? It depends on the application.

[jcupitt]

https://en.wikipedia.org/wiki/ScRGB

[daniellovera]

I think I understand my confusion now. It just so happens that sRGB is both a device profile AND a colourspace, so for my intents and purposes .icc_transform and .colourspace are equivalent in how the numpy array would look using my code, though if I exported to a file the attached ICC profile may be different depending on which method was used and which (if any) ICC profile was on the input image.

My primary need was to convert the image into a sRGB numpy array (uchar or ushort) which is ultimately converted to float32 and processed with CRAFT text detection. Thanks to your help, image_array = image_vips.icc_transform("srgb").numpy() gets me what I need. If processing speed was my bottleneck, I could probably speed up processing even more by rewriting the normalization functions in libvips and adding it to the transform.

My secondary need, and where I think I wasn't communicating well, is that I need to also to understand the intended device profile of the image and then create a secondary copy of the original image in memory using bucketing for each image to have one of three device profiles [sRGB, p3, a98], so that I can perform the correct calculations on the image based on its intended device space. And if I'm understanding the difference between colourspaces and ICC_transform correctly, it means that I want any image with profile a98 or VIPS_INTERPRETATION_CMYK to be .ICC_transform(adobe98), any image with a sRGB or None profile to be .ICC_transform(srgb), and then all other images to be .ICC_transform(p3). Stellar, using libvips maximizes putting images in the correct bucket and then minimizing data loss on conversion.

scrbg would be the right colourspace to work in for all I need to do, it would simplify the code tremendously and be perfect for a float16 retraining of CRAFT, and future proofing for HDR. Unfortunately I don't have the time to rebuild that way, though it does seem like there's a big opportunity for text detection/recognition that's trained on something better than compressed 8-bit JPEGs given everyone's phone produces much higher quality images nowadays but the training datasets are from 2012-2015.

I have been replacing skImage, Pillow, and OpenCV functions with libvips as it makes sense for me, which is often. It's been excellent to replace 2 paragraph-length blocks with one liners that are faster, easier to read, and less error prone. Appreciate the work you all have done on this library, it's simply tremendous.