Quality in Image API v3: Replace color with full

[tomcrane]

Links

• Original Issue: Quality in Image API v3: Replace color with full api#1839
• Pull Request: Replace color quality with full api#1842
• Image API 2.1: https://iiif.io/api/image/2.1/#quality
• Preview: https://preview.iiif.io/api/1839-quality-full/api/image/3.0/#44-quality

Background and Summary

The 2.1.1 spec says:

The quality parameter determines whether the image is delivered in color, grayscale or black and white.

For example, .../full/max/0/gray.jpg would return a grayscale image.

This can be confusing if the source image is grayscale, or uses a color space but happens to not have any non-gray pixels. Should a server offer the color quality? What should it return?

The editors propose that potential ambiguity about whether a server should offer color, and what it should do with requests for color, could be resolved by using the term full to avoid the implication that a color quality image should contain non-gray-shade pixels, or use particular color encodings.

Proposed Solution

Possible qualities become:

• bitonal: The image returned is bitonal, where each pixel is either black or white.
• gray: The image is returned in grayscale, where each pixel is black, white or any shade of gray in between.
• full: The image is returned with as much color information as available.
• default: The image is returned using the server’s default quality (e.g. full, gray or bitonal) for the image.

A full image may contain only gray shades, if that is all that is available: it is legitimate for the server to list full in this case even if the response would be identical to that for gray. No prescription about the color space to be used in the returned image is implied.

There may be reasons for a server to return something different for default than for full (IIIF/api#1839 (comment)), so default remains, and will continue to be the standard request, as in the current API.

This appears to be quite a significant change, but the usage of explicit color is, we think, very low to non-existent.

[ahankinson]

A few thoughts & questions:

The spec says:

A request for an image with an unsupported quality value should return a 400 (Bad Request) status code.

If a server cannot respond with a colour image, will this change have an undesired effect on the ability for a server to respond appropriately? Or will clients need to inspect the images they return to figure out the colourspace? Would there ever be a time when full would return a 400?

Both bitonal and gray are explicit about what colourspace is being requested. color is as well, but replacing it with full would make this less explicit. We would have two parameters that can return a variable colour response (default and full), and two that return a specific colour response (gray and bitonal). This change would seem to have the counter-intuitive result of reducing support for colour images, rather than increasing it.

Imagining a scenario:

A client wishes to work with colour images only to feed into a machine-learning workflow built specifically on the need for colour images. If the client can't get a colour image, they expect to catch a 400 response and move on. Most services respond with a colour image with a request for full, but one service provides a variable response for full of either colour or bitonal. In this scenario it seems like the client has no mechanism to inspect the specific types of image, pre-download, and are forced to download every image and discard the ones they can't use.

---

Of less importance: full is a quite loaded term in the Image API. By my count it can mean "All pixels in an image in all dimensions", "All pixels in an image at the largest unscaled size", and now "The most complete RGB values available for each pixel in an image".

https://example.com/{ident}/full/full/0/full.jpg

Is full in danger of being overused? Or has it become the default value for "give me everything" in the Image API?

[tomcrane]

Hi @ahankinson

A client wishes to work with colour images only

The issue with the current 2.1 spec is that it's not clear whether asking for color every time (assuming the image service declares it available) means that any image returned for this quality will have at least one non-grayscale pixel in it. I don't think it does mean that, I think it means that the server won't transform the image to reduce it to bitonal or grayscale, and instead will return it with as much color information as it has (or wants to make) available. Which for some images, will mean a response that has no non-grayscale pixels in it, because there simply aren't any in the source.

So by renaming the quality color to full, it makes it clearer what is being promised and what is being asked for.

It does mean that your use case would require the client to inspect the image to see whether it actually does contain non-gray pixels, but I think that is also the case with 2.1, it's just not clear (what do Loris, IIPImage etc do?)

On the overloading of full - I personally am OK with that, as it always make sense in context (vague echo of items in Presentation 3). And the canonical request for the biggest image you can give me without me taking a look at the info.json is now in Image 3 /full/max/0/default.jpg, so only one full parameter. (The size parameter no longer permits full - it was deprecated in 2.1 and shown the door in 3).

Also, I think we have to be careful about the phrase "color space", and be clear that the spec isn't mandating anything about the technical implementation of your returned images, other than the color values of pixels. For discussion later, maybe:

Is it valid for a bitonal request to return an image that only has black and white pixel values in it, but uses 32 bits per pixel to encode them?

I think the answer is yes.

https://iiif.bodleian.ox.ac.uk/iiif/image/daf51525-07c5-43ca-ac63-4e3060694572/full/1000,/0/bitonal.jpg

If a client asks for .../bitonal.png it's up to the server to be efficient in encoding it.

See also @beaudet's comment - IIIF/api#1839 (comment)

[ahankinson]

Thanks @tomcrane

bitonal and grayscale are interesting cases, as it's possible to derive these from a colour image. So asking the server for bitonal when the original is colour is possible, but asking the server for colour when the original is bitonal should result in an error.

If a client can only work with or wants a colour image, and they specifically request that, then I think they should receive an error when the server is unable to respond with one.

I'm OK with having a full parameter, in addition to default, but I'm also not really sold on removing color. If anything, having full would give us the opportunity to make the interpretation of color stricter.

[cubap]

I agree that full is a bit loaded, but it does not dissuade me from including this. In fact, it is a sort of fixed vocab we're using to say (within the API) "give me the most you have". This may be size, spectrum, or crop and is not imprecise in its meaning.

[aisaac]

One question that may complexify the problem space, or provide a litmus test for these parameter values and how they should be interpreted and used: what should be the value for a colorized image? 'full' or something 'more than full'? (or should it should be treated as a different image?)

(I'm trying to understand how much 'full' means 'original' or something else)

[cubap]

A client wishes to work with colour images only to feed into a machine-learning workflow built specifically on the need for colour images. If the client can't get a colour image, they expect to catch a 400 response and move on.

Using/retaining color here may be useful, but I only imagine this is helpful when getting a 400. The concept of color is way to broad to rely on just the image path instead of other image metadata for cleaning a set for processing.

[ahankinson]

I think I understand the problem this issue is trying to address, but keeping 'bitonal' and 'gray' in the spec, while removing 'color', will lead to further behaviour inconsistencies about how servers can be expected to respond.

While a variable response from 'default' can establish a sensible global response (that is, services can set a sensible default for their responses), 'full' will give services the ability to respond with variable image types and no real guarantee that what the client will receive will be what is expected. For example:

• Does 'full' mean that a client can request the lossless version?
• Does 'full' mean that the server can respond with the 16-bit version of the image, rather than the 8-bit? Or with RGBA instead of RGB?

I think if the goal of this change is to clarify the behaviours of image servers and make it consistent, then we need to further clarify what the expected behaviours are. Without a definition of what 'full' actually means, I'm not sure that this change has that effect.

[gigamorph]

It seems to me we could use any of full, native, max and such and they all are kind of overloaded in meaning. If we interpret the quality labels as request for "best effort" match, as in @ahankinson's example (during the call) of a greyscale version delivered from a bitonal source image (or color from greyscale or bitonal), the current color might just work, that is, we don't seem to gain much from using full instead.

[tomcrane]

I think there are two overlapping issues being discussed here. One is about commands, the quality param advertising what transformations the server will allow you to request, and then in making the request using a valid supported quality instruction, the client telling the image server to make that transformation.

The other is about information: does there exist in this image at least one pixel whose colour value is not R==G==B?

Which is not a supported command, but an additional piece of information, like an extremely minimal palette service. Something an image server could support in some other way, to be defined.

I agree with @ahankinson's use case, it's a valid scenario, and the proposed spec change clears up the ambiguity about whether it's possible to meet that use case by inspecting supported qualities in the info.json (it isn't).

For the use case, information about the range of values present in the image's palette would then allow the client to request full quality for one image, or skip another image because it isn't colourful and therefore not suited to the machine learning task. But that info doesn't belong in the list of supported quality "operations", I think.

On a different note the proposed spec change includes this:

The quality parameter does not determine the color space or impose other technical constraints on the returned image, other than the color values of the pixels it contains. It is valid for a server to return an image with gray or only black and white pixels in response to a full quality request, if that is all that is available.

The "color values of the pixels it contains" could be clarified some more. What it's trying to say is, regardless of the bits per pixel, RGBa or whatever nature of the returned image:

• bitonal => give me an image in which only black and white can be seen
• gray => give me an image in which only shades of grey can be seen
• full => give me an image with as many shades/colours as you can

[ahankinson]

As a point of order: Is this the right place for the conversation? I'm happy to continue it, but I seem to remember there being an agreement that TRC issues aren't the place to hash things out, and should be restricted to voting and minor clarifications. @hadro ? @glenrobson ?

[alexwlchan]

I have no strong feelings about this particular change, but a few notes below.

---

Your use case would require the client to inspect the image to see whether it actually does contain non-gray pixels, but I think that is also the case with 2.1, it's just not clear (what do Loris, IIPImage etc do?)

Loris makes a half-hearted attempt at checking if color is an appropriate quality..

When it receives a request, it checks the colour model of the original image (RGB, CMYK, LA, etc.). If the colour model supports non-grey pixels, it allows the color quality, but it doesn’t check to see if the image actually contains non-grey pixels.

If you try to request an image in a quality that the colour profile doesn’t support, Loris returns a 400 error. Examples:

https://iiif.wellcomecollection.org/image/M0000701.jpg/info.json (default, gray, bitonal)
https://iiif.wellcomecollection.org/image/M0000701.jpg/full/full/0/color.jpg

---

I don’t have details to hand, but Wellcome’s use case could make an argument in both directions (or for having both parameters, which is no decision at all).

Some of Wellcome’s images have embedded colour profiles that popular browsers ignore, so in Loris we convert every image to sRGB before sending it as color or default. You’re not getting the original colour palette because most of the time that’s probably not what you actually want. We did that for a while, and got complaints that a bunch of images looked wrong (they did, it was yucky).

In favour of full: AFAIK, there’s no way to get the image with the original colour profile. You have to email somebody and ask for the original file.

In favour of color: The term “full” carries a suggestion that you’re getting the original, unmodified colours, which in our case would be the not-sRGB, icky-looking image. How does somebody ask for a colour image that’s been tweaked to look right in their browser?

---

One subtlety in “does this image contain at least one non-grey pixel” is that it can vary on a per-transformation basis. If I have a massive image that’s entirely grey except one red pixel, eventually resizing will cause that red pixel to be lost. You could certainly work that out on every request, but I don’t think IIIF has a way to convey information about specific transformations? The info.json is transformation-agnostic.

e.g. You can’t say “quality color is supported if dimensions are greater than 800x600, but unsupported otherwise”

[hadro]

@ahankinson Sorry for my delay, but here's my take: While these issues aren’t exactly the preferred place for lengthy discussion, as TRC process has evolved we've seen it a fair bit, and almost always helpfully. Here these seem like comments and clarifications useful to deciding the issue, and so while we won’t be changing the item being voted on the discussion may inform future issue creation

[jbhoward-dublin]

I appreciate the perspectives voiced in this discussion as well as the issue that is being addressed. However I think there is potential for confusion in avoiding the use of plain language. One expects the term 'color' to be present when considering image attributes--especially if the plain language terms 'bitonal' and 'gray' are used in the same context. In the context of a greyscale image, would not 'full' also refer to the full spectrum of non-colour pixels to be returned? It strikes me that the definition 'The image is returned with as much color information as available' inherently implies that a colour images is requested; if so, then why not say so with 'color' instead of the ambiguous term 'full'?

[ahankinson]

@hadro ok, thanks!

[scossu]

I agree that using full in multiple places is not an issue; it actually makes things consistent. "Full" can be interpreted as "unmodified": for region, no cropping; for quality, no color transformation; etc.

I'm also in favor of leaving colorin because a client may require, for consistency, to process a batch of images as RGB, regardless of what the source color model is—e.g. to perform color analysis.

[azaroth42]

Issue 25 (Quality in Image API v3: Replace color with full)

+1: 14 [azaroth42 beaudet cubap emulatingkat irv jpstroop markpatton mcwhitaker mikeapp mixterj rentonsa scossu tomcrane zimeon]
0: 11 [aisaac andrewgunther dismorfo gigamorph jbhoward-dublin jtweed julsraemy regisrob rsinghal sredick tpendragon]
-1: 2 [ahankinson joshuago78]

Result: 14 / 27 = 0.52

No super majority, issue is referred to ex officio for decision

[zimeon]

The ex officio members of the TRC have decided to reject this proposal for now, pending further research and discussion (see TRC process)