Support images with channels != 3

[agoode]

_Imported from Trac. This issue originally by _agoode* at 2010-05-06T16:45:01Z.*

For fluorescence imagery, channels may not equal 3. If channels = 1, then we can use the existing read_region API to produce gray. Otherwise, some new API will need to be invented.

openslide_open may fail if channels != 1 or 3. We may need a new open function as well.

[mrymsadeghi]

Hi,

I also have this problem. that I need to extract fluorocent channels separately. I'm using mirax file. But the channels when I extract the image, and extract eg green channel, you can still see traces of the cells in red channel. wheras, those traces are not visible in the 3DHistech slide viewer software.

[bgilbert]

Quoting @iewchen in #611:

With fluorescence, one frequently need to change the pseudo color and intensity of a channel. A special viewer is needed anyway. I can think of two ways to support fluorescence and other formats in grayscale.

• remix fluorescence channels into ARGB to output 3 or 4 gray channels. If OpenSlide can use CAIRO_FORMAT_RGBA128F, then the 128-bit format might be abused to output eight 16-bit, or sixteen 8-bit gray channels. I am currently remixing at most 3 fluorescence channels into ARGB with another branch of CZI driver.

• or add something similar to openslide_read_region(), but with a channel argument, to read one gray channel. Perhaps the 16-bit CAIRO_FORMAT_RGB16_565 can be used to output 16-bit gray, or CAIRO_FORMAT_A8 for 8-bit gray.

It is up to the viewer to separate fluorescence channels from ARGB so that channel can be manipulated individually, and assign pseudo colors to channels for display.

---

It is possible to support fluorescence with a small amount of work without breaking current API. For CZI, extending fluorescence support to z-stack and time series should be straightforward with appropriate testing slides.

In addition to the current osr->levels pointer array, we can add an osr->levels_hashtable, using keys like time_series << 48 | z-stack << 32 | channel << 16 | pyramid_level. So each channel/z-stack/time_serie/pyramid combination has its own grid. CZI tile is added to the grid and level it belongs. When read region, the level and grid are retrieved by looking up osr->levels_hashtable and region is painted.

Let's call the new read_region function openslide_read_region_gray16() or openslide_read_region_gray8(), which accepts arguments for channel number, z-stack index, and time series index. Other related functions can be duplicated and replace CAIRO_FORMAT_ARGB32 with a 16-bit or 8-bit format.

Edit: I am a bit excited because with 16-bit output, conventional fluorescence, confocal, DIC, TIE(Zeiss Transport of Intensity Equation, similar to DIC), even other grayscale DICOM images such as MRI and CT are all possible.

[bgilbert]

Just to set expectations, even small additions to the external API are a substantial amount of work, taking into account testing, documentation, and updates to OpenSlide Python. For the internal API, we should plan to refactor the existing drivers as appropriate, rather than bolting in a parallel mechanism. The goal there is to keep the future complexity of maintaining the library reasonably low.

The reason fluorescence support hasn't happened yet is that once we need to extend openslide_read_region() — and I think it's clear we should — it's not clear how general to make its replacement. As you mention, there's not only fluorescence, but also: high-bit-depth images, including RGB (#41); Z-stacks (#166); time series; and slides with multiple pyramids (#99). The dimensionality label lists issues that may be relevant here.

If we list each possible dimension explicitly as a read_region argument, there's the risk of missing one, and it also makes the API pretty unwieldy. I had the idea of creating a cursor object, e.g.

// one setter for each dimension
openslide_cursor_t *openslide_cursor_new(void);  // selects a default for each dimension
void openslide_cursor_set_level(openslide_cursor_t *curs, int32_t level);
void openslide_cursor_set_channel(openslide_cursor_t *curs, int32_t channel);
// one enumerator for each dimension
int32_t openslide_get_level_count(openslide_t *osr);
const char **openslide_get_channels(openslide_t *osr);
// validate cursor
bool openslide_cursor_is_valid(openslide_cursor_t *curs);
// read functions
// X and Y are still separate arguments here, but could be part of the cursor
void openslide_read_region_rgb_cursor(openslide_t *osr, openslide_cursor_t *curs, uint32_t *dest,
                                      int64_t x, int64_t y, int64_t w, int64_t h);
// 16-bit gray and 16-bit alpha
void openslide_read_region_gray_cursor(openslide_t *osr, openslide_cursor_t *curs, uint32_t *dest,
                                      int64_t x, int64_t y, int64_t w, int64_t h);

This would allow adding dimensions in the future without replacing any API functions. The cursor struct is opaque for this reason.

Or, instead of separate enumeration functions for each dimension:

// provide getters for each dimension, but no setters
int32_t openslide_cursor_get_level(openslide_cursor_t *curs);
const char *openslide_cursor_get_channel(openslide_cursor_t *curs);
// list all available cursors
openslide_cursor_t **openslide_get_cursors(openslide_t *osr);

This would make it easier to have irregular combinations of dimensions (for example: in slides with multiple pyramids, the pyramids may have different heights), at the expense of making the application search for the cursor it wants to use.

Or, we could provide a combination of both APIs: provide an API that lists all cursors, but also allow the application to construct a cursor which may or may not be valid. (This could potentially introduce confusion between cursors that must be freed by the application and cursors that must not be.)

I'm thinking we'd keep the existing 32-bit ARGB format for RGB images, but also allow reading individual RGB channels with the 16-bit gray API. As you observe, it's not obvious how best to support 16bpc with Cairo.

Thoughts?

[bgilbert]

In the latter approach, we could drop the cursor abstraction, and just have the application provide the offset into the level array:

// this is just the existing read_region function
void openslide_read_region(openslide_t *osr, uint32_t *dest, int64_t x, int64_t y,
                           int32_t level, int64_t w, int64_t h);
// 16-bit gray and 16-bit alpha
void openslide_read_region_gray(openslide_t *osr, uint32_t *dest, int64_t x, int64_t y,
                                int32_t level, int64_t w, int64_t h);

For compatibility, the existing levels would need to be the first ones in the level array. Without an explicit cursor, though, it would be difficult to support reading individual brightfield channels through the gray API.

Here the level array might be exposed via a function similar to openslide_get_cursors above, with an array of opaque structs read via accessors. But since OpenSlide is doing the allocating, it's safe to drop the accessors and return an array of non-opaque structs:

struct _openslide_level_desc {
  int32_t pyramid_level;
  double pyramid_downsample;
  const char *channel;
  ...
};
typedef struct _openslide_level_desc openslide_level_desc;
const openslide_level_desc_t **openslide_describe_levels(openslide_t *osr);

This is potentially easier for the application (or maybe not, depending how well the various language FFIs support parsing structs directly), but FWIW is contrary to the design of the current API.

[iewchen]

In the latter approach, we could drop the cursor abstraction, and just have the application provide the offset into the level array:
For compatibility, the existing levels would need to be the first ones in the level array. Without an explicit cursor, though, it would be difficult to support reading individual brightfield channels through the gray API.

Levels beyond the existing ones are not pyramid level in the traditional sense. Although it is backward compatible, it may confuse users when they call openslide_read_region(). The cursor abstraction is more elegant than extend the level array.

For 16-bit output, CAIRO_FORMAT_RGB16_565 might work. My impression is cairo cannot output png with CAIRO_FORMAT_RGB16_565 otherwise it is similar to other cairo surface formats. Output a 16-bit alpha with 16-bit gray is less efficient since this alpha is not useful to end user.

[iewchen]

I can confirm CAIRO_FORMAT_RGB16_565 indeed works for 16-bit gray. I tested a prototype and was able to read CZI fluorescence channel. With PIL.Image.frombuffer('I;16', size, buf, 'raw', 'I;16', 0, 1), openslide-python is able to convert Openslide buffer into 16-bit Pillow Image.

[iewchen]

Our lab recently started working with 4 channel CZI fluorescence slides. Here is the version we are using. It shall work with more than 4 channels, though I haven't tested that.

OpenSlide: https://github.com/iewchen/openslide/tree/zeiss-czi-jxr-simd-gray

OpenSlide Python: https://github.com/iewchen/openslide-python/tree/gray

It extends the osr->levels array to include levels from all channels. Two functions are added:

void openslide_read_region_gray8(openslide_t *osr, uint8_t *dest,
                                 int64_t x, int64_t y, int32_t level,
                                 int64_t w, int64_t h);
    
void openslide_read_region_gray16(openslide_t *osr, uint8_t *dest,
                                  int64_t x, int64_t y, int32_t level,
                                  int64_t w, int64_t h);

So they accept similar set of parameters as current read_region API. Caller needs to calculate level as:

level = (total_levels_in_one_channel * channel_num) + level_in_channel

If the image is stored as gray16 but user asks for gray8, then openslide_read_region_gray8 will convert gray16 to gray8.

The Python binding has two new functions:

read_region_gray8((x, y), channel, level, (w, h)): return a 8 bit Pillow image
read_region_gray16((x, y), channel, level, (w, h)): return a 16 bit Pillow image

Deepzoom can also read gray tile by calling get_tile_gray8(channel, level, (col, row)) function.