Enhance gallery example on color separation of stained tissues

[grlee77]

Moved from now deleted Discussion https://github.com/scikit-image/scikit-image/discussions/6065

^{Originally posted by mkcor November 26, 2021}
Hello,

I'm starting this discussion as per #5172 (comment).

First, let me thank @alessiamarcolini!

you can search for "h&e -commons" (the - to exclude wikipedia results), and then under search tools there is "Usage Rights", where I selected "Creative Commons licenses"

Wonderful, you've definitely upped my search game! 💪

@alexdesiqueira I was thinking of enhancing the current example with an additional section based on an additional image (i.e., an H&E image such as the one @alessiamarcolini spotted).

IMHO, we could solve that using the example we already have, and presenting three subplots: Original, H&E (their sum), DAB — as discussed previously.

I think the three subplots for the IHC image should be: Original, H, DAB; for the H&E image, they should be: Original, H, E (as the name suggests). The H&E sum does not make sense in the IHC case, because no Eosin staining was used in the first place... (I'm trying to tell a story which relates to real use cases in the life sciences, I'm not trying to rearrange subplots so that they render nicely...)

But...

this H&E image has all three stains!! Hematoxylin in blue-purple, Eosin in pink, and DAB in brown. How come? @crisluengo I expected no DAB stain. 😕

---

crisluengo on Nov 26, 2021
Well, there are several things going on.

First of all, note how the unmixed Eosin color doesn't match the color in the original image. Colors on a slide vary significantly depending on the Eosin vendor, other chemicals used, the pH value of the tissue, etc. etc. Then, inside the microscope, the light color might change depending on the type of light source, and how the light color is converted to an RGB triplet depends on the camera make and model. Microscope systems typically don't bother with color correction like we do in photography. So the RGB values for pure Eosin stain depend on many variables... The rgb2hed() function uses some values that were measured at some point by someone in one lab, for one slide. The likelihood that it's a good match for your image is slim.

Next, Eosin tends to produce different colors on different parts of the tissue. Note how there are some red regions in the original image, which are even further away from the Eosin color as encoded in the rgb2hed() function. Those red regions produce a stronger signal in the DAB channel. It's just a residue. Adding up the Hematoxylin, Eosin and DAB channels will not give you the colors in the original image in this case, I bet a lot of the white regions in the Hematoxylin channel had negative values in the algorithm and were clamped to zero. Think of the red as some amount of pink, plus some amount of brown, minus some amount of blue.

Actually Eosin doesn't perfectly match the physical model used in stain unmixing, because of how it changes color.

---

crisluengo
on Nov 26, 2021

...of course you could create a clean image to unmix by setting hne_hed[:,:,2]=0, and then hne_rgb=hed2rgb(hne_hed).

Looking through the documentation, I see there is a matrix for Hematoxylin + DAB (rgb_from_hdx), but no Hematoxylin + Eosin; instead there is Hematoxylin + Eosin + DAB (rgb_from_hed). This latter is the only one with a function made around it, note that all this does is separate_stains(rgb, hed_from_rgb), it's just as easy to call separate_stains(rgb, hdx_from_rgb). And it'd be just as easy to create a matrix for Hematoxylin + Eosin. If you do that, this whole demo will be cleaner and easier to explain.

I also see that the rgb_from_hdx matrix is a 3x3 matrix, meaning it outputs 3 channels. I presume the 3rd channel is a bogus channel. Why not make this a 3x2 matrix? Is there a reason skimage always wants 3 channels?

rgb_from_hdx = np.array([[0.650, 0.704, 0.286],
                         [0.268, 0.570, 0.776],
                         [0.0, 0.0, 0.0]])
rgb_from_hdx[2, :] = np.cross(rgb_from_hdx[0, :], rgb_from_hdx[1, :])
hdx_from_rgb = linalg.inv(rgb_from_hdx)

The Hema+DAB array gets a 3rd row composed as the cross product, which is a color orthogonal to the other two. This should make the inverse robust. But it is also possible to do a pseudo-inverse of the 2x3 matrix, yielding a 3x2 matrix that is equal to the first two columns of the hdx_from_rgb constructed above. Avoiding that additional channel would be neat, because then there is no confusion as to what that channel means.

I also see that rgb_from_bex (Methyl Blue + Eosin) uses a different color for Eosin ([0.07, 0.99, 0.11] for HED, [0.092789, 0.954111, 0.283111] for BEX). This must have been someone else in another lab measuring an Eosin stain on one other slide... :)

wahaha98 on Mar 1
Hello, I would like to ask how the values in the Eosin ([0.07, 0.99, 0.11] matrix are calculated by RGB?

crisluengo on Mar 1
What is typically done is stain a slide with only one stain (Eosin in this case) and measure its color. The idea being that a slide stained with only Eosin, and a slide stained with only Hematoxylin, will show the same colors as a slide stained with both.

---

mkcor on Dec 3, 2021

Oh, I forgot that we already have data.skin()... 🤦‍♀️
https://scikit-image.org/docs/dev/auto_examples/data/plot_scientific.html

---

crisluengo
on Mar 2

Here's a suggestion for how to modify skimage's stain unmixing functionality. HED is just confusing and useless. If there are two stains on the slide, you want the stain unmixing to yield two channels, not three.
crisluengo@431aa9a

These changes involve backward-incompatible changes, so I'm not making this into a PR. I just made those changes to help a conversation about this topic.

---

grlee77
on Mar 2

Yeah, I think that could be helpful. We are actually planning on making an skimage2 release later this year with a limited number of breaking changes so that would be a good opportunity to do this. If you want to open PR I can tag it for that purpose. (Mainly, I just don't want this to get lost as we are currently in the process of migrating GitHub Discussions back to the issues page).

On a related note in the RAPIDS cuCIM package, the NVIDIA team proposed a PR for a different stain normalization than the one in skimage. That difference is why it isn't currently under the cucim.skimage namespace, but I was planning to review and see if we should port it here. I haven't reviewed it properly yet, but on initial glance it does look like it produces a 2-channel H & E image.

---

crisluengo
on Mar 3

@grlee77 See #6274.

There are two options there, the commit I referenced earlier, and an even more invasive one I just did now, after hearing about skimage2. This new version, I hope, makes the stain unmixing look and work differently from color space conversions, to avoid the confusion I've seen so far.

[alexdesiqueira]

(I'm trying to tell a story which relates to real use cases in the life sciences, I'm not trying to rearrange subplots so that they render nicely...)

@mkcor sorry about not being clear enough. While researching for #4725, I checked a bunch of papers and they aligned with @crisluengo's H&E proposal. On top of that, now we have @grlee77 tip on rapidsai/cucim#186.

[medha-chippa]

Hello,

I have also been facing issues with performing accurate stain separation with scikit image. Has anybody been able to find a fix?

Thank you

[crisluengo]

@medha-chippa "issues" is hard to debug. What is the problem? The stain separation currently in skimage should be working correctly. You need to make sure the background (no stain) is pure white, and you need to find the appropriate stain vectors for whatever combination of stains you're using. The default stain vectors were measured decades ago in one lab, with stains from one specific manufacturer, applied in one specific way, and imaged with one specific color camera. Your colors will likely be different, ideally you'd measure the colors you produce in your particular lab. Or you use one of the methods that attempt to estimate the color vectors from the image, but those methods have not been implemented in skimage.