Come up with a global normalization strategy instead of a per-tile strategy

[lacan]

Background

Things like the StarDist extension or cell detection work by tiling the image data in order to process these tiles in parallel, saving time.

While for cell detection, local background subtraction schemes make sense, as a hard threshold follows the preprocessing steps, this is more questionable for Deep learning approaches, where on "empty" or very sparse tiles, the model will dream up cells because each tile is normalized individually.

Moreover, the actual implementation of StarDist does perform the normalization on the full image, before splitting it into tiles. It would be nice to reach that point in QuPath as well.

Project

The challenge lies in being able to compute a meaningful quantile (the most common normalization approach) on the data at the same resolution as the one when the processing.

The goals of this project would be

• Define a way to estimate a global normalization value (min, max or mean/sd) to use when tiling images for prediction. One approach would be to use a downsampled version (that allows to make the calculation in an acceptable time. The quantile estimation should not be perfect, as most robust models allow for some small variations from a perfect normalization

[petebankhead]

Yes!

This method was intended as a starting point (until I forgot about it...): https://github.com/qupath/qupath/blob/main/qupath-core-processing/src/main/java/qupath/opencv/tools/OpenCVTools.java#L931

Basically, the idea was that you'd need to create a Mat for the relevant image, then mask out pixels you don't want by setting them to NaN (maybe using OpenCVTools.fill?). Then you can get the percentiles using that.

But as you pointed out on the forum, we also need to think about per-channel and joint-channel normalization... so defining how it should look at the end sounds like the trickiest bit.

Two other thoughts:

• Is it worth the effort of developing some kind of 'histogram accumulator' to work on tiles? I guess pre-requesting all relevant tiles at the output resolution would sometimes be efficient enough.
• Would there be any benefit in adding a method that tries to find the global minimum and maximum for all channels, using information across scales? I'm thinking something relatively simple like:
  • Find the min and max in a downsampled image
  • Request the tiles nearby, and search them for minima and maxima

It wouldn't be guaranteed to be correct, but it is likely to be a better estimate than using the downsampled image alone.

[lacan]

Would there be any benefit in adding a method that tries to find the global minimum and maximum for all channels, using information across scales?

My thoughts are that for most biological images the minimum and maximum values are not too relevant (Always 0 for confocal images, unless the offset is wrong, small clipping for the max pixel values in a lot of cases), which leads me to think that histogram/quantile based methods are definitely more profitable. Hence:

Is it worth the effort of developing some kind of 'histogram accumulator' to work on tiles?

I really like that idea! Avoids wondering if the downsampled image data would contain a good approximation of the histogram.

Otherwise, something else that we could quickly do, is grab a bunch of images, and compute their downsampled histograms and use a simple metric to see how similar the relative histograms are, as well as compare the quantile results at different scales. A bit like assesing what level of JPEG compression is acceptable for a given image analysis task (https://dx.doi.org/10.1186%2F1746-1596-7-29)

[MichaelSNelson]

(Always 0 for confocal images, unless the offset is wrong, small clipping for the max pixel values in a lot of cases)

Way off topic, but nooooooooo! Haha. One of the things QUAREP is striving against is incorrectly taken microscopy (sometimes confocal) data, which I learned far too late should *never be 0. I innocently trusted my Zeiss reps at first...

*Never say never.

[petebankhead]

I like the sound of all of this - although I'm in the 'never 0' group :)
Even if the offset is 'ideal' and only noise is clipped, I like to keep all the noise so it can be estimated to give a baseline.

But this does remind me of another value in getting the min and max: the max helps determine the 'true' bit-depth of the 16-bit image (e.g. 12-bit, 14-bit...). And that could be handy for the brightness/contrast dialog.

Meanwhile, check out @dkurt's PR at qupath/qupath#827
This should probably be the default way to calculate a histogram (at least for RGB) in ImageOps. I keep meaning to check if it gives identical results to the current approach, but I haven't got around to it yet.