Image Demosaicing Quality in PlantCV versus LemnaBase

[AdamDimech]

I have been using PlantCV for a while now to analyse LemnaTec Scanalyser visible-spectrum images, but it has come to my attention that the quality of the images generated by PlantCV (using OpenCV functions) is not as good as the native LemnaTec software and I am unsure why.

Images are downloaded using a forked version of Data Science Tools (LT-db-extractor.py). For our system, we initially found that OpenCV image conversion number 47 (cv::COLOR_BayerGB2BGR) worked best for us. Here's a close-up:

You can see, the quality isn't great around the edges of the object. Let's look at the same image exported via LemnaBase:

Here the colours are better and edges sharper. In the past few days, I have played around with LT-db-extractor.py and changed the function on line 184 from:

img = cv2.cvtColor(raw_img, db['colour'])
to
img = cv2.demosaicing(raw_img, db['colour'])

I am not entirely sure what the difference between these functions is, but this change along with a modification in the OpenCV conversion format to #63 (cv::COLOR_BayerGB2BGR_VNG) has lead to considerable improvement:

Whilst better, it's still not quite as good as the LemnaTec software version (ie there is still some colour artifacts on the edges). I am trying to work out whether there is a problem with OpenCV's image demosaicing process or some other factor. I have written a small Python script that will open a LemnaTec blob file and attempt to make all 143 conversions for easy comparison (of course most fail, but it generates a PNG of those that work for easy assessment).

The other thing I am trying to figure out is the format of the blobs. Do you know are they encoded and what actually sits within? Each seems to contain multiple images, encoded in some unknown format and the metadata seems to be obtained from the database via a SQL query. There is a "data" file in there, but what does that contain? In what format is it encoded?

So many questions, but some insight would be a great help.

[nfahlgren]

Hi @AdamDimech!

The blob files (e.g. blob49) on our system are zip files containing a file called data. Our data files are raw 8-bit binary files. The data encoded in the raw image file is the raw camera sensor data, which is a Bayer filter.

In OpenCV there are a lot of Bayer demosaicing methods but they can be grouped into some basic sets. First, you could ignore all the COLOR_BayerXX2RGB methods as OpenCV functions (and PlantCV since we are using OpenCV primarily) want the images in BGR order. Unless you use an alpha or transparency channel, all the COLOR_BayerXX2RGBA can be ignored. The COLOR_BayerXX2GRAY can also be ignored since you're trying to convert to a color image.

All that's left are a few groups. The first part codes the Bayer filter pattern type (COLOR_BayerXX2BGR) and different sensors may have different patterns. The RG pattern has worked with our data. The COLOR_BayerXX2BGR methods use the default interpolation method, which I think is bilinear. The other two methods with the _VNG and _EG extensions do the same thing but use either the Variable Number of Gradients or Edge-Aware interpolation methods.

When you use cv2.cvtColor it uses a demosaicing algorithm if provided one of the Bayer methods. I ran both to compare and I got the same results:

img = cv2.cvtColor(raw_img, cv2.COLOR_BAYER_RG2BGR)
imgd = cv2.demosaicing(raw_img, cv2.COLOR_BAYER_RG2BGR)
(img == imgd).all()
# True

It looks like we need a method to specify the type of Bayer filter so the correct method can be used. I have not compared the Bilinear, VNG, or EA algorithm results but sounds like you are getting better results with VNG, which is interesting! I need to do some comparisons on our end too.

LemnaTec might not be doing the conversion with OpenCV. They could be using a different interpolation algorithm and/or it could be an algorithm from the camera/camera sensor manufacturer that uses a proprietary method. It's also possible they might do some other preprocessing to tweak the color gains in the raw image before demosaicing.

[dschneiderch]

When you use the Bayer import in lemnagrid you get options about which interpolation alg to use. Looks like the default is CFA from IPP, an intel library https://software.intel.com/content/www/us/en/develop/documentation/ipp-dev-reference/top/volume-2-image-processing/image-color-conversion/color-gray-scale-conversions/cfatorgb.html  Also, fwiw, I did some comparisons of phenotypes based on different demosaicing methods from opencv and never noticed a significant difference. There was some discussion of this on twitter too and basically it seems if they all have similar issues than everything compensates. There may be absolute differences between plants but it mostly appears to come out in the wash when comparing relative differences.

…

________________________________ From: Noah Fahlgren <notifications@github.com> Sent: Tuesday, September 15, 2020 10:30:30 PM To: danforthcenter/plantcv <plantcv@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [danforthcenter/plantcv] Image Demosaicing Quality in PlantCV versus LemnaBase (#624) Hi @AdamDimech<https://urldefense.com/v3/__https:/github.com/AdamDimech__;!!JmPEgBY0HMszNaDT!-0FaEFEEEw2qMUL-Dc0ZKKl1ENPoq6-GV5wuPdtkEPDQUIBVjGTh8uY3upTcJ0wLVj1Mlg$>! The blob files (e.g. blob49) on our system are zip files containing a file called data. Our data files are raw 8-bit binary files. The data encoded in the raw image file is the raw camera sensor data, which is a Bayer filter<https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Bayer_filter__;!!JmPEgBY0HMszNaDT!-0FaEFEEEw2qMUL-Dc0ZKKl1ENPoq6-GV5wuPdtkEPDQUIBVjGTh8uY3upTcJ0yz_zcGjw$>. In OpenCV there are a lot of Bayer demosaicing methods but they can be grouped into some basic sets. First, you could ignore all the COLOR_BayerXX2RGB methods as OpenCV functions (and PlantCV since we are using OpenCV primarily) want the images in BGR order. Unless you use an alpha or transparency channel, all the COLOR_BayerXX2RGBA can be ignored. The COLOR_BayerXX2GRAY can also be ignored since you're trying to convert to a color image. All that's left are a few groups. The first part codes the Bayer filter pattern type (COLOR_BayerXX2BGR) and different sensors may have different patterns. The RG pattern has worked with our data. The COLOR_BayerXX2BGR methods use the default interpolation method, which I think is bilinear. The other two methods with the _VNG and _EG extensions do the same thing but use either the Variable Number of Gradients or Edge-Aware interpolation methods. When you use cv2.cvtColor it uses a demosaicing algorithm if provided one of the Bayer methods. I ran both to compare and I got the same results: img = cv2.cvtColor(raw_img, cv2.COLOR_BAYER_RG2BGR) imgd = cv2.demosaicing(raw_img, cv2.COLOR_BAYER_RG2BGR) (img == imgd).all() # True It looks like we need a method to specify the type of Bayer filter so the correct method can be used. I have not compared the Bilinear, VNG, or EA algorithm results but sounds like you are getting better results with VNG, which is interesting! I need to do some comparisons on our end too. LemnaTec might not be doing the conversion with OpenCV. They could be using a different interpolation algorithm and/or it could be an algorithm from the camera/camera sensor manufacturer that uses a proprietary method. It's also possible they might do some other preprocessing to tweak the color gains in the raw image before demosaicing. - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/danforthcenter/plantcv/issues/624*issuecomment-693180176__;Iw!!JmPEgBY0HMszNaDT!-0FaEFEEEw2qMUL-Dc0ZKKl1ENPoq6-GV5wuPdtkEPDQUIBVjGTh8uY3upTcJ0xQFVfvnA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ABY5SZJ6EJ5ILTYEYO2IUJ3SGBEPNANCNFSM4ROFJXOA__;!!JmPEgBY0HMszNaDT!-0FaEFEEEw2qMUL-Dc0ZKKl1ENPoq6-GV5wuPdtkEPDQUIBVjGTh8uY3upTcJ0ySP_lFsg$>.

[dschneiderch]

one of these algorithms might give better results https://github.com/colour-science/colour-demosaicing

a quick look through github and it appears that these are the most commmonly used in python code

[nfahlgren]

FWIW I ran a raw image through the three methods in colour-demosaicing. The bilinear algorithm gave visually similar results to the OpenCV bilinear method. The Malvar and Menon methods also looked visually similar to me, but they both returned some funky spots around what I think are completely saturated pixels.

[dschneiderch]

not sure images with large white areas are a great example...? It would be interesting to see if the edge aware captures that edge better. I could only find c++ versions of the intel sak98 algorithm although I guess it'e be an interesting comparison

but again, downstream I don't think ti will matter much.

[nfahlgren]

Structural similarity index comparisons:
OpenCV bilinear vs colour-demosaicing bilinear: 0.99711175070708258
OpenCV bilinear vs OpenCV EA: 0.99442848407307005
OpenCV bilinear vs OpenCV VNG: 0.82513981615385668

[dschneiderch]

did/could you do the others from colour? now i'm curious :-)

[nfahlgren]

For sure!

OpenCV bilinear vs colour-demosaicing Malvar: 0.93852859491338958
OpenCV bilinear vs colour-demosaicing Menon: 0.90777965142929185

[dschneiderch]

so which one is "best"? 😂

[dschneiderch]

It looks like we need a method to specify the type of Bayer filter so the correct method can be used. I have not compared the Bilinear, VNG, or EA algorithm results but sounds like you are getting better results with VNG, which is interesting! I need to do some comparisons on our end too.

fyi pcv.read_bayer() allows you to choose any of the 3 opencv methods so you could import plantcv to the extractor if thats easier

[nfahlgren]

I guess I really need a matrix of SSIM comparisons :)

[nfahlgren]

pcv.readbayer reads a grayscale Bayer filter image file at the moment. We could plug it into the process if we convert the data file to one of these images first and save it as a file. We could update the function to take in a raw data NumPy array instead and have a separate function read in the data from a file? Right now what pcv.readbayer reads in could technically be read by pcv.readimage, minus the demosaicing of course.

[dschneiderch]

oh, right. 🤐

[AdamDimech]

Hi @nfahlgren, thanks for that valuable information about the blob files. Both yours and our blob files seem to be the same.

@dschneiderch I don't think it was making any real difference to our results either, but we have a particular application now where we need a better-quality demosaicing which is why this issue has come up.

Our default in LemnaGrid is AHD, but here is some information reprinted from the software that I noticed:

Technical description 
This device supports demosaicing of the following raw image patterns. 

GreyScale8Bit
BaslerBayerBG8
plain PNG
BaslerBayerBG12
GreyScale12Bit
GreyScale14Bit
GreyScale16Bit
KappaRGB888 

The parameter GreyToRGB conversion can be used to convert a grey scale image with a value depth up to 24 bits/pixel into a RGB image with the R values representing the most significant bits and blue the least significant.

I have had a bit more of a play around with all of this and discovered a Python module called colour-demosaicing. I downloaded this, installed the module and amended LT-db-extractor.py accordingly:

import colour
from colour.plotting import *
from colour_demosaicing import *

...

img = demosaicing_CFA_Bayer_Menon2007(raw_img, db['bayer'])

I also amended the config file to have a parameter called "bayer" which includes the Bayer grid configuration (which is GBRG in our case), hence db['bayer'].

I think you'll agree that it certainly improves the image quality. As @dschneiderch would appreciate, the colour-demosaicing package uses CFA (Colour Filter Array) demosaicing algorithms.

I'll think we'll incorporate this into our (Data Science Tools) PlantCV workflow, but you may want to consider using it too?

I haven't updated my public-facing (year-old) fork of LT-db-extractor.py with all the changes I've made since but I think it would be good for us to combine/compare our various changes with a view to improving Data Science Tools for general use.