Improve Science Camera Image Quality

[kbrowne15]

Copying over Trey's initial email/thoughts:

I was thinking again about how SciCam image quality is one of the key issues for our close-up inspections ... Something that might be easier is avoiding JPEG compression artifacts. We could almost certainly save the color image as lossless PNG (modulo PNG compression would take longer and result in higher data volume saved on the HLP, would need to assess if we could handle that). I’ve spent a lot of time staring at zoomed-in-too-far SciCam images and can confirm there are a lot of JPEG artifacts.

A more intriguing possibility is saving RAW format images. It looks like the Android Camera2 API supports saving RAW images in DNG format, if the camera supports it. I tried to find more information about the SciCam’s Sony IMX 230 sensor… one spec sheet I found suggested it might be natively producing images with 10 bits per pixel. If so, having access to that extra bit depth could be especially helpful if we are trying to both zoom way in and contrast stretch images with poor illumination. Note that it might not be possible to save RAW at other than full resolution, in which case we would definitely need to double-check we could handle the data volume.

[kbrowne15]

Programmatically, the camera supports the following formats:

• RAW_SENSOR (32)
• JPEG (256)
• PRIVATE (34)
• YUV_420_888 (35)
• RAW_PRIVATE (36)
• RAW10 (37)

More information on formats can be found here

The only size supported by RAW10 is the full size images.

[kbrowne15]

Further Trey thoughts:

My initial thought is it would be good to:

• Set the image format to be a run-time configuration option between JPEG and RAW10, defaulting to RAW10. I don’t fully understand how DNG wrapper files work, but that seems to be the way to go for Android RAW image data.
• Output the part of the Android CameraDevice information relevant for RAW10 data interpretation in some way (like console output). Otherwise the Bayer pattern is annoying to figure out.
• Verify we can decode the RAW images on the ground before merging the change. (I have no doubt we can eventually figure out decoding as long as the saved images aren’t garbage. Unfortunately, I don’t trust the saved images to not be garbage, between questionable quality of the camera driver and lack of clarity on how to use the API.)

Here’s some context logic:

• I think support for RAW10 probably implies > 8 bit depth per color channel, as we suspected, so immediately that suggests a benefit to working with RAW images in low light, if nothing else.
• On RAW10 vs. RAW_SENSOR or RAW_PRIVATE: I assume these would all contain identical data, just possibly with different byte-order/padding. Among them, RAW10 is densely packed, which should equate to marginally smaller files.
• On default format: Default to RAW to minimize the chance of accidentally degrading image data due to a legacy config file that doesn’t specify.

[kbrowne15]

I decide to try RAW10 since it seemed like an easy change. However, when I ran it, an exception occurred when creating the capture builder. I didn't look into the error but my theory is Android needs a surface to display the image on and it is probably unable to display the raw image.

[marinagmoreira]

seems like this is not an easy change. closing this issue for now and possibly re-opening this later in the future