Packed Formats - How Much Effort Required?

[gittb]

Is your feature request related to a problem? Please describe.
I am working with a camera from SVS-VISTEK, the camera supports bayer12packed.
Camera

• The harvester package seems much more appealing to use as the camera is complaint with genIcam than using manufactures C++ SDK.

• While it also supports 8 bit output, I would really prefer not to lose the bit depth as it is very important in my use case

I am curious how much work would be required to make this happen. I am happy to contribute.

Describe the solution you'd like
Support for packed 12 & 14 bit bayer formats.

Additional context
I have found this method from stack overflow using numpy to split 2 8 bit bytes into 3 12 bit elements quite effective. This could be a possible simple solution:

https://stackoverflow.com/a/51967333

[gittb]

Additional questions:
If it would be too much effort to add packed formats to the project:

• Would it be possible to enable access to the raw data that make up the image from the buffer and users could then process into whatever they desire?

• Would it be possible to force buffer.payload... to read the data as a forced type such as unit8? If the camera is set to BayerRG12Packed, yet the data is read in as unit8 I can transform it into the correct 12 bit pattern I desire.

Either of these could potentially make the package viable for odd use cases where cameras output a strange format. (Like packed)

I also noticed #35 could affect this.

The goal would be that buffer doesn't return a None (Currently what I am getting when setting the camera to BayerRG12Packed) Data in an improper format would be better than None :)

[kazunarikudo]

@gittb Hi, Ben. Thank you for your message and for the proposal. I see a huge benefit in it but could you wait for a while until I can take some time for that, please? I will write you back when the time is ready. /Kazunari

[gittb]

@kazunarikudo Kazunari, sounds great!

Please let me know if you need any assistance.

[kazunarikudo]

@gittb Hi. Thank you again for the hint. It looks very promising and I would like to support the packed formats in a seamless way. I will let you know once I made it. /Kazunari

[kazunarikudo]

@gittb Hi, I'm working on this now. To get the progress forward, could you check how BayerRG12Packed place each pixel data in a consecutive byte array? Does it pack data with no spacing as follows?

|MSB  LSB|M      L|M      L|M      L|
|1st Byte|2nd     |3rd     |4th     |
-------------------------------------
|7......0|3..0B..8|B......4|7......0|
|<-----L1|    L1<-|        |<-----L3|
|        |<-M2    |M2<-----|        |
-------------------------------------

Or does it pack data as follows?

|MSB  LSB|M      L|M      L|
|1st Byte|2nd     |3rd     |
----------------------------
|7......0|7......0|B..8B..8|
|<-----L1|        |    L1<-|
|.       |<-----M2|M2<-    |
----------------------------

Your suggested method looks to be assuming the latter case but I would like you to check the reality. I would appreciate it if you could show diagrams for 12bit and 14bit formats.

[gittb]

@kazunarikudo To be honest, I have been having trouble locating this information.

Here is what I am using in numpy, but I am not 100% certain it is correct. It does display the image well, but it shows the dynamic range as very small, with darks being very dark, and brights being very over exposed. My camera has a dynamic range of 72db, and I am not sure if this is caused by the camera or decoding the 12bitpacked improperly.

data = numpy.frombuffer(self._buffer.raw_buffer, dtype=np.uint8)
fst_uint8, mid_uint8, lst_uint8 = np.reshape(data, (data.shape[0] // 3, 3)).astype(np.uint16).T
fst_uint12 = (fst_uint8 << 4) + (mid_uint8 >> 4)
snd_uint12 = (lst_uint8 << 4) + (np.bitwise_and(15, mid_uint8))
self._data = np.reshape(np.concatenate((fst_uint12[:, None], snd_uint12[:, None]), axis=1), 2 * fst_uint12.shape[0])

While the above may be very close, I am not certain it is correct.

Would you like me to send you a buffer from the cameras output?

[gittb]

I did find a PDF from another manufacture speaking about GB10Packed, which has the layout as found below: https://drive.google.com/file/d/1_OhXPsnhvrGjWBg_Go0HrIKoCRiolJSp/view?usp=sharing

[kazunarikudo]

@gittb Hi. I have pushed the required changes to a branch called _issue_146. Could you try that when you can, please? I would appreciate it if you could try pixel formats as many as possible because I do not have any tangible camera here; it'd be very nice if you could tell me all the pixel formats that you tested. If you faced an issue please just let me know. /Kazunari

[kazunarikudo]

@gittb Hi. This is just a friendly reminder: I'm still waiting for your feedback. Thanks.

[gittb]

@kazunarikudo Sorry, work has had me extremely busy, I have time this weekend and I will test it out.

Thank you for all your help. I will post again soon.

[kazunarikudo]

@gittb Hi, how's it going?

[justinbacle]

@kazunarikudo I've been playing with harvesters for the last few weeks. And I am stuck on this issue as well. So sad that @gittb never answered.
I am trying to get BayerRG12p data from a Flir camera (model : BFLY-U3-23S6C-C). When asked for 16b bayer data everything is fine but it fails on 12bits packed data.
buffer.payload.components only holds a [None]

Here is a simplified code snippet.

from harvesters.core import Harvester
import cv2
import numpy as np
import config

h = Harvester()
h.add_cti_file("path_to_cti_file")
h.update_device_info_list()

ia = h.create_image_acquirer(0)
ia.remote_device.node_map.PixelFormat.value = "BayerRG12p"  # Format of 'BayerRG16' works fine, but not 12 bits packed
# ... Other node manipulation goes here...

ia.start_image_acquisition()

with ia.fetch_buffer(timeout=1) as buffer:
    component = buffer.payload.components[0]
    if component is not None:
        _2d = component.data.reshape(component.height, component.width)
        binImage = cv2.cvtColor(np.uint16(_2d), cv2.COLOR_BAYER_RG2RGB_EA)
    else:
        # FIXME Component is None when asked for 12b packed format
        ...

ia.stop_image_acquisition()
ia.destroy()
h.reset()

cv2.imshow('img', binImage)
cv2.waitKey(0)

Let me know if I can be of any help to help you debug that.

[gittb]

Hi everyone. I am back from the dead. I actually solved it and frankly forgot. After hitting the wall of not being able to run Harvesters on ARM I had to abandon the package. When I get back home I will post my code as well.

…

On Sat, Mar 13, 2021, 7:37 PM justinbacle ***@***.***> wrote: @kazunarikudo <https://github.com/kazunarikudo> I've been playing with harvesters for the last few weeks. And I am stuck on this issue as well. So sad that @gittb <https://github.com/gittb> never answered. I am trying to get BayerRG12p data from a Flir camera. When asked for 16b bayer data everything is fine but it fails on 12bits packed data. buffer.payload.components only holds a [None] Here is a simplified code snippet. from harvesters.core import Harvester import cv2 import numpy as np import config h = Harvester() h.add_cti_file("path_to_cti_file") h.update_device_info_list() ia = h.create_image_acquirer(0) ia.remote_device.node_map.PixelFormat.value = "BayerRG12p" # Format of 'BayerRG16' works fine, but not 12 bits packed # ... Other node manipulation goes here... ia.start_image_acquisition() with ia.fetch_buffer(timeout=1) as buffer: component = buffer.payload.components[0] if component is not None: _2d = component.data.reshape(component.height, component.width) binImage = cv2.cvtColor(np.uint16(_2d), cv2.COLOR_BAYER_RG2RGB_EA) else: # FIXME Component is None when camera is asked for 12b packed format ... ia.stop_image_acquisition() ia.destroy() h.reset() cv2.imshow('img', binImage) cv2.waitKey(0) Let me know if I can be of any help to help you debug that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#146 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GRTA77M6X5KAOK53B3M3TDQHORANCNFSM4MID6JNA> .

[kazunarikudo]

@justinbacle Hi, does even the latest master give you None for the pixel format? Regards, Kazunari.

[justinbacle]

I used the latest version in pypi so 1.2.8. is it different from the current master ?
I can check with the latest git master this evening 👌

[justinbacle]

Well my bad @kazunarikudo it looks like I had an old version of harvesters installed (becaused I tried Harvesters-gui which is not compatible with the latest harvesters).
I can get data correctly on the latest version 1.2.8. Thanks for your help :)
(Need to fix the image saving now, but should be okay 👍 )