Handling of 16-bit bayer Endianness

[schoolpost]

Trying to get the ClearHDR 16bit mode of an IMX585 working with Raspberry Pi 5 through libcamera.

Here is a capture from libcamera-hello with default viewfinder mode:

pi@imx585:~ $ libcamera-hello --framerate 24 --analoggain 1.0 --shutter 20833 --width 3856 --height 2180 -t 0 --awbgains 1.2,3.5 
[1:27:32.570571057] [4647]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+147-a409bd53-dirty (2023-12-14T00:22:27-07:00)
[1:27:32.571924478] [4648]  INFO RPI pisp.cpp:662 libpisp version v1.0.1 5652744a74d7 14-12-2023 (00:22:27)
[1:27:32.573327733] [4648]  WARN CameraSensorProperties camera_sensor_properties.cpp:261 No static properties available for 'imx585'
[1:27:32.573347048] [4648]  WARN CameraSensorProperties camera_sensor_properties.cpp:263 Please consider updating the camera sensor properties database
[1:27:32.577095715] [4648]  INFO RPI pisp.cpp:1119 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx585@37 to CFE device /dev/media0 and ISP device /dev/media3 using PiSP variant BCM2712_C0
Made DRM preview window
Mode selection for 1910:1080:12:P(24)
    SRGGB12_CSI2P,1928x1090/90.1713 - Score: 7.10831
    SRGGB12_CSI2P,3856x2180/50 - Score: 761.608
    SRGGB16,1928x1090/30.0003 - Score: 2007.11
    SRGGB16,3856x2180/30.0003 - Score: 2761.61
Stream configuration adjusted
[1:27:32.635195864] [4647]  INFO Camera camera.cpp:1183 configuring streams: (0) 1910x1080-YUV420 (1) 1928x1090-RGGB16_PISP_COMP1
[1:27:32.635293328] [4648]  INFO RPI pisp.cpp:1403 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx585@37 - Selected sensor format: 1928x1090-SRGGB12_1X12 - Selected CFE format: 1928x1090-PC1R

Here is a capture from libcamera-hello, with a forced 16-bit unpacked viewfinder mode:

pi@imx585:~ $ libcamera-hello --framerate 24 --analoggain 1.0 --shutter 20833 --width 3856 --height 2180 --viewfinder-mode 3856:2180:16:U -t 0 --awbgains 1.2,3.5 
[1:28:54.404346340] [4694]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+147-a409bd53-dirty (2023-12-14T00:22:27-07:00)
[1:28:54.405863763] [4695]  INFO RPI pisp.cpp:662 libpisp version v1.0.1 5652744a74d7 14-12-2023 (00:22:27)
[1:28:54.407285129] [4695]  WARN CameraSensorProperties camera_sensor_properties.cpp:261 No static properties available for 'imx585'
[1:28:54.407307185] [4695]  WARN CameraSensorProperties camera_sensor_properties.cpp:263 Please consider updating the camera sensor properties database
[1:28:54.411138816] [4695]  INFO RPI pisp.cpp:1119 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx585@37 to CFE device /dev/media0 and ISP device /dev/media3 using PiSP variant BCM2712_C0
Made DRM preview window
Mode selection for 3856:2180:16:U(24)
    SRGGB12_CSI2P,1928x1090/90.1713 - Score: 8036
    SRGGB12_CSI2P,3856x2180/50 - Score: 2000
    SRGGB16,1928x1090/30.0003 - Score: 6036
    SRGGB16,3856x2180/30.0003 - Score: 0
Stream configuration adjusted
[1:28:54.468039706] [4694]  INFO Camera camera.cpp:1183 configuring streams: (0) 1910x1080-YUV420 (1) 3856x2180-RGGB16_PISP_COMP1
[1:28:54.468141466] [4695]  INFO RPI pisp.cpp:1403 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx585@37 - Selected sensor format: 3856x2180-SRGGB16_1x16 - Selected CFE format: 3856x2180-PC1R

How can libcamera / libpisp be configured to handle this?

EDIT: or is this related to the compressed RAW format in some way?

[schoolpost]

Same issue noted here, but for the CM4/Pi4:
https://forums.raspberrypi.com/viewtopic.php?p=2123617#p2123617

[6by9]

Same issue noted here, but for the CM4/Pi4: https://forums.raspberrypi.com/viewtopic.php?p=2123617#p2123617

That was a misinterpretation of the captured data. The produced data should match https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/pixfmt-srggb16.html and has to be processed accordingly.

For this issue am I reading that when using --viewfinder-mode 3856:2180:16:U that the ISP is apparently incorrectly processing the data when producing the viewfinder image?
I don't believe we've had a 16bit sensor to test with. The fact that the format definition in cfe_fmts.h is missing a .csi_dt entry looks suspicious (it isn't defined in CSI2 v1.3, but is now defined as 0x2e or use MIPI_CSI2_DT_RAW16).

The FE is always producing compressed or 16bpc data, so it seems unlikely that the back end is misinterpreting the data.

[6by9]

#5806 includes a patch to add the csi_dt value for 16bit formats, but I need others to confirm whether that is needed or not. I have a suspicion that all CSI2 data (including metadata) will be going through the CFE without that.

[naushir]

Not sure if the csi_dt change will make a difference, but does no harm to try it out. I'm pretty sure the ISP Backend correctly reads in 16-bit Bayer as that's one of the normal modes of operation. Can you try asking for 16-bit compressed to 8-bit Bayer and see if that makes any difference? You can do this by using --viewfinder-mode 3856:2180:16:P as the command line argument.

Also, have you managed to verify this 3856x2180 16-bit mode is correctly streaming from the sensor? Possibly the easiest way to verify this is to use a CM4 and drive Unicam directly via v4l2-ctl and save the output Bayer frames to a file.

[naushir]

Any update on this?

[naushir]

Closing this now due to inactivity.... Feel free to reopen if this is still an issue.

[schoolpost]

Ok, now having time to revisit this; I'm able to better present the issue at hand; and hopefully there is a mechanism somewhere in this software/hardware stack to handle it:

the IMX585 has a clearHDR mode that combines pixel readings at 2 different gains within the same exposure into a single 16-bit frame.

Now that I am able to capture uncompressed 16-bit bayer frames, I have been able to confirm the issue has to do with endianness as originally suspected. Shown from the datasheet:

I can further confirm with a python script that will display the bayer raw as a 16-bit mono image, when I perform a byte swap the image is rendered correctly ( albeit possessing some artifacts / noise / clipping )

Unswapped:

Swapped:

Now outside of my python script, I can perform the same operation within a libcamera based app by doing:

		BufferWriteSync w(static_cast<RPiCamApp*>(&app), completed_request->buffers[app.RawStream()]);
		const std::vector<libcamera::Span<uint8_t>> mem = w.Get();
		uint16_t *ptr = (uint16_t *)mem[0].data();
		for (unsigned int i = 0; i < mem[0].size() / 2; ++i) {  
			uint16_t value = *ptr;  
			*ptr = (value >> 8) | (value << 8); 
			++ptr;  
		}

How to configure libcamera to interpret the big-endian data appropriately? can it be done via the Pi5 ISP via libpisp? Can it be done in software before it gets further down the ISP pipeline ( RGB pipe ) ?

As it stands the ISP / preview is rendered unusable:

EDIT:

if it's of any use, I have attached the SRGGB16 bayer raw frame. ( 7744 byte stride x 2180 rows )
output.zip

[naushir]

Unfortunately, there is no HW mechanism to endian swap directly in the ISP pipeline - we only deal with little endian for 16-bits.

Are you able to do an endian swap in the sensor itself? Or perhaps get the sensor to write out 14-bit RAW (losing 2-bits of precision) which ought to be handled correctly?

[njhollinghurst]

This seems to be a hardware issue, and is not something that can easily be fixed in libcamera.

• For Unicam (up to Raspberry Pi 4), RAW16 is simply not supported; it was added to CSI-2 after the chip was designed.
• For RP1 (Raspberry Pi 5) it was supposed to be supported, but an endian-swap was omitted from the hardware pipeline.

[schoolpost]

If not in hardware, are there any avenues in software for accomplishing this? I noticed there are already a number of format conversions happening with CPU and not on ISP when I investigated a couple months back: #93

libcamera/src/libcamera/pipeline/rpi/pisp/pisp.cpp

Line 321 in 075b54d

void do32BitConversion(void *mem, unsigned int width, unsigned int height,

Understandably those happen after the fact once the image is out of the ISP, It would need to perform a byte swap on the raw image while it is in memory before it is processed by the ISP

[naushir]

If not in hardware, are there any avenues in software for accomplishing this?

This might be possible, we will explore the options. However, you end up in a situation where you cannot rely on the statistics for the 3A algorithms. This may not be a problem if you want to control everything (exposure/gain/white balance) manually.

[schoolpost]

If not in hardware, are there any avenues in software for accomplishing this?

This might be possible, we will explore the options. However, you end up in a situation where you cannot rely on the statistics for the 3A algorithms. This may not be a problem if you want to control everything (exposure/gain/white balance) manually.

Yes,

My application is full manual control, losing 3A would not be a major concern.

Thank you for looking into this, I eagerly await a solution and hope to test soon.

[schoolpost]

Hi, following up to see if you've had the chance to look into this further.

thanks!

[naushir]

Not yet unfortunately. I will try to get to it next week if I can.

[naushir]

@schoolpost I've attempted to add the SW endian swap code here: 9c80dba (from the next tree).

Unfortunately, I can't really try it out properly since I don't have a 16-bit sensor. Would you be able to give it a try and let me know if it works. Note that you must run in fully manual mode as statistics will not be functional in this mode.

[schoolpost]

@schoolpost I've attempted to add the SW endian swap code here: 9c80dba (from the next tree).

Unfortunately, I can't really try it out properly since I don't have a 16-bit sensor. Would you be able to give it a try and let me know if it works. Note that you must run in fully manual mode as statistics will not be functional in this mode.

Thanks @naushir,

I have been able to test and can confirm that it does in fact seem to bring me closer to a working preview.

I'm only now realizing how many other ISP functions depend on the 3A for their function, because even though I set the:

• red/blue gains
• analog gain
• exposure time

colors and such are way off, maybe there are some things I'll need to address via a tuning.json file?

[naushir]

You are definitely going to have to switch off alsc in the config. In the tuning config file, under the "rpi.alsc section, set n_iter to 0.

[schoolpost]

Ok, so this is currently where I stand with preview:

I am simultaneously juggling whether or not certain artifacts are from the PiSP or from incorrect / malformed sensor configuration in the driver.

Getting some reasonable color is my first goal, any ideas for why I am getting such heavily magenta image? issue with the CCM?

I am using the same same tuning.json file and same gain, shutter speed and red/blue gains in between testing sensor modes, as demonstrated here with the 12-bit non clearHDR mode I get correct colors ( other then my white balance being set too warm )

[schoolpost]

@naushir can you comment on where in the FE process the byte swap is happening?

Like you mention before 3A would be broken still, so I imagine the byte swap is happening at the AXI point at the end not at the input?

Is it possible the FE processing is messing with the bayer pixels before the byteswap? i.e. DPC mis-identifying pixels and correcting? I notice a ton of speckly white noise in my RAW image...

Again I can only speculate at this point... I'd love to know your thoughts

[naushir]

Have you tried taking the captured 16-bit raw frame and processing it through something like dcraw? That would indicate if the RAW data from the sensor (+ the SW endian handling) is correct or not.

can you comment on where in the FE process the byte swap is happening?

The byte swap to correct the endiness is happening entirely in software after the FE has written the buffer to memory, so not in the hardware pipeline at all. If things are setup correctly (which I think is the case), the FE will not be doing any pixel processing at all, so the wrong endianness in the pipeline should not matter.

You could try disabling some ISP stages and see if that improves things. To do this, remove (or simply rename) the blocks in the json config file to disable the pipeline stages. For example, rename rpi.sharpness to xrpi.sharpness.

[naushir]

Also, what command are you using for the preview?

[njhollinghurst]

Byte swap is not happening in ISP-FE -- what is happening is that big-endian RAW16 data are being passed into ISP-FE input, which assumes little-endian. This is due to an omission from the specification of the preceding (CSI2AXI) block.

In order to write out the RAW16 data unchanged, it is important that Front End Decompand, BLA, DPC, Downscale and Compress stages are all disabled (or if BLA is enabled, it must be configured as a no-op; which I think is the default setup).

Edit; If you've changed the black level, make sure BLA is disabled here.

Software byte-swap now happens after ISP-FE output.

[schoolpost]

Byte swap is not happening in ISP-FE -- what is happening is that big-endian RAW16 data are being passed into ISP-FE input, which assumes little-endian. This is due to an omission from the specification of the preceding (CSI2AXI) block.

In order to write out the RAW16 data unchanged, it is important that Front End Decompand, BLA, DPC, Downscale and Compress stages are all disabled (or if BLA is enabled, it must be configured as a no-op; which I think is the default setup).

Edit; If you've changed the black level, make sure BLA is disabled here.

Software byte-swap now happens after ISP-FE output.

Turns out in addition to disabling BLA/BLC on the FE, there some peculiarities with bayer order with the this specific sensor mode. I am now able to get a proper ISP preview:

@naushir I am using a custom EGL/OpenGL app for preview.

[naushir]

Glad to see you got things working. However, I'm a bit curious on the details.

Turns out in addition to disabling BLA/BLC on the FE

How did you disable the FE BLA/BLC? Did you remove the rpi.black_level block in the config file? I would expect that if you set black_level in the rpi.black_level block to a value of 4096, BLA/BLC in the FE should not do anything and the pixels would go through the FE untouched. What value do you use in the config file?

[schoolpost]

I tried a variety of black levels in the tuning file, but all produced visible artifacts in the resulting image.

Anything below ~4000 would reveal strange white grain / speckle noise. At first I thought this may have been the FE DPC.

At & above 4096, the noise artifacts would be gone, but the image had a very clearly reduced dynamic range and crushed shadow region. So that didn't seem right either.

Wasn't until I turned of Bla / Blc via modifying the pisp.cpp in the libcamera ipa module that my suspicions were confirmed.

So now rpi.black_level has no effect whatsoever; which isn't ideal but it is a workaround for the moment.

[naushir]

At & above 4096, the noise artifacts would be gone, but the image had a very clearly reduced dynamic range and crushed shadow region. So that didn't seem right either.

So this implies that the actual black level of the sensor is not 4096 but higher. This in turn means that what you did to disable BLA/BLC is the only solution. If you remove the rpi.black_level does it also disable BLA/BLC for you? I would hope it does, and that should be the better solution than editing pisp.cpp to disable black level. If it does not, it's something for me to look at.