Color distorted on ov5647 (AEGHB-907)

[mutatrum]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

Proper color reproduction. I have a test image, picture takes with my phone:

Colors are good when running f.e. the simple_video_server_example:

Picture is flipped, but color reproduction is correct.

Actual behavior (suspected bug)

I'm trying to capture frames and show them on an LVGL 9 RGB565 canvas, on an ST7789 display. I push the frame to the canvas, I get the following result:

The display is only 320x240, and this is unscaled so it's a small crop of the frame buffer.

Error logs or terminal output

No response

Steps to reproduce the behavior

I've takes the setup code from the esp_video_server_example, set up the capture format:

    struct v4l2_format format = {
        .type = V4L2_BUF_TYPE_VIDEO_CAPTURE,
        .fmt.pix.width = 800,
        .fmt.pix.height = 800,
        .fmt.pix.pixelformat = V4L2_PIX_FMT_RGB565,
    };

    if (ioctl(fd, VIDIOC_S_FMT, &format) != 0) {
        ESP_LOGE(TAG, "failed to set format");
        goto exit_0;
    }

Reading back the format shows it's set up as RGB565.

The deviation from the video_server_example is the capture loop, instead of reading the fd and push it into a JPEG converter, I push the frame directly onto an lv_image:

    lv_image_dsc_t img_dsc = {
        .header = {
            .magic = LV_IMAGE_HEADER_MAGIC,
            .cf = LV_COLOR_FORMAT_RGB565,
            .w = 800,
            .h = 800,
        }
    };

    while (1) {
        memset(&buf, 0, sizeof(buf));
        buf.type   = type;
        buf.memory = MEMORY_TYPE;

        esp_err_t res = ioctl(video_fd, VIDIOC_DQBUF, &buf);
        if (res != 0) {
            ESP_LOGE(TAG, "failed to receive video frame");
            break;
        }

        lvgl_port_lock(0);

        img_dsc.data_size = buf.bytesused;
        img_dsc.data = buffer[buf.index];
        lv_image_set_src(img1, &img_dsc);

        lv_refr_now(NULL);

        lvgl_port_unlock();

        if (ioctl(video_fd, VIDIOC_QBUF, &buf) != 0) {
            ESP_LOGE(TAG, "failed to free video frame");
        }
   }

Project release version

0.7.0

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Linux

Operating system version

Ubuntu 24.04

Shell

Bash

Additional context

ESP32P4 Function Evaluation Board chip revision: v0.1

ESP-IDF v5.4-dev-4291-g7a305c0284-dirty 2nd stage bootloader

Dependencies:

[1/9] espressif/cmake_utilities (0.5.3)
[2/9] espressif/esp_cam_sensor (0.7.0)
[3/9] espressif/esp_h264 (1.0.4)
[4/9] espressif/esp_ipa (0.1.0)
[5/9] espressif/esp_lvgl_port (2.4.2)
[6/9] espressif/esp_sccb_intf (0.0.4)
[7/9] espressif/esp_video (0.7.0)
[8/9] lvgl/lvgl (9.2.2)
[9/9] idf (5.5.0)

Maybe similar to #8 ?

The only setting in sdkconfig.default is CONFIG_CAMERA_OV5647=y, the rest is not altered from default value.

[mutatrum]

To rule out the display side, I've added a few coloured squares directly into the buffer, which seems to work. Colors are reproduced correctly:

I've tried byte endian swap, bit endian swap, but no luck getting the output right. If I read through the esp_video examples, which all seem to product correct color reproduction, I don't see anything to suggest endianness issues or other pixel layouts, but maybe I'm missing something.

[WangYuxin-esp]

This is a known issue that is currently being fixed. I believe there will be a new MR this week to solve this issue.

[mutatrum]

Thank you, looking forward to the update. When it's released I'll test it and report back to close the issue.

I also wanted to say thanks, these components are an impressive feat and makes handling camera operations so much more feasible!

[mutatrum]

I might be impatient and it's not complete, but I just tested 6f07326 and the color distortion looks the same. This is with MIPI_2lane_24Minput_RAW8_800x800_50fps. I tried changing the bayer format, but they all looked the same, so maybe something else is still missing.

I tested all formats, the RAW8 are all the same, the RAW10 formats produce complete noise.

While writing this, I did some more testing and tried to set V4L2_CID_HFLIP back to 0. That fixed the colors! HFLIP (and I assume VFLIP as well) flip the picture before the bayer transform, so the format goes from GBRG to something else. Logically would be BGGR, but that doesn't yield correct results. I tried all the other bayer formats, but all of them have distorted colors with HLIP=1, so there might be something else interfering with the demosaicing.

Now it also makes sense why the demos worked, as they do not flip the image!

[mutatrum]

A second thing I noticed with this change is that it looks like the auto-exposure is turned off on some formats. It's very easy to completely blow out the picture by adding a light.

[WangYuxin-esp]

Yes, the installation direction of the sensor lens affects the configuration of some ISPs.
To facilitate problem identification, you can take a 24 color chart to help solve the problem.

[mutatrum]

It's easy to reproduce with esp_video_server by adding

    struct v4l2_ext_controls controls;
    struct v4l2_ext_control control[1];

    controls.ctrl_class = V4L2_CTRL_CLASS_USER;
    controls.count      = 1;
    controls.controls   = control;
    control[0].id       = V4L2_CID_HFLIP;
    control[0].value    = 1;
    if (ioctl(fd, VIDIOC_S_EXT_CTRLS, &controls) != 0) {
        ESP_LOGW(TAG, "failed to mirror the frame horizontally and skip this step");
    }    

to the end of app_video_open. That yields this result:

With value 0 (no HFLIP) this is the result:

I'll also test with the SC2336 camera.

[WangYuxin-esp]

Yes, if flip is enabled, this may require updating its bayer fmt. Because this flip feature comes from the sensor, it affects the Bayer format of RAW format.
If you really need the flip function, you can customize a register list at the application layer and use it through cmd VIDIOC_S_SENSOR_FMT.

[Horion0415]

To rule out the display side, I've added a few coloured squares directly into the buffer, which seems to work. Colors are reproduced correctly:

I've tried byte endian swap, bit endian swap, but no luck getting the output right. If I read through the esp_video examples, which all seem to product correct color reproduction, I don't see anything to suggest endianness issues or other pixel layouts, but maybe I'm missing something.

Hello, I noticed that the LCD you are using is the ST7789. For SPI interface displays like this, it is often necessary to swap the two bytes of the RGB565 format. In LVGL v8, this feature was available as an optional configuration (as shown in the image). However, since you are using LVGL v9, this configuration has been removed, and you need to manually call the lv_draw_sw_rgb565_swap interface to handle byte swapping. Therefore, I believe the color abnormality in this case is caused by the byte order issue. As for the mirror effect, it needs to be addressed on the camera sensor side.

[WangYuxin-esp]

The new submission allows opening IPA to load default ISP configuration.

[mutatrum]

Hello, I noticed that the LCD you are using is the ST7789. For SPI interface displays like this, it is often necessary to swap the two bytes of the RGB565 format. In LVGL v8, this feature was available as an optional configuration (as shown in the image). However, since you are using LVGL v9, this configuration has been removed, and you need to manually call the lv_draw_sw_rgb565_swap interface to handle byte swapping. Therefore, I believe the color abnormality in this case is caused by the byte order issue. As for the mirror effect, it needs to be addressed on the camera sensor side.

Color reproduction with LVGL 9.2 is good for all other screens, including when the camera is not swapped. I'm using esp_lvgl_port2.4.3, which has swap_bytes` as a setup parameter:

This also shows on the picture with the small RGB squares, they come out as intended. If I disable swap_bytes, there is a lot of color banding, as high and low significant bits are mangled.

[mutatrum]

Yes, if flip is enabled, this may require updating its bayer fmt. Because this flip feature comes from the sensor, it affects the Bayer format of RAW format. If you really need the flip function, you can customize a register list at the application layer and use it through cmd VIDIOC_S_SENSOR_FMT.

IMO horizontal flip is a mandatory feature, as it's usage is determined which way the camera is facing. If the camera is pointing the same way as the screen (e.g. a 'selfie' mode), the video doesn't need to be flipped, and the image makes sense as if you're looking into a mirror. If the camera is pointing through the screen (by lack of better terminology), the video needs to be flipped, otherwise everything is mirrored where it shouldn't be.

I have the camera facing away from the screen and everything is mirrored:

I hope this makes sense. I can try to fix the bayer demosaic for this sensor, but that would negate the tremendous advantage of the esp_video components with support for so many different sensors.

Addition: you can test this with your phone. If you use the selfie camera the preview is mirrored. If you use the back camera the preview is not mirrored.

[WangYuxin-esp]

Yes, the mirroring function affects the format of Bayer. Regarding the issue of sensor initialization, we made a bold attempt to allow the initialization list to be placed in the application layer code instead of always relying on the initialization list in esp-video.

You can customize an initialization list and modify the bayer fmt in the structure describing the initialization list.

[mutatrum]

If the bayer format should be adjusted, that wouldn't happen in the sensor initialisation, but in the ISP pipeline, or am I mistaken? I've seen the ISP controller, but I'm not sure how to go about that to adjust the bayer demosaic configuration, or I haven't found an example of this yet.