ov5640 Camera Support

[shmuelzon]

Hey,

First of all, thanks for publishing this. It's really very helpful!
I was wondering if there was any chance in getting the ov5640 camera to work with your image.
I saw that kernel v4.20.0 includes the relevant driver and I can hear the camera click when the driver is loaded, so something works, but I don't have a /dev/video0 entry available. In addition, I can see the following log messages during boot:

[    7.533164] ov5640 2-003c: Linked as a consumer to regulator.5
[    7.533246] ov5640 2-003c: Linked as a consumer to regulator.6
[    7.533384] ov5640 2-003c: Linked as a consumer to regulator.4
[    7.572154] ov5640 2-003c: ov5640_read_reg: error: reg=300a
[    7.577781] ov5640 2-003c: ov5640_check_chip_id: failed to read chip identifier
[    7.588902] ov5640 2-003c: Dropping the link to regulator.5
[    7.589038] ov5640 2-003c: Dropping the link to regulator.6
[    7.589112] ov5640 2-003c: Dropping the link to regulator.4

I saw a few entries in the DTB regarding the camera and relevant i2c definitions but still something is off.
If it helps, the BSP's kernel has this configuration.

Do you have any ideas or leads I can pursue to get the camera to work?

Thanks in advance!

[meisterbasti]

Does the Ubuntu Version with the Kernel version 5.3.11 include driver for the ov5640 camera ?

[avafinger]

@shmuelzon @meisterbasti

Sorry for the long delay to answer but i was expecting to find a way to use the sensor.
Unfortunately, the way the sun6i_csi and ov5640 are designed is not possible to use the sensor with the mainline kernel without modifications. I have tried without success.

[meisterbasti]

Is there any camera for the csi working with the BPI Zero and a Linux Kernel > 5.3 ?

[meisterbasti]

Any new infos ? Is there any camera for the csi working with the BPI Zero and Linux Kernel > 5.3 ?

[avafinger]

Sorry. only kernel 3.4.y at the moment. Not ready for the mainline.

[avafinger]

Here is the correct DTB with support for OV5640 in Kernel 5.6.13-server. This was made possible with a hint from Armbian forum.

The current ov5640 driver for the mainline kernel 5.x can work with frame 640x480, with some effort it can be changed.

Tested with

./mjpg_streamer -i "./input_uvc.so -d /dev/video1 -y 3 -r 640x480 -f 30 -q 90 -n" -o "./output_http.so -w ./www"
MJPG Streamer Version: svn rev: 
 i: Using V4L2 device.: /dev/video1
 i: Desired Resolution: 640 x 480
 i: Frames Per Second.: 30
 i: Format............: YUV - 0x59565955
 i: JPEG Quality......: 90
 o: www-folder-path...: ./www/
 o: HTTP TCP port.....: 8080
 o: username:password.: disabled
 o: commands..........: enabled

In case someone would like to try it, here is the DTB with support for OV5640. (Kernel 5.6.13-server)

bpi-m2-zero-v4.dtb_5.6.13-server.tar.gz

[software3daerospace]

@shmuelzon did you solve the issue? I have been having similar trouble. your solution might be handy for me

[shmuelzon]

Sadly, I never got around to try @avafinger's new DTB but it looks promising

[avafinger]

I will try to build the latest kernel with ov5640 support this weekend.

[software3daerospace]

@avafinger thank you
I am just using petalinux from xilinx and using ov5640 camera with zedboard to capture some images. I have similar issuse as him.

[avafinger]

You can test this: https://github.com/avafinger/bananapi-zero-ubuntu-base-minimal/releases/tag/v3.9

@software3daerospace
This is for the BPI M2Z board, your petalinux is not the same and you may need to find and adjust the correct values for your board in the ov5640 node.

Decompile the dtb, find the ov5640 node and see what you can transplant to your board.

[software3daerospace]

@avafinger I have node for the camera. The Mipi is not initializing properly and due to which the camera is not able to switch on

/include/ "system-conf.dtsi"
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/media/xilinx-vip.h>

/{
xlnk {
compatible = "xlnx,xlnk-1.0";
};
};

/{

model = "Zynq Zed Development Board";
compatible = "xlnx,zynq-zed", "xlnx,zynq-7000";

 usb_phy0: phy0@e0002000 {
    compatible = "ulpi-phy";
    #phy-cells = <0>;
    reg = <0xe0002000 0x1000>;
    view-port = <0x0170>;
    drv-vbus;
};

};

&v_tc_0 {
compatible = "xlnx,bridge-v-tc-6.1";
xlnx,pixels-per-clock = <1>;
};

&usb0 {
dr_mode = "host";
usb-phy = <&usb_phy0>;
};

&sdhci0 {
u-boot,dm-pre-reloc;
};

&uart1 {
u-boot,dm-pre-reloc;
};

&qspi {
u-boot,dm-pre-reloc;
};

&flash0 {
compatible = "micron,m25p80", "s25fl128s", "spi-flash";
};

&mipi_csi2_rx_subsyst_0 {
xlnx,vc = <0x4>;
csiss_ports: ports {
#address-cells = <1>;
#size-cells = <0>;
csiss_port0: port@0 {
reg = <0>;
xlnx,cfa-pattern = "rggb";
xlnx,video-format = <12>;
xlnx,video-width = <8>;
mipi_csi2_rx_0_to_demosaic_0: endpoint {
remote-endpoint = <&demosaic_0_from_mipi_csi2_rx_0>;
};
};
csiss_port1: port@1 {
reg = <1>;
xlnx,cfa-pattern = "rggb";
xlnx,video-format = <12>;
xlnx,video-width = <8>;
csiss_in: endpoint {
data-lanes = <1 2>;
remote-endpoint = <&ov5640_to_mipi_csi2>;
};
};
};
};
&v_demosaic_0 {

compatible = "xlnx,v-demosaic";
reset-gpios = <&gpio0 57 GPIO_ACTIVE_LOW>;
ports {
    #address-cells = <1>;
    #size-cells = <0>;
    port@0 {
        reg = <0>;
    xlnx,cfa-pattern = "rggb";
        xlnx,video-width = <8>;
        demosaic_0_from_mipi_csi2_rx_0: endpoint {
            remote-endpoint = <&mipi_csi2_rx_0_to_demosaic_0>;
        };
    };
port@1 {
        reg = <1>;
    xlnx,cfa-pattern = "rggb";
        xlnx,video-width = <8>;

        demosaic_0_to_fb: endpoint {
            remote-endpoint = <&vcap_in>;
        };
    };
   
};

};

/ {
misc_clk_0 {
#clock-cells = <0>;
compatible = "fixed-clock";
clock-frequency = <12000000>;
};
cam_reg_1v8: regulator-1v8 {
compatible = "regulator-fixed";
regulator-name = "1v8";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
gpio = <&gpio0 56 GPIO_ACTIVE_HIGH>;
startup-delay-us = <50000>;
enable-active-high;
};
cam_reg_2v8: regulator-1v8 {
compatible = "regulator-fixed";
regulator-name = "2v8";
regulator-min-microvolt = <2800000>;
regulator-max-microvolt = <2800000>;

};
cam_reg_1v5: regulator-1v8 {
	compatible = "regulator-fixed";
	regulator-name = "1v5";
	regulator-min-microvolt = <1500000>;
	regulator-max-microvolt = <1500000>;
};	

chosen {
bootargs = "earlycon console=ttyPS0,115200 loglevel=8 clk_ignore_unused root=/dev/ram rw";
};
};

&i2c0 {
clock-frequency = <100000>;
status = "okay";
i2cswitch@70{ /* U3 on FMC /
compatible = "ti,tca9546";
#address-cells = <1>;
#size-cells = <0>;
reg = <0x70>;
i2csw_0: i2c@0 { / cam 1 /
#address-cells = <1>;
#size-cells = <0>;
reg = <0>;
/ I2C0 /
};
i2csw_1: i2c@1 { / cam 2 /
#address-cells = <1>;
#size-cells = <0>;
reg = <1>;
/ I2C1 /
};
i2csw_2: i2c@2 { / cam 3*/
#address-cells = <1>;
#size-cells = <0>;
reg = <2>;
/* I2C2 /
};
i2csw_3: i2c@3 { / cam 4 /
#address-cells = <1>;
#size-cells = <0>;
reg = <3>;
/ I2C3 */
};
};
};
&i2csw_0 {
ov5640: camera@3c {
compatible = "ovti,ov5640";
reg = <0x3c>;
clocks = <&misc_clk_0 >;
clock-names = "xclk";
vdddo-supply = <&cam_reg_1v8>;
vdda-supply = <&cam_reg_2v8>;
vddd-supply = <&cam_reg_1v5>;
powerdown-gpios = <&gpio0 58 GPIO_ACTIVE_HIGH>;
reset-gpios = <&gpio0 59 GPIO_ACTIVE_LOW>;
port {
ov5640_to_mipi_csi2: endpoint {
remote-endpoint = <&csiss_in>;
clock-lanes = <0>;
data-lanes = <1 2>;
};
};
};
};

&v_frmbuf_wr_0 {
reset-gpios = <&gpio0 54 GPIO_ACTIVE_LOW>;
};

&csiss_in {
data-lanes = <1 2>;
remote-endpoint = <&ov5640_to_mipi_csi2>;
};

&csiss_port0 {
xlnx,video-format = <XVIP_VF_YUV_422>;
};

&csiss_port1 {
xlnx,video-format = <XVIP_VF_YUV_422>;
};

&amba_pl {
video_in: video_cap {
compatible = "xlnx,video";

    dmas = <&v_frmbuf_wr_0 0>;
   dma-names = "port0";

   ports {
        #address-cells = <1>;
        #size-cells = <0>;

         port@0 {
            reg = <0>;
            direction = "input";
            vcap_in: endpoint {
                remote-endpoint = <&demosaic_0_to_fb>;
            };
        };            
    };
};

};

[shmuelzon]

Hey @avafinger,

I'm trying out release v3.9 but I can't get it to work. It seems to communicate with the ov5640 fine:

ubuntu@bpi-m2z:~/mjpg-streamer/mjpg-streamer-experimental$ media-ctl -v --device /dev/media1 --known-mbus-fmts
RGB444_1X12                     0x1016
RGB444_2X8_PADHI_BE             0x1001
RGB444_2X8_PADHI_LE             0x1002
RGB555_2X8_PADHI_BE             0x1003
RGB555_2X8_PADHI_LE             0x1004
RGB565_1X16                     0x1017
BGR565_2X8_BE                   0x1005
BGR565_2X8_LE                   0x1006
RGB565_2X8_BE                   0x1007
RGB565_2X8_LE                   0x1008
RGB666_1X18                     0x1009
RBG888_1X24                     0x100e
RGB666_1X24_CPADHI              0x1015
RGB666_1X7X3_SPWG               0x1010
BGR888_1X24                     0x1013
BGR888_3X8                      0x101b
GBR888_1X24                     0x1014
RGB888_1X24                     0x100a
RGB888_2X12_BE                  0x100b
RGB888_2X12_LE                  0x100c
RGB888_1X7X4_SPWG               0x1011
RGB888_1X7X4_JEIDA              0x1012
ARGB8888_1X32                   0x100d
RGB888_1X32_PADHI               0x100f
RGB101010_1X30                  0x1018
RGB121212_1X36                  0x1019
RGB161616_1X48                  0x101a
Y8_1X8                          0x2001
UV8_1X8                         0x2015
UYVY8_1_5X8                     0x2002
VYUY8_1_5X8                     0x2003
YUYV8_1_5X8                     0x2004
YVYU8_1_5X8                     0x2005
UYVY8_2X8                       0x2006
VYUY8_2X8                       0x2007
YUYV8_2X8                       0x2008
YVYU8_2X8                       0x2009
Y10_1X10                        0x200a
Y10_2X8_PADHI_LE                0x202c
UYVY10_2X10                     0x2018
VYUY10_2X10                     0x2019
YUYV10_2X10                     0x200b
YVYU10_2X10                     0x200c
Y12_1X12                        0x2013
UYVY12_2X12                     0x201c
VYUY12_2X12                     0x201d
YUYV12_2X12                     0x201e
YVYU12_2X12                     0x201f
UYVY8_1X16                      0x200f
VYUY8_1X16                      0x2010
YUYV8_1X16                      0x2011
YVYU8_1X16                      0x2012
YDYUYDYV8_1X16                  0x2014
UYVY10_1X20                     0x201a
VYUY10_1X20                     0x201b
YUYV10_1X20                     0x200d
YVYU10_1X20                     0x200e
VUY8_1X24                       0x2024
YUV8_1X24                       0x2025
UYYVYY8_0_5X24                  0x2026
UYVY12_1X24                     0x2020
VYUY12_1X24                     0x2021
YUYV12_1X24                     0x2022
YVYU12_1X24                     0x2023
YUV10_1X30                      0x2016
UYYVYY10_0_5X30                 0x2027
AYUV8_1X32                      0x2017
UYYVYY12_0_5X36                 0x2028
YUV12_1X36                      0x2029
YUV16_1X48                      0x202a
UYYVYY16_0_5X48                 0x202b
SBGGR8_1X8                      0x3001
SGBRG8_1X8                      0x3013
SGRBG8_1X8                      0x3002
SRGGB8_1X8                      0x3014
SBGGR10_ALAW8_1X8               0x3015
SGBRG10_ALAW8_1X8               0x3016
SGRBG10_ALAW8_1X8               0x3017
SRGGB10_ALAW8_1X8               0x3018
SBGGR10_DPCM8_1X8               0x300b
SGBRG10_DPCM8_1X8               0x300c
SGRBG10_DPCM8_1X8               0x3009
SRGGB10_DPCM8_1X8               0x300d
SBGGR10_2X8_PADHI_BE            0x3003
SBGGR10_2X8_PADHI_LE            0x3004
SBGGR10_2X8_PADLO_BE            0x3005
SBGGR10_2X8_PADLO_LE            0x3006
SBGGR10_1X10                    0x3007
SGBRG10_1X10                    0x300e
SGRBG10_1X10                    0x300a
SRGGB10_1X10                    0x300f
SBGGR12_1X12                    0x3008
SGBRG12_1X12                    0x3010
SGRBG12_1X12                    0x3011
SRGGB12_1X12                    0x3012
SBGGR14_1X14                    0x3019
SGBRG14_1X14                    0x301a
SGRBG14_1X14                    0x301b
SRGGB14_1X14                    0x301c
SBGGR16_1X16                    0x301d
SGBRG16_1X16                    0x301e
SGRBG16_1X16                    0x301f
SRGGB16_1X16                    0x3020
JPEG_1X8                        0x4001
S5C_UYVY_JPEG_1X8               0x5001
AHSV8888_1X32                   0x6001

And I don't get an error when setting it to JPEG with

media-ctl --device /dev/media1 --set-v4l2 '"ov5640 3-003c":0[fmt:JPEG_1X8/1280x720]'

But running mjpg_streamer doesn't work:

ubuntu@bpi-m2z:~/mjpg-streamer/mjpg-streamer-experimental$ ./mjpg_streamer -i "./input_uvc.so -d /dev/video1 -r 1280x720 -q 90 -n" -o "./output_http.so -p 8080 -w ./www"
MJPG Streamer Version: git rev: 310b29f4a94c46652b20c4b7b6e5cf24e532af39
 i: Using V4L2 device.: /dev/video1
 i: Desired Resolution: 1280 x 720
 i: Frames Per Second.: -1
 i: Format............: JPEG
 i: TV-Norm...........: DEFAULT
 i: Could not obtain the requested pixelformat: MJPG , driver gave us: BA81
    ... will try to handle this by checking against supported formats.
Init v4L2 failed !! exit fatal
 i: init_VideoIn failed

I also tried 640x480 in case what you previously wrote was still valid but no luck.
Am I missing something?

Thanks!

[avafinger]

@shmuelzon

The fix is here: #56 (comment)

[avafinger]

@software3daerospace

the Mipi is not initializing properly

The sensor in bpi-m2z is a DVP and not Mipi, I don't know if you need anything else. In order to talk to the sensor, I2C bit-bang must be enabled in your kernel.
Have you tried with SDK to check if your sensor is working?

Try to format the DTS you posted, mark the start of the DTS and the end the DTS and click ctrl+e.

[avafinger]

Have you read this: https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/MicroZed-Chronicles-PetaLinux-Image-Processing-System/ba-p/1119412

[software3daerospace]

@avafinger yes, I have been looking for all the posts from forum. I just figured out that I2C switch of the camera is not setting right

[shmuelzon]

@avafinger

The fix is here: #56 (comment)

Nice, thanks!
Also, I needed to run media-ctl --device /dev/media1 --set-v4l2 '"ov5640 3-003c":0[fmt:JPEG_1X8/1280x720]' with sudo.
Can you explain why I need that in the first place?

BTW, I did have one issue with the new where the apt source list didn't work. It was pointing to archive.ubuntu.com/ubuntu where I needed to change it to ports.ubuntu.com/ubuntu-ports. Is this something you're aware of?

I do get a rather low frame-rate by default, I'll try to play around with it. Other than that, things are looking pretty good and I don't feel it running too hot yet, I'll let it work through the night and see if it's stable.

[avafinger]

Also, I needed to run media-ctl --device /dev/media1 --set-v4l2 '"ov5640 3-003c":0[fmt:JPEG_1X8/1280x720]' with sudo.
Can you explain why I need that in the first place?

This is to set the sensor to deliver JPEG (compressed images), this releases the CPU to compress YUV to JPEG.

archive.ubuntu.com/ubuntu where I needed to change it to ports.ubuntu.com/ubuntu-ports. Is this something you're aware of?

Ouch, indeed, it should be ports.ubuntu.com/ubuntu-ports everywhere in the file, will try to fix it.. Thanks.

[shmuelzon]

This is to set the sensor to deliver JPEG (compressed images), this releases the CPU to compress YUV to JPEG.

That part I understand, but I was under the impression that when specifying the pixel format in V4L, as with the change you suggested from mjpg_streamer, it would be enough.

Ouch, indeed, it should be ports.ubuntu.com/ubuntu-ports everywhere in the file, will try to fix it.. Thanks.

Except archive.canonical.com/ubuntu which seems to be fine.

[shmuelzon]

BTW - It seems the mjpg_streamer is taking 80-90% CPU, I though it would be much lower if it doesn't need to encode the image.

[avafinger]

is taking 80-90% CPU
I believe JPEG quality is somewhat around 80 or 90% and takes a lot of CPU to send it via wlan0.

[avafinger]

as with the change you suggested from mjpg_streamer, it would be enough.

This is not implemented yet in the mainline kernel, you need to tell the sensor what should be delivered.

[shmuelzon]

I believe JPEG quality is somewhat around 80 or 90% and takes a lot of CPU to send it via wlan0.

I don't think that's it. I tried sending a 50MB file via netcat over WiFi and it it took ~30% CPU for 17 seconds but what consumed the CPU was kworker/u8:0+brcmf_wq/mmc1:0001:1 which, despite the mmc in the name, I assume is the WiFi worker thread since I was working off a tmpfs when reading the file.
Also, mjpg_streamer consumed CPU even when no client is connected to it. I seem to recall it being ~15% on a Raspberry Pi Zero but I don't remember if that was while idle or streaming.

Maybe the camera isn't truly providing JPEGs or, for some reason, mjpg_streamer thinks it needs to transcode the image?
I guess that's out of your scope so, as far as I'm concerned, you can close this issue.

Thanks again for all you help with getting this working!

[avafinger]

I will investigate this issue, it may take some time... No timeline, maybe next weekend.. :)

[avafinger]

Also, mjpg_streamer consumed CPU even when no client is connected to it.

So, grabbing the frame from the sensor is taking CPU. Original mjpg_streamer is never idle, always grabbing frames, even without clients connected.

It would be interesting to have a look at the mjpg_streamer source code you use in Raspberry Pi Zero, can you provide the sources? I think there is no memory content transfer when grabbing frames, so not much CPU, let's see the code.

[avafinger]

I am not sure the driver has support for the DMA buffer...

[software3daerospace]

@avafinger I think it ssuports DMA buffer.

[shmuelzon]

I'm running mjpg_streamer from https://github.com/jacksonliam/mjpg-streamer on both on my Raspberry Pi Zero and on the Banana Pi Zero. On the RPi, it's on top of commit 501f6362c5afddcfb41055f97ae484252c85c912 and it consumes 1-2% CPU regardless if a client is connected or not, so I guess it also keeps grabbing frames.