Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to enable platforms with ov01a1s or hm11b1 sensors on TGL #19

Closed
vicamo opened this issue May 20, 2022 · 41 comments
Closed

failed to enable platforms with ov01a1s or hm11b1 sensors on TGL #19

vicamo opened this issue May 20, 2022 · 41 comments

Comments

@vicamo
Copy link
Contributor

vicamo commented May 20, 2022

Differences from media-ctl -p output of oem-5.14 kernel to oem-5.17 gives:

--- media-ctl.5.14.0-1036-oem.ANDW-DVT2-C1.Vicamo-TGL-ov01a1s.log       2022-05-18 00:49:15.849439919 +0800
+++ media-ctl.5.17.0-9004-oem.ANDW-DVT2-C1.Vicamo-TGL-ov01a1s.log       2022-05-18 00:49:15.869440701 +0800
@@ -1,4 +1,4 @@
-Media controller API version 5.14.21
+Media controller API version 5.17.3
 
 Media device information
 ------------------------
@@ -7,7 +7,7 @@
 serial          
 bus info        pci:pci0000:00
 hw revision     0x0
-driver version  5.14.21
+driver version  5.17.3
 
 Device topology
 - entity 1: Intel IPU6 CSI-2 0 (2 pads, 1 link)
@@ -19,12 +19,11 @@
                [fmt:Y10_1X10/4096x3072 field:none]
                -> "Intel IPU6 CSI2 BE SOC 0":0 []
 
-- entity 4: Intel IPU6 CSI-2 1 (2 pads, 2 links)
+- entity 4: Intel IPU6 CSI-2 1 (2 pads, 1 link)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:Y10_1X10/4096x3072 field:none]
-               <- "ov01a1s 20-0036":0 []
        pad1: Source
                [fmt:Y10_1X10/4096x3072 field:none]
                -> "Intel IPU6 CSI2 BE SOC 0":0 []
@@ -107,10 +106,3 @@
        pad0: Sink
                <- "Intel IPU6 CSI2 BE SOC 0":1 [DYNAMIC]
 
-- entity 68: ov01a1s 20-0036 (1 pad, 1 link)
-             type V4L2 subdev subtype Sensor flags 0
-             device node name /dev/v4l-subdev9
-       pad0: Source
-               [fmt:SGRBG10_1X10/1296x798 field:none]
-               -> "Intel IPU6 CSI-2 1":0 []
-

It seems some device is no longer probed/created.

The following logs were captured using ipu6/ivsc DKMS from vicamo/ppa-1972106 and vicamo/ppa-1972109.

@hao-yao
Copy link
Contributor

hao-yao commented May 26, 2022

ANDW-DVT2-C3: (TGL, ov01a1s)

Seems kerner driver is OK. We need more info to locate the issue.
@vicamo , could you stop the v4l2-relayd service, try this command and post the log?
sudo -E cameraDebug=3 gst-launch-1.0 icamerasrc device-name=ov01a1s-uf ! video/x-raw,format=YUY2,width=1280,height=720 ! videoconvert ! ximagesink

@vicamo
Copy link
Contributor Author

vicamo commented May 26, 2022

Seems kerner driver is OK. We need more info to locate the issue. @vicamo , could you stop the v4l2-relayd service, try this command and post the log? sudo -E cameraDebug=3 gst-launch-1.0 icamerasrc device-name=ov01a1s-uf ! video/x-raw,format=YUY2,width=1280,height=720 ! videoconvert ! ximagesink

@hao-yao
Copy link
Contributor

hao-yao commented May 26, 2022

In this log, seems kernel driver failed again. Would you try to create /etc/modprobe.d/blacklist-int3472.conf and add: blacklist int3472-discrete then attach a dmesg log of the same device again?

@vicamo
Copy link
Contributor Author

vicamo commented May 26, 2022

@hao-yao
Copy link
Contributor

hao-yao commented May 27, 2022

There is no such module 'int3472-discrete', do you mean intel-skl-int3472?

Seems i2c transfer failed. It is weird that 5.14.0-oem is OK but 5.15-generic fails while 5.15-generic OK on my device.

@ccgjimlai
Copy link
Contributor

ccgjimlai commented May 27, 2022

Hi Vicamo,
Could you also provide the oem5.14 debs which could work on your device?
and also, could you let us know your test result when you update your BIOS to 1.1.2?

@hao-yao
Copy link
Contributor

hao-yao commented May 30, 2022

Hi @vicamo , could you try to get the MCU firmware version on the failed device?
cat /sys/bus/usb/devices/usb3/3-6/3-6:1.0/version
My device is 0.8.0.8.

@vicamo
Copy link
Contributor Author

vicamo commented May 30, 2022

Hi @vicamo , could you try to get the MCU firmware version on the failed device? cat /sys/bus/usb/devices/usb3/3-6/3-6:1.0/version My device is 0.8.0.8.

  • ANDW-DVT2-C1.Vicamo-TGL-ov01a1s.BIOS-1.10.0: 0.6.0.42
  • ANDW-DVT2-C3.KC-TGL-ov01a1s.BIOS-1.4.3: 0.8.0.16

@hao-yao
Copy link
Contributor

hao-yao commented May 30, 2022

  • ANDW-DVT2-C1.Vicamo-TGL-ov01a1s.BIOS-1.10.0: 0.6.0.42
  • ANDW-DVT2-C3.KC-TGL-ov01a1s.BIOS-1.4.3: 0.8.0.16

According to the logs here, we can see on KC's DVT2-C3 ipu6 kernel driver works on 5.15, but fails when using cheese. On Vicamo's DVT2-C1, kernel driver failed at i2c reading with kernel 5.15, but OK on 5.14.

I think this should be 2 different issues. On KC's DVT, I think we can stop the v4l2-relayd service try try this command sudo -E cameraDebug=3 gst-launch-1.0 icamerasrc device-name=ov01a1s-uf ! video/x-raw,format=YUY2,width=1280,height=720 ! videoconvert ! ximagesink and save the log. And it is better to open a new issue in ipu6-camera-hal repo.

As for Vicamo's DVT kernel issue, I noticed that 5.14.0-1036-oem is OK but 5.15.12-for-ubuntu fails on the same device, while both build are OK on my side. So far I think this could be caused by hardware or firmware issue, especially MCU firmware. I will check this with MCU team.

@vicamo , could you help to check if anything wrong?

@ccgjimlai
Copy link
Contributor

Hi Vicamo, Could you also provide the oem5.14 debs which could work on your device? and also, could you let us know your test result when you update your BIOS to 1.1.2?

@vicamo do you have any feedback for these two?

@vicamo
Copy link
Contributor Author

vicamo commented May 31, 2022

Hi Vicamo, Could you also provide the oem5.14 debs which could work on your device?

You may find experimental kernels for oem projects in https://launchpad.net/~canonical-hwe-team/+archive/ubuntu/intermediate-kernel. Add that PPA with:

$ sudo add-apt-repository ppa:canonical-hwe-team/intermediate-kernel

Click the link View package details, click the triangle next to linux-oem-5.14 in the new page, then it will give all the links to the built packages. The ones you'll need are:

and also, could you let us know your test result when you update your BIOS to 1.1.2?

I don't see a reason to try an outdated BIOS version. We have failures with v1.4.3 and v1.10.0 already.

@ccgjimlai
Copy link
Contributor

Hi Vicamo, Could you also provide the oem5.14 debs which could work on your device?

You may find experimental kernels for oem projects in https://launchpad.net/~canonical-hwe-team/+archive/ubuntu/intermediate-kernel. Add that PPA with:

$ sudo add-apt-repository ppa:canonical-hwe-team/intermediate-kernel

Click the link View package details, click the triangle next to linux-oem-5.14 in the new page, then it will give all the links to the built packages. The ones you'll need are:

and also, could you let us know your test result when you update your BIOS to 1.1.2?

I don't see a reason to try an outdated BIOS version. We have failures with v1.4.3 and v1.10.0 already.

Hi @vicamo
The information is now confusing, and the error symptom is difference between KC's and your device.

Let's focus on your device now. In our previous sync meeting, you have mentioned that you were using BIOS 0.9, do you mean that you have already upgraded your device to BIOS 1.10?

Please use that device and do the following test

  1. install 5.14 debs
  2. open camera
  3. save dmesg logs
  4. print MCU fw version
  5. dump dsdt
  6. install 5.15 deb and repeat steps 2-5

@vicamo
Copy link
Contributor Author

vicamo commented May 31, 2022

@vicamo
Copy link
Contributor Author

vicamo commented May 31, 2022

How do I upgrade MCU fw?

@hao-yao
Copy link
Contributor

hao-yao commented May 31, 2022

Hi @vicamo , would you try the 5.15.12-for-ubuntu kernel image we sent you before? And also the devices info at /sys/bus/acpi/devices and /sys/bus/i2c/devices.

How do I upgrade MCU fw?

We will try to send you a MCU FW and flash guide if you can make sure version 0.6.0.42 is on your device (BIOS=1.10). It is weird that the latest BIOS contains an very old FW.

@vicamo
Copy link
Contributor Author

vicamo commented May 31, 2022

Hi @vicamo , would you try the 5.15.12-for-ubuntu kernel image we sent you before? And also the devices info at /sys/bus/acpi/devices and /sys/bus/i2c/devices.

See #19 (comment) and #19 (comment). 5.15.12-for-ubuntu is your kernel.

How do I upgrade MCU fw?

We will try to send you a MCU FW and flash guide if you can make sure version 0.6.0.42 is on your device (BIOS=1.10). It is weird that the latest BIOS contains an very old FW.

Maybe MCU fw is not included in BIOS updates.

We're also tracking an intermittent failure to probe ov01a1s running last few revisions of oem-5.14 kernels on both ANDW-DVT2-C1 and C3. It appears the sensor may sometimes not be probed, and following message is missing from dmesg:

intel-ipu6-isys intel-ipu6-isys0: bind ov01a1s 20-0036 nlanes is 1 port is 1

Last known good is 5.14.0-1027-oem, and roughly 50% failure rate in 1035, and 100% in 1038 and 1040.

@hao-yao
Copy link
Contributor

hao-yao commented Jun 1, 2022

See #19 (comment) and #19 (comment). 5.15.12-for-ubuntu is your kernel.

@vicamo , I noticed that your log from 5.15.12 kernel shows BIOS=89.5.23. Is this the same device as you tried 5.15.0-oem and 5.14.0-oem kernel on? Or you did a BIOS upgrade?

@vicamo
Copy link
Contributor Author

vicamo commented Jun 1, 2022

See #19 (comment) and #19 (comment). 5.15.12-for-ubuntu is your kernel.

@vicamo , I noticed that your log from 5.15.12 kernel shows BIOS=89.5.23. Is this the same device as you tried 5.15.0-oem and 5.14.0-oem kernel on? Or you did a BIOS upgrade?

BIOS upgrade. Both devices.

@ccgjimlai
Copy link
Contributor

How do I upgrade MCU fw?

Hi @vicamo please refer to the attached file and the steps below for upgrading to 0.8.0.9 fw

d21 fw download

  1. install dfu-util
    sudo apt install dfu-util

  2. update d21 fw

echo dfu > /sys/bus/usb/drivers/ljca/1-5\:1.0/cmd
dfu-util -v -R -D usb_bridge_dev_fw.dfu

I have also included fw version 0.6.0.42, which is what you have in your device now

d21_fw.zip

@vicamo
Copy link
Contributor Author

vicamo commented Jun 1, 2022

How do I upgrade MCU fw?

Hi @vicamo please refer to the attached file and the steps below for upgrading to 0.8.0.9 fw

d21 fw download

  1. install dfu-util
    sudo apt install dfu-util
  2. update d21 fw
echo dfu > /sys/bus/usb/drivers/ljca/1-5\:1.0/cmd
dfu-util -v -R -D usb_bridge_dev_fw.dfu

I have also included fw version 0.6.0.42, which is what you have in your device now

d21_fw.zip

I don't have /sys/bus/usb/drivers/ljca/1-5:1.0 but /sys/bus/usb/drivers/ljca/3-6:1.0. After flashed fw 0.8.0.9 on ANDW-DVT2-C1(Vicamo-TGL-ov01a1s), which had 0.6.0.42 previously, now doesn't load ljca driver on both oem-5.14 and 5.15 kernels. The usb device become:

Bus 003 Device 002: ID 04d8:0037 Microchip Technology, Inc. USB DFU

While on ANDW-DVT2-C3(KC-TGL-ov01a1s), it has:

Bus 003 Device 003: ID 8086:0b63 Intel Corp. USB Bridge

And ljca driver matches only 8086:0b63. Is that really a right fw to flash?

@ccgjimlai
Copy link
Contributor

Hi @vicamo it means that FW flash has failed
you could try to flash it back to the original FW

@vicamo
Copy link
Contributor Author

vicamo commented Jun 2, 2022

original

Nope. The ljca driver now never loads. There is no such .../ljca/.../cmd node anymore.

@hao-yao
Copy link
Contributor

hao-yao commented Jun 2, 2022

Nope. The ljca driver now never loads. There is no such .../ljca/.../cmd node anymore.

Just use dfu-utils command is OK, no need to echo to cmd. The dfu device should still be there.

@vicamo
Copy link
Contributor Author

vicamo commented Jun 2, 2022

Nope. The ljca driver now never loads. There is no such .../ljca/.../cmd node anymore.

Just use dfu-utils command is OK, no need to echo to cmd. The dfu device should still be there.

$ sudo dfu-util -l
[sudo] password for ubuntu: 
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [04d8:0037] ver=0401, devnum=2, cfg=1, intf=0, path="3-6", alt=0, name="UNKNOWN", serial="MCHP-DF-005"

Reflashed 0.6.0.42. No driver bound to this device. What else can we do?

@ccgjimlai
Copy link
Contributor

When you reboot your device, will it exit dfu mode?
if the device is still in dfu mode, it means that the fw was not flashed successfully.
Otherwise, if the device exits dfu mode after reboot, then ljca will be loaded.

@vicamo
Copy link
Contributor Author

vicamo commented Jun 6, 2022

When you reboot your device, will it exit dfu mode? if the device is still in dfu mode, it means that the fw was not flashed successfully. Otherwise, if the device exits dfu mode after reboot, then ljca will be loaded.

it stays as is even after warm/cold reboots.

@ccgjimlai
Copy link
Contributor

After updating the MCU fw on Vicamo's device, the camera can work now,
SW is untouched (supposedly 5.15 kernel)

PXL_20220609_043107069

@fourdollars
Copy link

I can not upgrade the fw.

$ cat /sys/bus/usb/drivers/ljca/*/version
0.6.0.42
u@u:~/d21_fw/0.8.0.9$ sudo dfu-util -v -R -D usb_bridge_dell_fw.dfu
[sudo] password for u: 
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

DFU suffix version 100
Match vendor ID from file: 04d8
Match product ID from file: 0037
dfu-util: No DFU capable USB device available
u@u:~/d21_fw/0.8.0.9$ uname -a
Linux u 5.15.0-9036-generic #37+exp.32-Ubuntu SMP Thu Jun 9 08:34:09 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
u@u:~/d21_fw/0.8.0.9$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 8086:0b63 Intel Corp. USB Bridge
Bus 003 Device 003: ID 8087:0032 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

@fourdollars
Copy link

It needs to echo dfu into /sys/bus/usb/drivers/ljca/3-6:1.0/cmd to make it flashable.

u@u:~/d21_fw/0.8.0.9$ sudo su -c "echo dfu > /sys/bus/usb/drivers/ljca/3-6\:1.0/cmd"
u@u:~/d21_fw/0.8.0.9$ sudo dfu-util -v -R -D usb_bridge_dell_fw.dfu
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

DFU suffix version 100
Match vendor ID from file: 04d8
Match product ID from file: 0037
Opening DFU capable USB device...
ID 04d8:0037
Run-time device DFU version 0100
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 0100
Device returned transfer size 64
Copying data from PC to DFU device
Download    [=========================] 100%        32752 bytes
Download done.
Sent a total of 32752 bytes
state(2) = dfuIDLE, status(0) = No error condition is present
Done!
dfu-util: can't detach
Resetting USB to switch back to runtime mode

Some errors in dmesg.

[  208.527602] usb 3-6: USB disconnect, device number 2
[  208.528517] gpio gpiochip1: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED
[  208.528597] ljca 3-6:1.0: LJCA disconnected
[  208.873284] usb 3-6: new full-speed USB device number 4 using xhci_hcd
[  209.022757] usb 3-6: New USB device found, idVendor=04d8, idProduct=0037, bcdDevice= 4.01
[  209.022767] usb 3-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  209.022771] usb 3-6: Product: USB DFU
[  209.022774] usb 3-6: Manufacturer: MCHP
[  209.022776] usb 3-6: SerialNumber: MCHP-DF-005
[  217.535916] usb 3-6: reset full-speed USB device number 4 using xhci_hcd
[  220.970735] usb 3-6: device descriptor read/64, error -71

@fourdollars
Copy link

There is no /sys/bus/usb/drivers/ljca/*/version now.

$ cat /sys/bus/usb/drivers/ljca/*/version
cat: '/sys/bus/usb/drivers/ljca/*/version': No such file or directory

$ sudo dfu-util -l
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [04d8:0037] ver=0401, devnum=2, cfg=1, intf=0, path="3-6", alt=0, name="UNKNOWN", serial="MCHP-DF-005"

$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 04d8:0037 Microchip Technology, Inc. USB DFU
Bus 003 Device 003: ID 8087:0032 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

@ccgjimlai
Copy link
Contributor

d21_fw_0.8.0.16.zip

Hi $4,
Could you try again with the attached FW?

@fourdollars
Copy link

@ccgjimlai It is also trapped in DFU state now.
So I can not flash the fw again.

@ccgjimlai
Copy link
Contributor

Hi @fourdollars
Just use dfu-utils command is OK, no need to echo to cmd. The dfu device should still be there, so that you could still perform fw flash

@fourdollars
Copy link

u@u:~/0.8.0.16$ sudo dfu-util -v -R -D usb_bridge_dell_fw_production.dfu
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

DFU suffix version 100
Match vendor ID from file: 04d8
Match product ID from file: 0037
Opening DFU capable USB device...
ID 04d8:0037
Run-time device DFU version 0100
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 0100
Device returned transfer size 64
Copying data from PC to DFU device
Download	[=========================] 100%        32752 bytes
Download done.
Sent a total of 32752 bytes
state(2) = dfuIDLE, status(0) = No error condition is present
Done!
dfu-util: can't detach
Resetting USB to switch back to runtime mode

@fourdollars
Copy link

[  737.413699] usb 3-6: reset full-speed USB device number 2 using xhci_hcd
[  741.117481] usb 3-6: device descriptor read/64, error -71
[  741.373748] usb 3-6: device firmware changed
[  741.374034] usb 3-6: USB disconnect, device number 2
[  741.501406] usb 3-6: new full-speed USB device number 4 using xhci_hcd
[  741.650814] usb 3-6: New USB device found, idVendor=8086, idProduct=0b63, bcdDevice= 8.16
[  741.650823] usb 3-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  741.650826] usb 3-6: Product: USB Bridge
[  741.650829] usb 3-6: Manufacturer: MCHP
[  742.249265] ljca 3-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[  742.249277] ljca 3-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[  742.249517] ljca 3-6:1.0: LJCA USB device init success
[  742.249602] usbcore: registered new interface driver ljca
[  742.271237] int3472-discrete INT3472:06: error -EBUSY: Failed to map GPIO to clock
[  742.271472] int3472-discrete: probe of INT3472:06 failed with error -16
[  742.274546] intel-ipu6-isys intel-ipu6-isys0: bind ov01a1s 20-0036 nlanes is 1 port is 1
[  742.274655] intel-ipu6-isys intel-ipu6-isys0: All sensor registration completed.
u@u:~/0.8.0.16$ cat /sys/bus/usb/drivers/ljca/*/version
0.8.0.16

@fourdollars
Copy link

@ccgjimlai Thx. The camera works on 5.15.0-9036-generic #37+exp.32-Ubuntu.

@jamesps-ebi
Copy link

We're also tracking an intermittent failure to probe ov01a1s running last few revisions of oem-5.14 kernels on both ANDW-DVT2-C1 and C3. It appears the sensor may sometimes not be probed, and following message is missing from dmesg:

intel-ipu6-isys intel-ipu6-isys0: bind ov01a1s 20-0036 nlanes is 1 port is 1

Last known good is 5.14.0-1027-oem, and roughly 50% failure rate in 1035, and 100% in 1038 and 1040.

Is there an open github issue for this? We have a Latitude 9420 (Ubuntu 20.04.4 LTS, kernel 5.14.0.1044.40) in our organisation with seemingly identical behaviour. IPU6 camera is intermittently failing between boots.
If there is an open issue, I would like to at least track the progress.

@vicamo
Copy link
Contributor Author

vicamo commented Aug 26, 2022

Closing this as resolved for me.

@vicamo vicamo closed this as completed Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants