5.10.83 will not reboot on armv6/7

[gearhead]

The 'shipping' kernel 5.10.63+ does reboot. When the 'bleeding edge' kernel is built, it fails to reboot. I believe this bug was introduced at 5.10.80 or 5.10.81. The kernel I built last night reported itself as 5.10.83-v7+ and was built from commit: 20f6415.
What appears to happen is that the Rpi does not shut down and therefore never reboots. This is after issuing the 'sudo reboot' command from the command line. This is a serious issue with headless devices.

[pelwell]

A fresh build from that commit reboots for me on a 3B+:

raspberrypi login: pi
Password:
Linux raspberrypi 5.10.83-v7+ #1884 SMP Thu Dec 16 15:20:44 GMT 2021 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Dec 16 15:22:50 GMT 2021 on tty1
pi@raspberrypi:~$ uname -a
Linux raspberrypi 5.10.83-v7+ #1884 SMP Thu Dec 16 15:20:44 GMT 2021 armv7l GNU/Linux
pi@raspberrypi:~$ sudo reboot
[   63.469379] reboot: Restarting system




Raspberry Pi Bootcode
Read File: config.txt, 2123
Read File: start.elf, 2963616 (bytes)
Read File: fixup.dat, 7221 (bytes)
MESS:00:00:03.375595:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:03.380744:0: brfs: File read: 2123 bytes
...

1. Which model of RPi are you using?
2. What bootmode (SD card, network, USB MSB etc.)?
3. Which peripherals are attached?

[gearhead]

This is on a RPi 3B+ off an SD card. The only peripherals attached are a tft screen. There is a usb fob in one of the ports. It was connecting by wifi. I have seen this with or without the dtoverlay enabled for the tft screen. I am running 'gpu_mem=16' to boot it headless w/o the vc. I see that you are using start.elf and I think I should be using start_cd.elf...

[pelwell]

Hmmm - gpu_mem=16 does break rebooting for me (I'd just written the opposite in anticipation, and was forced to edit it).

I notice that the ISP driver is crashing on startup with (and only with) the cutdown firmware:

[   10.465985] 8<--- cut here ---
[   10.474001] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[   10.487970] pgd = fdf63d78
[   10.506723] [00000000] *pgd=00000000
[   10.519338] Internal error: Oops: 5 [#1] SMP ARM
[   10.528788] Modules linked in: snd_bcm2835(C+) v4l2_mem2mem bcm2835_isp(C+) videobuf2_vmalloc bcm2835_mmal_vchiq(C) snd_pcm videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 snd_timer videobuf2_common snd videodev mc vc_sm_cma(C) fixed uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[   10.569533] CPU: 0 PID: 160 Comm: systemd-udevd Tainted: G         C        5.10.83-v7+ #1884
[   10.583395] Hardware name: BCM2835
[   10.592115] PC is at bcm2835_isp_remove+0x34/0x1e4 [bcm2835_isp]
[   10.603427] LR is at bcm2835_isp_probe+0xa4/0x5f0 [bcm2835_isp]
[   10.614559] pc : [<7f24a0ec>]    lr : [<7f24c1e8>]    psr: 60000113
[   10.626070] sp : 8405fb98  ip : 8405fbd8  fp : 8405fbd4
[   10.636534] r10: 00010001  r9 : 7f24db80  r8 : 7f24dba8
[   10.647091] r7 : fffffffc  r6 : 00000000  r5 : 814f4040  r4 : fffffffc
[   10.647099] r3 : 00000000  r2 : 00000003  r1 : 00000000  r0 : 81799200
[   10.647110] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   10.647124] Control: 10c5383d  Table: 0339406a  DAC: 00000055
[   10.693801] Process systemd-udevd (pid: 160, stack limit = 0xc2e8fc30)
[   10.705550] Stack: (0x8405fb98 to 0x84060000)
[   10.715065] fb80:                                                       7f1af1e4 8071d0d8
[   10.728498] fba0: ffffffed 814f4040 8405fbd4 ffffffed 814f4040 81799210 00000000 810ee074
...
[   11.206062] Backtrace:
[   11.214357] [<7f24a0b8>] (bcm2835_isp_remove [bcm2835_isp]) from [<7f24c1e8>] (bcm2835_isp_probe+0xa4/0x5f0 [bcm2835_isp])
[   11.231540]  r10:00010001 r9:7f24f01c r8:810ee074 r7:00000000 r6:81799210 r5:814f4040
[   11.245465]  r4:ffffffed
[   11.253981] [<7f24c144>] (bcm2835_isp_probe [bcm2835_isp]) from [<80725e10>] (platform_drv_probe+0x58/0xa8)
[   11.269950]  r10:8323adc0 r9:7f24f01c r8:810ee074 r7:00000000 r6:7f24f01c r5:81799210
[   11.284007]  r4:00000000
[   11.292625] [<80725db8>] (platform_drv_probe) from [<80723a60>] (really_probe+0x108/0x3bc)
[   11.307118]  r7:00000000 r6:00000000 r5:810ee06c r4:81799210
[   11.319010] [<80723958>] (really_probe) from [<80723f04>] (driver_probe_device+0x6c/0xc4)
[   11.319029]  r10:8323adc0 r9:00000000 r8:00000002 r7:80feeba8 r6:7f24f01c r5:7f24f01c
[   11.319045]  r4:81799210
[   11.337845] [<80723e98>] (driver_probe_device) from [<80724154>] (device_driver_attach+0x68/0x70)
[   11.346847]  r5:00000000 r4:81799210
[   11.350480] [<807240ec>] (device_driver_attach) from [<807241c4>] (__driver_attach+0x68/0xdc)
[   11.359133]  r7:80feeba8 r6:81799210 r5:7f24f01c r4:00000000
[   11.364883] [<8072415c>] (__driver_attach) from [<80721a0c>] (bus_for_each_dev+0x88/0xd4)

Can you confirm that same crash on startup while I revert to 5.10.81 to see if it goes away?

[pelwell]

I think c293474 is the culprit - after reverting, the crash goes away and reboot is successful.

Any ideas, @naushir? Is creating the second instance using resources the cutdown firmware doesn't have?

[6by9]

The ISP isn't present at all with the cutdown firmware.
bcm2835_isp_probe_instance should fail on the first instance, and then break out of the loop and clean up. Something obviously isn't protected in the way it is intended to be, and it's being dereferenced.

[pelwell]

The problem is that the drvdata is unset until bcm2835_isp_probe_instance has been called successfully for all instances - the platform_set_drvdata needs to go before the loop.

[pelwell]

I'll fix it.

[6by9]

Setting platform_set_drvdata(pdev, bcm2835_isp_instances); at the end of bcm2835_isp_probe when we could have done the goto error which then calls bcm2835_isp_remove which bcm2835_isp_instances = platform_get_drvdata(pdev);.

(EDIT) Naush^H^H^H^H^HPhil beat me to it!

[naushir]

:(

[pelwell]

Fixed by 3ec9b52.

[gearhead]

wow! that was a quick fix. I'll re-build and see how it goes.

[gearhead]

I did a git pull, verified that the 3ec9b52 commit was applied (from git log) then I rebuilt the kernel and installed it. It advertises as 5.10.83-v7+. It still will not reboot when 'gpu_mem=16'. The console shows that it reached target "Reboot" then a lot of systemd-shutdown lines then the final line systemd-shutdown[1]: Rebooting. but nothing happens after that. I have to power cycle to get it to reboot.
I've commented everything out of my config except 'gpu_mem=16'. Still will not reboot when I am logged in via SSH or when I am logged in at the console. If I change to 'gpu_mem=32', it will reboot and act normally. This is with a clean Raspberry Pi OS Lite image.

[pelwell]

You are not running the correct build - a kernel built for Pi 3 from 3ec9b52 reports itself as 5.10.85-v7+.

[gearhead]

This seems fixed. I verified it works on armv6 and armv7.