sunxi: `reboot` does not perform a reboot

[djurny]

Seems that a recent rolling update has made my sunxi Orange Pi zero boards stuck at reboot. Kernel shuts down fine, no errors, prints out "Restarting system" or "Rebooting system" but it seems to just get stuck. Magic SysRq not working so kernel has left the building.

[84345.585698] systemd-shutdown[1]: Using hardware watchdog 'sunxi-wdt', version 0, device /dev/watchdog0
[84345.595240] systemd-shutdown[1]: Failed to set watchdog hardware timeout to 10min: Invalid argument
[84345.684052] systemd-shutdown[1]: Syncing filesystems and block devices.
[84345.755564] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[84345.780255] systemd-journald[228]: Received SIGTERM from PID 1 (systemd-shutdow).
[84345.877323] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[84345.902339] systemd-shutdown[1]: Unmounting file systems.
[84345.912897] (sd-umount)[11817]: Unmounting '/tmp'.
[84345.922920] (sd-remount)[11818]: Remounting '/' read-only with options 'background_gc=on,nogc_merge,discard,discard_unit=block,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,barrier,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,memory=normal,errors=continue'.
[84346.040634] systemd-shutdown[1]: All filesystems unmounted.
[84346.046333] systemd-shutdown[1]: Deactivating swaps.
[84346.051504] systemd-shutdown[1]: All swaps deactivated.
[84346.056778] systemd-shutdown[1]: Detaching loop devices.
[84346.066741] systemd-shutdown[1]: All loop devices detached.
[84346.072373] systemd-shutdown[1]: Stopping MD devices.
[84346.078030] systemd-shutdown[1]: All MD devices stopped.
[84346.083398] systemd-shutdown[1]: Detaching DM devices.
[84346.089090] systemd-shutdown[1]: All DM devices detached.
[84346.094538] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[84346.112307] systemd-shutdown[1]: Syncing filesystems and block devices.
[84346.119273] systemd-shutdown[1]: Rebooting.
[84346.164738] reboot: Restarting system

No further output or activity observed after this. Board does not actually reboot/reset. It requires a power cycle to start again.

Could be that the board just halts or powers down, but not quite sure how to determine that with for example a multimeter.

Will update the description here after I found out what rolling release version caused this to happen.

Groetjes,

[github-actions]

Jira ticket: AR-2684

[djurny]

Downgraded from 25.2.3 to 24.11.1 seems to resolve the issue (includes tio timestamps):

[19:49:45] [   87.616053] systemd-shutdown[1]: Stopping MD devices.
[19:49:45] [   87.621767] systemd-shutdown[1]: All MD devices stopped.
[19:49:45] [   87.627230] systemd-shutdown[1]: Detaching DM devices.
[19:49:45] [   87.632944] systemd-shutdown[1]: All DM devices detached.
[19:49:45] [   87.638393] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[19:49:45] [   87.656493] systemd-shutdown[1]: Syncing filesystems and block devices.
[19:49:45] [   87.663469] systemd-shutdown[1]: Rebooting.
[19:49:45] [   87.720599] reboot: Restarting system

[19:49:46] U-Boot SPL 2024.01-armbian (Feb 23 2024 - 10:47:39 +0000)
[19:49:46] DRAM: 256 MiB
[19:49:46] Trying to boot from MMC1
[19:49:46] ns16550_serial serial@1c28000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19


[19:49:46] U-Boot 2024.01-armbian (Feb 23 2024 - 10:47:39 +0000) Allwinner Technology

[DOWNGRADE] armbian-bsp-cli-orangepizero-current:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] armbian-firmware:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] linux-dtb-current-sunxi:armhf 25.2.3 -> 24.11.1
[DOWNGRADE] linux-image-current-sunxi:armhf 25.2.3 -> 24.11.1
[UPGRADE] linux-u-boot-orangepizero-current:armhf 24.2.1 -> 24.11.1

Note that I at first thought it was due to U-Boot, so I tried with built U-Boots and downgraded linux-u-boot-orangepizero-current packages.

Will dive into the diffs to see if I can find the culprit.

Grt,

[djurny]

In 25.2.2 reboot also works.

========================================
...
[DOWNGRADE] armbian-bsp-cli-orangepizero-current:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] armbian-firmware:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-dtb-current-sunxi:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-image-current-sunxi:armhf 25.08.0-trunk -> 25.2.2
[DOWNGRADE] linux-u-boot-orangepizero-current:armhf 25.08.0-trunk -> 25.2.2
========================================

[djurny]

Seems like issue introduced when moving from 6.6.75-current-sunxi to 6.12.23-current-sunxi

[EvilOlaf]

Downgrading u-boot package only replaces binaries available for flashing. Though flashing does not happen automatically since it is rarely necessary to actually do that.
sunxi family is very heavily patched so regressions are likely. Finding the cause though can be quite a challenge.