rpi-4.19 fails to boot on arm64

[eccgecko]

Is this the right place for my bug report?
I know neither arm64 nor arch linux are officially supported, but I feel like this needs to be reported nonetheless and didn't know where else best to do that.

Describe the bug
rpi-4.19.y does not boot on arm64/aarch64.

To reproduce
Try building 4.19.y kernel for arm64 pi and booting.

Expected behaviour
Raspberry Pi 3B+ should boot into aarch64 arch linux os on 4.19.y kernel.

Actual behaviour
Tried building the 4.19 kernel for use on arm64 architecture. I am currently running arch linux arm (aarch64) on my pi 3b+, which has been running on the 64-bit rpi-4.14.y branch (4.14.79-v8+).

When making the baseline config, using the command make arch=arm64 bcmrpi3_defconfig I am receiving the following errors:

arch/arm64/configs/bcmrpi3_defconfig:472:warning: override: reassigning to symbol USB_LAN78XX
arch/arm64/configs/bcmrpi3_defconfig:643:warning: symbol value 'm' invalid for LIRC
arch/arm64/configs/bcmrpi3_defconfig:1288:warning: override: reassigning to symbol MMC_BCM2835_MMC
arch/arm64/configs/bcmrpi3_defconfig:1289:warning: override: reassigning to symbol MMC_SDHCI_IPROC

However, these errors are not fatal to the build, so am unsure if these are the reason kernel fails to boot. I can continue the build using the usual methods (methods that have worked fine running aarch64 and using exact same commands when compiling rpi-4.14.y kernels on and for the pi).

make ARCH=arm64 Image dtbs modules -j 4 works fine, building initramfs works fine, but as soon as I move the newly compiled kernel over to /boot/ as I usually do (as well as necessary modules and dtbs) and reboot, pi fails to boot. Moving back to old 4.14.79 kernel fixes it.

System
Raspberry Pi 3B+
Arch Linux Arm aarch64
4.14.79-v8+

[lategoodbye]

Doesn't boot is a interpretation of some observation which you didn't describe (LED behavior, output on HDMI or serial).
Do you use an additional bootloader?

[eccgecko]

Apologies. I run the pi headless, and don't have a monitor to hook it up to to interpret any messages. Would I need to get hold of one before I can get you any really useful information?

As for LED behaviour, green is flickering as normal as it starts to boot, but it eventually just hangs on solid green and red lights, after about 10 seconds or so of green flickering.

I don't believe I have an additional bootloader. Do you mean something like NOOBS or berryboot? In which case no. It’s just arch running on the rpi-4.14.y kernel (trying to upgrade go 4.19.y). In trying to upgrade, I have an initramfs built using the 4.19.1 modules that are installed as part of the kernel compilation, and the kernel8.img. Would it helpful for me to copy in my config.txt?

[pelwell]

I just tried a clean aarch64 compile of rpi-4.19.y, producing a 4.19.1-v8+ build. It boots to the desktop without errors (except for:

[    1.001156] Error: Driver 'sdhost-bcm2835' is already registered, aborting...

which is due to having two drivers for the SDHOST interface).

The complete dmesg output can be found here.

[eccgecko]

@pelwell Interesting. So I am most likely doing something wrong, or have some unique quirk on my system that is causing this.

May I ask, using the bcmrpi3_defconfig did your make throw up any of the same warnings I referenced it my first post? Also, what were the dtbs that you copied over to /boot/?

[pelwell]

Yes, I get exactly the same warnings:

arch/arm64/configs/bcmrpi3_defconfig:472:warning: override: reassigning to symbol USB_LAN78XX
arch/arm64/configs/bcmrpi3_defconfig:643:warning: symbol value 'm' invalid for LIRC
arch/arm64/configs/bcmrpi3_defconfig:1288:warning: override: reassigning to symbol MMC_BCM2835_MMC
arch/arm64/configs/bcmrpi3_defconfig:1289:warning: override: reassigning to symbol MMC_SDHCI_IPROC

I've regenerated bcmrpi3_defconfig, also taking out CONFIG_MMC_BCM2835 to avoid the conflict with the sdhost-bcm2835 driver. It now boots without errors.

There are quite a lot of compile-time warnings on aarch64 - if there's time I may tidy up some of them.

[eccgecko]

Nice, I can get it to boot now :) I'm not sure if it's the changes you committed that I pulled down, or whether it was because before I had not updated the *.elf, *.dat, and bootcode.bin files in /boot/ from the next branch of raspberrypi/firmware, which I have now done. Either way, I just recompiled and updated those files, and I can now boot and I now get 4.19.1-v8+ aarch64 :D

Also, with this compile I didn't get those warnings during make either, so good job :) thanks for your help.

[eccgecko]

Damn. Spoke too soon. Just rebooted using sudo shutdown -r now and the pi refused to boot again. At least some progress was made and I now know it is possible to boot into 4.19.1-v8 aarch64 kernel. Not sure what exactly the problem is though :(

[eccgecko]

Having plugged the sd card into another linux machine, fsck is reporting 0x41 Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. on the /boot/ partition. Fixing this on the other machine enables me to boot back into the pi. Tried rebooting again, and the same thing happens. Seems there is something about the 4.19.y kernel that is stopping the /boot/ partition from being properly unmounted at shutdown. As the original reason I opened this issue was, I think, a different issue to do with the kernel build, is this new problem something I should open a new issue on, or maybe report upstream? Do you have a similar problem on 4.19.1-v8+ @pelwell ?

[pelwell]

I'm not using arm64 builds on a regular basis, but on the few occasions I've (re)booted into it in the last few days I've not had a problem.

[pelwell]

Upstream will want to know that you are running a kernel built from a pure upstream tree, but there's no harm in asking if anybody else has seen the problem. You can also discuss the issue here, although official support is likely to be sporadic - close this issue and open a new one, so the title is more accurate.

[lategoodbye]

Without a real indication (stacktrace, kernel msg) that the issue is kernel related, your chance is very little get any feedback. I usually test the mainline kernel (arm64/defconfig) on top of Arch Linux and didn't have any issues with 4.19. So i suggest to connect a HDMI monitor / serial adapter to your Pi to see what's the problem.

[eccgecko]

@lategoodbye Yeah, I’ve realised that without connecting to an external monitor for feedback, the chances of getting to the bottom of this are next to zero. I will have to borrow a monitor, as right now I have nothing to connect it to at home. Thanks.

Thanks @pelwell I will do as @lategoodbye says first, and get hold of a monitor, and then report my findings either in another issue here, or upstream. I realise arm64 support is sporadic here, but I appreciate the efforts to help out. Certainly some progress was made, as I was able to actually boot, even if it isn’t clear what exactly the issue was. Thanks for all your help.