Unable to boot with latest firmware on RPI4

[Nopel2020]

On a RPI4 with 4GB with the latest kernel (5.4.65-V8+) and the latest firmware (086d9d9) my RPI4 freezes on the rainbow screen.

The only files from the firmware I'm using are fixup4cd.dat and start4cd.elf. After replacing just these two files with the ones from the latest commit before that (a490197), the RPI4 is booting normally again.

With the latest firmware files, the freeze is reproduceable.

• Which model of Raspberry Pi?
  RPI4, 4GB

• Which OS and version (cat /etc/rpi-issue)?
  64bit Debian Buster with self-compiled 64bit 5.4.65 kernel (but I don't think this is relevant, the Pi freezes before it loads anything from the OS)

[pelwell]

I don't think this is relevant, the Pi freezes before it loads anything from the OS

What makes you say that? Are you capturing the output from the firmware over the UART?

[Nopel2020]

Because it freezes on the rainbow screen. Does it load anything from the OS while the rainbow screen is still visible?

[pelwell]

Yes - it's hard for the kernel to display raspberries and start to output messages until its been loaded.

[pelwell]

Do you have any custom settings in config.txt?

[mikey1963]

Mine frozen too, latest rpi-update, running 64bit kernel.

…

--- Regards, Mr Michael A Smith My apologies in advance for any errors or inaccuracies, mobile email and my brain are both prone to them.

________________________________ From: Nopel2020 <notifications@github.com> Sent: Tuesday, September 15, 2020 8:45:43 PM To: raspberrypi/firmware <firmware@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [raspberrypi/firmware] Unable to boot with latest firmware on RPI4 (#1467) Because it freezes on the rainbow screen. Does it load anything from the OS while the rainbow screen is still visible? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1467 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHYEUDLFZRRYQXCF3UCCWTSF676PANCNFSM4RNUOGAA>.

[Nopel2020]

Ah ok, sorry, then I'm not sure if it's loading something from the OS.

This is my full config.txt:
gpu_mem=16
hdmi_ignore_cec=1
hdmi_force_hotplug=1
hdmi_group=1
hdmi_mode=4
initramfs initrd8.img followkernel

[pelwell]

@mikey1963 are you also an initramfs user?

[mikey1963]

No, Only changes to standard are; gpu_mem=256 dtoverlay=i2c-rtc,ds3231 enable_uart=1 arm_64bit=1 Did the rpi-update this evening and it hangs on boot. I’m headless so can’t give you much more info, sorry.

…

--- Regards, Mr Michael A Smith My apologies in advance for any errors or inaccuracies, mobile email and my brain are both prone to them.

________________________________ From: Phil Elwell <notifications@github.com> Sent: Tuesday, September 15, 2020 8:50:49 PM To: raspberrypi/firmware <firmware@noreply.github.com> Cc: Mike Smith <mike@realtimetesting.co.uk>; Mention <mention@noreply.github.com> Subject: Re: [raspberrypi/firmware] Unable to boot with latest firmware on RPI4 (#1467) @mikey1963<https://github.com/mikey1963> are you also an initramfs user? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1467 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHYEUHIJEZ6SX2W4ADOC5LSF7ARTANCNFSM4RNUOGAA>.

[pelwell]

Oh well - we'll investigate tomorrow.

[Nopel2020]

Using the next-to-last firmware files is a workaround, so no rush. Thanks for looking into this!

[pelwell]

I have to - that commit has my fingerprints all over it.

[mikey1963]

That’s what makes it fun :)

…

--- Regards, Mr Michael A Smith My apologies in advance for any errors or inaccuracies, mobile email and my brain are both prone to them.

________________________________ From: Phil Elwell <notifications@github.com> Sent: Tuesday, September 15, 2020 9:12:21 PM To: raspberrypi/firmware <firmware@noreply.github.com> Cc: Mike Smith <mike@realtimetesting.co.uk>; Mention <mention@noreply.github.com> Subject: Re: [raspberrypi/firmware] Unable to boot with latest firmware on RPI4 (#1467) I have to - that commit has my fingerprints all over it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1467 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHYEUGGNUTRZZ4POMHPYCLSF7DCLANCNFSM4RNUOGAA>.

[Nopel2020]

FWIW, I'm running the exact same setup on an RPI3B, and almost identical (but 32bit) on an old RPI1 and some RPI0Ws. They are all booting fine, only the RPI4 is affected.

[pelwell]

If it's the change I think it is then the code path is only visited when arm_64bit=1, but the 64-bit 3B non-failure is an interesting data point.

[yvoy]

Same issue on RPI4 , need to temporaly disable arm_64bit=1

[pelwell]

Problem identified - sorry folks. Luckily, in the process of fixing this I spotted a corner case that also wouldn't have worked. I'll comment again when the fix is available.

[popcornmix]

Fix should be in latest rpi-update firmware - please test and report if issue is resolved.

[pelwell]

Thanks, @popcornmix.

[by]

Fix should be in latest rpi-update firmware - please test and report if issue is resolved.

Confirmed: it does work now.

[mikey1963]

Confirmed now back fully working, great work guys. Thanks. —- Regards, Mr Michael A Smith. My apologies in advance for any errors or inaccuracies, mobile email and my brain are both prone to them.

…

________________________________ From: popcornmix <notifications@github.com> Sent: Wednesday, September 16, 2020 11:18:24 AM To: raspberrypi/firmware <firmware@noreply.github.com> Cc: Mike Smith <mike@realtimetesting.co.uk>; Mention <mention@noreply.github.com> Subject: Re: [raspberrypi/firmware] Unable to boot with latest firmware on RPI4 (#1467) Fix should be in latest rpi-update firmware - please test and report if issue is resolved. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1467 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHYEUBDG6DT6N4KXJZEFUTSGCGHBANCNFSM4RNUOGAA>.

[pelwell]

If you're feeling brave you can now gzip your kernel8.img:

$ sudo gzip -k /boot/kernel8.img
$ sudo mv /boot/kernel8.img{,.orig}
$ sudo mv /boot/kernel8.img{.gz,}

[tomty89]

@pelwell It still fails to load u-boot in ArchLinuxARM (aarch64)

[vbauerster]

Can confirm that RPI3B+ affected as well.

[pelwell]

Can confirm that RPI3B+ affected as well.

With U-boot or a Linux kernel?

[vbauerster]

There is just a rainbow screen no any kernel boot messages, that's all I know.

[Nopel2020]

Original poster here. I can confirm that it's working on my RPI4 (and also on my RPI3B, (not 3B+)). Thanks!

[pelwell]

@vbauerster When did you install the update?

[pelwell]

@vbauerster A few more questions:

2. Are you using a Raspberry Pi OS image?
3. Have you customised config.txt in any way?
4. If you can put the SD card in another device, how large (in bytes) is start.elf?

[pelwell]

@tomty89 Are you definitely running the latest firmware? What does strings start.elf | grep VC_BUILD show?

What do you have in config.txt? (You ignore comments and blank lines)

[graysky2]

@pelwell - The raspberrypi-firmware package for Arch ARM is currently supplying 525f1eb

It is my understanding that the only time it needs to be updated is if there are changes to anything under https://github.com/raspberrypi/firmware/tree/master/hardfp/opt/vc ... Please correct me if I am mistaken.

[pelwell]

That's correct - if it works for you, stick with it. Only the brave and the foolish take every update that comes along.

[graysky2]

@pelwell - For Arch ARM as of today:

• raspberrypi-firmware is currently supplying 525f1eb
• raspberrypi-bootloader is currently supplying 28f031e

On Arch ARM armv7h, RPi4 boots fine.
On Arch ARM aarch64, RPi4 does NOT boot. I too only see the rainbow screen which does not go away.

% strings /boot/start.elf | grep VC_BUILD
VC_BUILD_ID_USER: dom
VC_BUILD_ID_TIME: 11:12:26
VC_BUILD_ID_VARIANT: start
VC_BUILD_ID_TIME: Sep 16 2020
VC_BUILD_ID_BRANCH: bcm2711_2
VC_BUILD_ID_HOSTNAME: buildbot
VC_BUILD_ID_PLATFORM: raspberrypi_linux
VC_BUILD_ID_VERSION: 0368c5cfa664e0534e5428dc45ac2a972802159c (clean)

[pelwell]

What does Arch put in its config.txt? Until we understand why you and some others are still affected I suggest you revert your raspberrypi-bootloader package to a490197.

[xvanc]

On a fresh install just "enable_uart=1".
A user on the forum is reporting that it hangs after a message from systemd saying "A start job is running for /dev/mmcblk0".

[pelwell]

@thisisnotev A fresh install of what?

[xvanc]

@pelwell ArchLinux ARM, aarch64

[graysky2]

Until we understand why you and some others are still affected I suggest you revert your raspberrypi-bootloader package to a490197.

For Arch ARM users of aarch64, the corresponding package is raspberrypi-bootloader-20200911-1 which boots just fine.

[pelwell]

What does Arch call its 64-bit kernel image?
Does including arm_64bit=1 in config.txt make it work?

[tomty89]

@pelwell Image and Image.gz (the former is used AFAIK). kernel8.img is u-boot.

[vbauerster]

@pelwell

When did you install the update?

Yesterday.

Are you using a Raspberry Pi OS image?

No, I use Arch linux.

Have you customised config.txt in any way?

No, I have not.

If you can put the SD card in another device, how large (in bytes) is start.elf?

I already restored from backup, so cannot check.

PS: I actually referred to this issue via this post.

[vbauerster]

Still "bricking" device with bootloader 20200916-1

[pelwell]

There's clearly something different about Arch that is triggering the boot failure. I'm installing the 64-bit Arch to test it.

[pelwell]

I think I've got this: Arch uses U-boot as its kernel8.img, and U-boot doesn't include the ARM64 header used by the Linux kernel. An unintentional result of this was to change the load address of the image, which U-boot doesn't like - it's either not position-independent code or it ended up clashing with other resources. The fix is in the internal repo and will be in the next rpi-update release. In the meantime you can work around the issue (for testing purposes) by adding kernel_address=0x80000 to config.txt.

[ghost]

I think I've got this: Arch uses U-boot as its kernel8.img, and U-boot doesn't include the ARM64 header used by the Linux kernel. An unintentional result of this was to change the load address of the image, which U-boot doesn't like - it's either not position-independent code or it ended up clashing with other resources. The fix is in the internal repo and will be in the next rpi-update release. In the meantime you can work around the issue (for testing purposes) by adding kernel_address=0x80000 to config.txt.

I can confirm this workaround enables a previously non-booting aarch64 install to boot (Raspberry Pi 4 1GB RAM).

[popcornmix]

Latest rpi-update firmware has the fix for kernel_address.

[graysky2]

@pelwell @popcornmix - Thank you both for the rapid engagement to detect and fix this. I can confirm that 7b99da7 fixes the issue on Arch ARM RPi4 (aarch64); I just updated Arch ARM's bootloader package. Safe to close this ticket from my perspective.

[pelwell]

Thanks - I don't plan to make any further changes in this area for a while...