RPi no longer booting after upgrading to latest start.elf/fixup.dat

[pbatard]

A regression appears to have been introduced with the latest firmware in that we are no longer able to boot binaries such as the Trusted Firmware or the UEFI firmware. Reverting to an older version of the RPi firmware solves this issue.

Steps to replicate

• Download bl31.bin (A Raspberry Pi 4 build of the ARM Trusted Firmware binary) from https://github.com/tianocore/edk2-non-osi/blob/master/Platform/RaspberryPi/RPi4/TrustedFirmware/bl31.bin and save it to SD/USB
• Add the latest start4.elf, fixup4.dat and bcm2711-rpi-4-b.dtb
• Add the following config.txt:

arm_64bit=1
enable_uart=1
uart_2ndstage=1
enable_gic=1
armstub=bl31.bin
disable_commandline_tags=1
disable_overscan=1
device_tree_address=0x1f0000
device_tree_end=0x200000

• Try to boot the platform

Result

The boot freezes after the following output on the serial console, with the multicoloured screen staying on:

recover4.elf not found (6)
recovery.elf not found (6)
Read start4.elf bytes  2277760 hnd 0x00000089 hash 'ae50b620c077baa5'
Read fixup4.dat bytes     5418 hnd 0x00000088 hash '3f2c561eec60a59c'
0x00c03111 0x00000000 0x0000001f
MEM GPU: 76 ARM: 948 TOTAL: 1024
Starting start4.elf @ 0xfec00200 partition 0

Then, if you replace start4.elf and fixup4.dat with a version published before 2020.07.14, everything works as expected, and you get the expected output:

recover4.elf not found (6)
recovery.elf not found (6)
Read start4.elf bytes  2277248 hnd 0x00000089 hash '8e98b15f075142da'
Read fixup4.dat bytes     5409 hnd 0x00000088 hash 'bdc1f053a4ad68f8'
0x00c03111 0x00000000 0x0000001f
MEM GPU: 76 ARM: 948 TOTAL: 1024
Starting start4.elf @ 0xfec00200 partition 0

NOTICE:  BL31: v2.3():v2.3
NOTICE:  BL31: Built : 10:40:51, Apr 21 2020

Note that commenting out the device_tree_address=0x1f0000, device_tree_end=0x200000 has no effect, so it's not a device tree setup issue.

We are in the process of pinpointing the exact revision of start4.elf that introduced this issue, but since we didn't expect such a major regression and now have to scramble to fix our UEFI firmware downloads as a result, it might be a while before we can do so...

[pbatard]

We also observe the same issue with the latest start.elf/fixup.dat on the Pi 3, where reverting to an older version "fixes" boot.

[pelwell]

I'll take a look.

[pelwell]

We've identified the problem - a fix will be released shortly.

[pbatard]

I appreciate that, thanks!

[pelwell]

The bug was introduced in bd816db, and caused your stub to not be loaded, the built-in default being used instead. As a result of the load-time optimisation in that release an external ARM stub would be loaded to address zero, which is fine for the VPU's memory system but our fread() implementation doesn't like it.

[popcornmix]

rpi-update firmware should have a fix for this.

[pbatard]

Many thanks for the quick turnover. Much appreciated!

We confirm that the new version of the firmware fixes the issue.

At this stage I have to ask, assuming that you already have some kind of CI/testing facility for your binaries: Are there any chances you could add the booting of the bl31.bin to the list of tests you run on the Raspberry Pi 4?

Provided you have the means to automate the copying of start files to a media, and then boot a Raspberry Pi 4 with it, we would greatly appreciate if you could add the boot process and boot files we described above as one of the tests you perform, with validation that you do get the NOTICE: BL31: v2.3():v2.3 message on the serial output (as this is a fixed notice that should always appears if the binary was able to boot).

Thanks again.

[pbatard]

I'm afraid that there is more to this issue than the initial problem we reported.

Whereas the commit from August 5th fixed the initial boot problem, where we couldn't even get the UEFI firmware to run, we are still being left with a major regression where GRUB (and thus Linux boot) is completely broken, whereas it worked absolutely fine with the start.elf/fixup.dat prior to bd816db.

The problem can be replicated as follows (this requires a few downloads):

1. Download the 1.30 version of the UEFI firmware from https://github.com/pftf/RPi3/releases/tag/v1.30/.
   Note that this zip archive includes start.elf/fixup.dat from bd816db.
2. Download the debian-10.6.0-arm64-netinst.iso from https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/
3. Create an MBR/FAT32 formatted media and extract both the content of the ISO and the content of the UEFI zip onto it
4. Insert the media into a Pi 3, and let it boot to the GRUB prompt (NB: you can press Enter during the UEFI prompt to speed this up)
5. At the GRUB menu, select Install or Graphical Install.

After a few seconds, you are presented with the Debian installation screen, as expected.

Now, replace start.elf/fixup.dat with a post August 5th version and repeat steps 4 & 5.

This time, booting to the installer either freezes or produces GRUB error error: failed to install/update FDT before forcing a return to the GRUB prompt...

Therefore, another behavioural change has been introduced between July 31st and August 5th, that broke the ability for GRUB ARM64 to run a Linux kernel...

[pbatard]

Just a quick update to mention that (after having to fiddle with the baudrate, since the post July 31st start.elf update also moved our UEFI serial baudrate, which, last time I checked, we do set up according to the official documentation, from 115200 to ~180000 bauds) that the new start.elf is triggering an assert here during our processing of the Device Tree to sort out psci.

So this explains why GRUB/kernel will complain about the Device Tree not being found and fail, since we don't pass something that we haven't been able to process.

I'm still trying to identify exactly what has changed, from our perspective, to trigger that assert. But it may take a while before I can provide more data.

Note that, if you want to get extended serial data that includes asserts, and after making sure that you set the baudrate to 180000, you can replace the release version of RPI_EFI.fd with the DEBUG version from https://ci.appveyor.com/project/pbatard/rpi3/builds/34965252/artifacts.

[markmi]

Another example context that looks to involve some of what is reported here is:

RPi4B 4GiByte FreeBSD (in gradual development) uses an armstub8-gic.bin port and a u-boot port as one way to boot. Such boots using 542aceb (2020-Jul-17). But, attempting booting via the later versions of the firmware tried hangs up with the rainbow screen showing and the end of the serial output looking like:

MESS:00:00:09.122140:0: brfs: File read: /mfs/sd/armstub8-gic.bin
MESS:00:00:09.125171:0: Loading 'armstub8-gic.bin' to 0x0 size 0x1700
MESS:00:00:09.131367:0: brfs: File read: 5888 bytes
MESS:00:00:09.251424:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:09.253939:0: Loading 'u-boot.bin' to 0x80000 size 0x8b9e8
MESS:00:00:09.260029:0: Device tree loaded to 0x4000 (size 0xbe0c)
MESS:00:00:09.269647:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:09.275976:0: uart: Baud rate change done...
MESS:00:00:09.278038:0: uart: Baud rate change done...
MESS:00:00:09.283427:0: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined

This is still true of 63b1922 (2020-Oct-08) firmware.

armstub8-gic.bin expects to be loaded at address zero and u-boot (as configured/patched) expects to internally reserve/avoid that area and to cause the area to show up as reserved. (armstub8-gic.bin is over a page in size.) With the problematical firmware versions, u-boot does not get far enough to start putting debug messages to the console (when built to do so).