Recognize efi partition (0xef) as a valid boot

[psgreco]

Normal vfat partitions with id 0xef are not seen as valid boot partitions, changing it to 0xc makes it work, but make it more complicated for using generic images for different devices.

[lurch]

Related to #120 ?

[timg236]

If/when GPT + EFI this will just work. Not adding any workarounds before then

[psgreco]

I don't see how this is related to #120 and I'm not talking about gpt here.
I'm saying that there is another valid partition id that is not being considered as such

[pbatard]

0xef is the ESP type for MBR partitions (Remember that ESPs can exist on either GPT or MBR).

And FYI, not having ESP support currently makes vanilla Debian 10 installation on an SD card more troublesome than it should be (when using the official UEFI firmware as start.elf) because, ideally, you want to just extract the small Debian netinst ISO on the ESP (it's a couple hundred MB) and then, during the installation process, have the Debian partition manager recognize that there already exists an ESP and use it to install its EFI bootloaders, with the end result being that you now have an industry-standard UEFI installation that looks and is handled in exactly the same way as what you'd see on an x86 PC (with end-users prompted by a graphical GRUB prompt where they can easily choose/edit their Linux boot options and everything).

However, by not having the ability to boot from a FAT/FAT32 partition of type 0xef, we are forced to use 0x0c or 0x0e, and the current Debian partition manager has trouble recognizing partitions using this type as ESPs by default, which makes the whole Debian netinst installation a lot less user-friendly than it ought to be... This is for instance illustrated in the Preparation and Appendix C paragraphs of this guide on how to install vanilla Debian 10.x on a Pi 3.

Considering that we are seeing a lot of interest in the Pi3 and Pi4 UEFI firmware (which are now an official part of the EDK2), being able to use a formal ESP as the partition from which a Raspberry Pi can fetch its boot files from would be a great plus. And since ESPs can exist on either GPT or MBR partitioned disks, and the Pis already support MBR/0x0c and MBR/0x0e as partition types to load the boot files from then, provided that this is not dictated only by the SoC mask ROM, the only requirement to achieve that is adding MBR/0xef as another supported partition type.

[timg236]

The 2711 (and earlier) ROMs do not recognise 0xef as a FAT partition so on Pi3 it won't be possible to load bootcode.bin from that partition and on Pi4 it will prevent recovery.bin from being loaded.

Therefore Pi4 would still need a 0xc or 0xe partition for recovery.bin bootloader upgrades even if the SPI bootloader recognises 0xef as a FAT partition.

[pbatard]

Therefore Pi4 would still need a 0xc or 0xe partition for recovery.bin bootloader upgrades even if the SPI bootloader recognises 0xef as a FAT partition.

The information we have found, from reverse engineering the mask ROM from the Pi 4 Bcm2711 indicates that it should be able to boot a recovery.bin on either a naked FAT (a.k.a. "Big floppy"/"superfloppy"), FAT under MBR, or FAT under GPT, with either the ESP or Microsoft Basic Partition types.

Anyone can try the following, using the files from https://github.com/raspberrypi/rpi-eeprom/releases/tag/v2020.04.16-137ad:

• Create a MBR or GPT ESP partition on a SD card
• Format it as FAT32 and copy the recovery files
• Boot the Pi 4

Unlike what would be expected if the mask ROM did not have the capability to read a recovery.bin from an ESP, you get a solid red screen. And if you simply rename or remove recovery.bin from that partition (and provided you have a recent eeprom), you get the HDMI diagnostics.

So, at the very least, the Bcm2711 ROM has code that is able to list FAT content from an ESP, be it a GPT or MBR one, and, depending on the presence or absence of a recovery.bin there, run a different code path.

In other words, from what we are seeing, it does seem like at least the 2711 does recognise 0xef as a FAT partition, as well as GPT ESPs, and it would seem that it's the recovery.bin code itself that is lacking that ability to handle such partitions in order to continue execution.

Maybe we are missing something here, but if you could shed some more light on this, and why we are observing the results above on the Pi 4, we would appreciate it.

[timg236]

The ROM doesn't explicitly handle this partition type. For GPT it looks for EFI or BasicData GUIDs and for MBR it will failover to assuming FAT if it doesn't find a recognized type.

The Pi4 bootloader and start.elf support GPT (USB only) in the USB mass-storage updates and when the changes are made to support this in start.elf SD file-system it would be reasonable to recognize 0xef as another FAT partition. It should be safe because very few Pi users will have a partition of this type in addition to the regular FAT boot partition containing start.elf

[pbatard]

For GPT it looks for EFI or BasicData GUIDs and for MBR it will failover to assuming FAT if it doesn't find a recognized type.

That makes sense. Thanks for clarifying.

when the changes are made to support this in start.elf SD file-system it would be reasonable to recognize 0xef as another FAT partition.

I'm understanding this to mean that you are considering adding support for MBR/0xef partitions in the future, which is good news! Of course we understand that this is a low priority change, but you may want to remove the wontfix label if that no longer applies... Thanks again.

[timg236]

Yep, it will be done. Probably in the next couple of weeks, just need to get the current rather large USB boot changes out of the door or more precisely, off a topic branch and into mainline!

[timg236]

This should be fixed by e87bc4d

The binaries are available in this release which can be flashed using the Raspberry Pi Imager
https://github.com/raspberrypi/rpi-eeprom/releases/tag/v2020.05.26-137ad

[pbatard]

From my testing, this seems to work great for USB drives, so many thanks for following up on this!

However, I can't see to get the expected results from SD.
Is this also supposed to work for SD boot?

On SD, I can see start4.elf being loaded from an EFI System Partition now (be it MBR/0xef or GPT/Generic Data or ESP), but it doesn't seem to go further than that...

You should be able to validate this by performing the test below

Prerequistes

• Fetch the current start4.elf, fixup4.dat and bcm2711-rpi-4-b.dtb from https://github.com/raspberrypi/firmware/tree/master/boot
• Fetch a Raspberry Pi 4 TF-A binary (bl31.bin) from https://github.com/tianocore/edk2-non-osi/raw/master/Platform/RaspberryPi/RPi4/TrustedFirmware/bl31.bin.
• Create the following config.txt:

  arm_64bit=1
  enable_uart=1
  uart_2ndstage=1
  enable_gic=1
  armstub=bl31.bin
  

Test

1. Create a standard MBR FAT32 partition on a SD card with type 0x0c or 0x0e
2. Copy all of the files above
3. Check the serial output. You will see:

recover4.elf not found (6)
recovery.elf not found (6)
Read start4.elf bytes  2272832 hnd 0x0000000e hash 'a439c7fcf5a16d52'
Read fixup4.dat bytes     5405 hnd 0x0000000d hash '75312421da0e036f'
0x00c03111 0x00000000 0x0000001f
MEM GPU: 76 ARM: 948 TOTAL: 1024
Starting start4.elf @ 0xfec00200 partition 0

NOTICE:  BL31: v2.3():v2.3
NOTICE:  BL31: Built : 10:40:51, Apr 21 2020

4. Now change the partition type to 0xef and try to boot again.
   This time the last 2 lines are missing, meaning that bl31.bin never ran (since the NOTICE lines come from early output).

Considering that the same test (MBR/0xef) also works fine from USB, it seems that there could still be some elements missing when it comes to SD...

[timg236]

Unfortunately, the SD card drivers in start.elf are from a different legacy stack which will need updating for this to work. That will be available in rpi-update but it's not started yet.

[pbatard]

Understood.

I'm just glad to hear that this is a known limitation and that you are working on removing it, even if that may take a while. The newly added USB support for ESP and GPT works beautifully as far as I'm concerned, so keep up the good work!

[timg236]

Great, thanks for the test. I'll mark this as closed but will post an update here as soon as the SD firmware changes are done. Feel free to nag if it's not done by the end of next week.

[timg236]

@pbatard ok, it took slightly longer than expected to get around to doing this but I've now updated the GPU firmware to support booting from a pure GPT partition table on sd-cards.

The linked firmware "works for me" but is not merged into a release branch yet, if you are able to try this (with the latest stable bootloader) then that would be great!

https://drive.google.com/drive/folders/1Qn6uAkgYntUr6Pu7OIgWxnHXEv_FeHJw

[pbatard]

Thanks a lot for following up on this.

I have just tested the files you linked too, and I can confirm that direct boot from a MBR ESP (type 0xef) off an SD card now works as expected.
I have also tested MBR ESP from USB for good measure, just to confirm that there wasn't a regression.

In other words, just like we requested, it is now possible to boot the Pi 4 from just an ESP, be it GPT or MBR, or be it located on SD or USB. This is awesome, and it should really help with UEFI environments, as this allows the installation of OSes in UEFI mode, while ensuring that the Pi 4 boot files, that reside on the ESP, are preserved through the installation process.

Once we have fixed a few open items, this will for instance enable the installation of vanilla Debian ARM64 as easily as if you were installing it on a UEFI PC, through a single drive where you simply extracted Debian's mini.iso or netinst.iso content into a small ESP. So I really can't thank you enough for helping make the goal of a more user-friendly OS installation experience a reality! 😄

[pbatard]

Oh and just to confirm, I have tested GPT ESP from SD card as well, and saw no issue.

[timg236]

@pbatard Thanks for the test, it's been merged so should appear in rpi-update before too long.