Can't live boot Talos on rpi 4 "/dev/loop no such device"

[OGKevin]

Bug Report

When using PXE to netboot a RPI, the live boot fails using the kernel from the RPI 4 image and initramfs.

Talos fails to mount rootfs with the error that "/dev/loop" does not exist.

The vmlinuz for arm just reboots the PI without any output.

Description

1. Extract metal-rpi_4-arm64.img.xz
2. Edit config.txt and cmdline.txt, the important change here is initramfs initramfs-arm64.xz followkernel in config.txt
3. netboot
4. it fails to boot due to failure to mount /dev/loop0

Logs

Environment

• Talos version: 0.10.2
• Kubernetes version: [kubectl version --short]
• Platform: Metal RPI4

I spent a few hours doing some research and so far I could not find the issue, feel free to point me in the right direction if I'm missing something.

[andrewrynhard]

@OGKevin thank you for reporting. My suspicion is that we are missing something in the arm64 config. Looks like blkdev loop is enabled, so I am not quite sure what else it could be. Will need to dig into this. Here is the kernel config for reference: https://github.com/talos-systems/pkgs/blob/master/kernel/kernel/config-arm64

[OGKevin]

The readme of the pkgs repo is a little too short for my taste :D. I've managed to get the menuconfig to work and play around in the config but after running make TARGETS=raspberrypi-firmware the result seems to be only available in docker build cache right? Any tips/docs/posts/pointers etc on how to get the result out of the docker build cache?

[OGKevin]

So using the ubuntu kernel from https://ubuntu.com/download/raspberry-pi i've managed to run init. However, now the latest error is "failed to open meta file". And some */bridge-nf-call-ip6tables no such file or directory errors.

So phase 2 of the boot process fails.

[andrewrynhard]

So using the ubuntu kernel from https://ubuntu.com/download/raspberry-pi i've managed to run init. However, now the latest error is "failed to open meta file". And some */bridge-nf-call-ip6tables no such file or directory errors.

So phase 2 of the boot process fails.

Ok, so we are missing something in the kernel config. Though I don't have any idea where to start looking just yet.

[OGKevin]

Would a diff in kernel config from the ubuntu vs talos shed some light? The rpi-os kernel also fails to init.

[andrewrynhard]

Would a diff in kernel config from the ubuntu vs talos shed some light? The rpi-os kernel also fails to init.

Certainly could help, but may take time going through the diff.

[OGKevin]

Here is the ubuntu kernel config: https://gist.github.com/OGKevin/b987cfce1401dbc90162e33763879bf7

The diff is huge indeed 😞

Regarding loop, CONFIG_BLK_DEV_CRYPTOLOOP is set to m in the ubuntu config where its not set in talos.

[smira]

@OGKevin what it might be is that kernel cmdline is missing initramfs=initramfs.xz arg, it's required for UEFI boot.

It might be early in the boot log when UEFI stub says it can't load initrd image.

[OGKevin]

hmm i'm not booting via EUFI if not mistaken. Im booting via EEPROM somewhat as explained here: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md except without the NFS root and with the talos metal boot folder & initram.

So my boot folder is from metal-rpi_4-arm64.img.xz

config.txt

initramfs initramfs-arm64.xz followkernel
#kernel vmlinuz-arm64
# See https://www.raspberrypi.org/documentation/configuration/config-txt/
# Reduce GPU memory to give more to CPU.
gpu_mem=32
# Enable maximum compatibility on both HDMI ports;
# only the one closest to the power/USB-C port will work in practice.
hdmi_safe:0=1
hdmi_safe:1=1
# Load U-Boot.
#kernel=u-boot.bin
# Forces the kernel loading system to assume a 64-bit kernel.
arm_64bit=1
# Enable the primary/console UART.
enable_uart=1
# Disable Bluetooth.

cmdline.txt

talos.platform=metal init_on_alloc=1 slab_nomerge pti=on talos.board=rpi_4 talos.shutdown=poweroff talos.config=http://192.168.1.201:8080/config.yam

Are you suggesting I try PXE booting via EUFI using https://github.com/pftf/RPi4 with /EFI/BOOT/BOOTAA64.EFI from metal-rpi_4-arm64.img.xz ?

I did however try and add the initramfs as you suggested but that also did not help.

[OGKevin]

So using https://github.com/sschaeffner/pipxe4/releases/tag/v20210515 I was able to netboot talos. Which solve the initial issue raised in this issue.

What I now need to figure out is how to do a 2 step boot process so I can achieve blank disk booting:

1. insert blank SD card
2. net boot pipxe4
3. ipxe boot talos via pipxe4

But this is outside the scope of Talos I would say.

Putting pipxe4 on sd card, I was able to successfully netboot Talos. If I figure this out I'm happy to contribute documentation.

[smira]

great that it works with pipxe!

I think this goes through UEFI boot somewhat, as it's using iPXE UEFI build which will launch Talos Linux kernel via EFI stub (in the same way as if it was loaded from disk via UEFI firmware).

[OGKevin]

Yes! Ok I've succeeded netbooting talos with a blank sd card 🎊.

tl;dr
1a. update EEPROM to latest

1. grab pipxe4 put it on a sd card
2. boot from this sd card and hit escape
3. Change the boot order and set httpv4 boot as first with a hardcoded URL where to find efi/boot/bootaa64.efi on the network which comes from pipxe4 zip/image hit save.
4. grab the sd card now and extract it to a tftp boot folder, if you had git initialized git should see a diff in RPI_EFI.fd
5. EEPROM netboots and grabs pipxe4 from the tftp boot folder
6. due to the bootorder change, EUFI does httpv4 boot and grabs the .efi file
7. This then start iPXE boot
8. Now boot Talos using iPXE with the initramfs and vmlinuz for arm that can be found in the releases.

Is a little more detailed doc maybe needed/welcome/good idea on https://www.talos.dev/docs/v0.10/single-board-computers/rpi_4/ ?

[andrewrynhard]

@OGKevin We would love docs on this! Please do!