Booting up Qualcomm SC8280XP with Linux on EL2

Tho goal

This short writeup should help bring up a Windows Dev Kit 2023 aka Project Volterra aka Blackrock, or other SC8280XP-based devices on EL2. It contains the paths I tried out, there may be others.

Running Linux on EL2 gives you /dev/kvm support which is great. The SC8280XP is higher performance and contains an arm-smmuv3 which is quite powerful. Linux needs to take control of it, though, to get PCIe devices working. When Linux is booted on EL1, Qualcomm's HYP firmware has control over this device, and its not accessible for Linux. That was a main hurdle to get PCIe operational again on EL2.

My setup

I boot Linux from an external SSD behind a USB3.0 hub on the WDK. UEFI is set to boot from USB first, Secure Boot is off, the internal Windows SSD is de-bitlockered. Otherwise it's unchanged. On the external SSD is Ubuntu 23.10 desktop image with grub, available here. This is no requirement, but it's the reference frame for the documentation:

• boot from USB
• grub as boot loader
• Ubuntu 23.10 (but this shouldn't matter)
• Screen via miniDP

Booting directly from internal SSD works, too. For testing purposes, I have a VM running (lxc-VM). This runs when everything went well.

What you need

1. slbounce

This of course is the core of it all. It is an efi module that invokes the secure launch mechanism with the only known and available file to secure launch, tcblauch.exe. It ends up running on el2. We need the newest version of slbounce, I would recommend to clone and build the repo. It also contains the dtbo files needed for changing the "normal" dtb to one that has no zap shader (needs hyp to load) and has the arm-smmuv3 activated and iommu-maps for the PCIe root ports. This enables us to use a plain vanilla device dtb for booting to EL2.

2. tcblaunch.exe

We need to copy this file from the Windows11 on ARM partition.

3. UEFI Shell

I have extracted bootaa64.efi from the ISO, renamed it to shellaa64.efi, copied it into the /EFI/BOOT directory where grub has put its boot loader. I also have wired up a menu entry configuration to bring up the EFI shell. This file goes to /etc/grub.d, needs owner root and must be executable. You need to edit the set-root ID to the UUID of your ESP, though. You obtain it with lsblk -f. 40_custom

4. Linux kernel 6.8.

With arm-smmuv3 active it won't work on earlier versions, since you need two IOMMU drivers active, arm-smmuv3 and arm-smmu. The jg/blackrock-v6.8.y will do nicely with the WDK, it is based on @jhovold's branch. Choose a config that fits you, and verify KVM support is enabled.

Setup

1. From slbounce, Windows, and the Linux kernel copy over the files we need to /boot/efi which should be on the ESP. It should look like this:

2. The prepared shellaa64.efi goes to /boot/efi/EFI/BOOT/:

3. The prepared 40_custom file goes to /etc/grub.d:

4. Integrate it into the grub setup: sudo update-grub

It is important that the menu entry for Linux 6.8 doesn't have a reference to the device tree. I achieve this by not setting the dtb- symlink. Normally, the dtb would be provided with this symlink. sudo ln -s /boot/dtbs/<kernelversion>/sc8280xp-microsoft-blackrock.dtb /boot/dtb-<kernelversion> By omitting this the update-grub script won't add the devicetree=/boot/dtb-<kernelversion>. If it's there, the dtb loaded with dtbhack would be overwritten.

5. Reboot the box. Select the EFI Shell menu. Now we need to find the right file system, in my case it was FS3:. A dir shows our files from /boot/efi. When we have the right filesystem, execute the following commands:

load slbounce.efi
dtbhack.efi your-dtb sc8280xp-symbols.dtbo sc8280xp-el2.dtbo
exit

This loads slbounce, which reports what its doing. dtbhack patches up the dtb and loads it into the RAM for the boot loader to find. After exiting you're back in the grub menu and select the 6.8 kernel (the one without the devicetree= statement) to boot.

Limitations

As it is always with this newfangled stuff, not everything works. Things known not to work are:

• ZAP Shader - needs hyp to boot
• Remote processors - needs hyp to attach
• Venus - needs hyp
• DP Altmode - needs pmic glink (remoteproc0) for some switchery
• all the other stuff that's not supported yet for the platform.

On a WDK these limitations are tolerable since it runs on main power anyway. And you have working thermal management with an EC-controled fan. That makes it viable as a VM server in this state. On the X13s you won't have battery charging.

Alternative setup - dedicated DTB adapted for EL2

You can omit dtbhack if the provided dtb already is for the EL2 configuration - no ZAP shader, arm-smmuv3 enabled. This is how I tested most of the time. In this case it is good to have the devicetree= statement in the grub menu entry. You need to use a dedicated EL2 branch like jg/el2-blackrock-v6.8.y in this case.