U-Boot NVMe broken on Ventura (13.0 beta) firmware

[marcan]

I upgraded to first beta of macOS Ventura on my MacBook Air (8/8/512) with M1 and have problem with booting into Asahi.
Upgraded from 12.4
One macOS and one Asahi.
The default boot volume after update was macOS.

Asahi boot stuck at U-Boot with:

Device 0: unknown device
Failed to boot from ESP with UUID <UUID>
Trying to boot from USB ...

Device 0: unknown device
Failed to find boot partition 'run distro_bootcmd' to boot from any partition
=>

Originally posted by @ivabus in AsahiLinux/asahi-installer#100 (comment)

[marcan]

@ivabus When did you install Asahi? Was it quite early after the alpha release? Early U-Boot builds had an exceedingly short timeout and didn't work on 4T+ machines, and I wouldn't be surprised if a firmware update made it slower to the extent where it stopped working on smaller models too. Though we definitely upgraded that (if you ever pacman upgraded your Asahi install you should have it).

[marcan]

@kettenis @svenpeter42 I think we might have to consider improving U-Boot's RTKit/NVMe support to stop making so many assumptions about the bootup flow and work more like m1n1's implementation; I think this stuff is going to keep biting us otherwise and it's no fun for users to end up with broken boot :/

[ivabus]

I installed Asahi in early May. I have a screenshot of it running 5.17.0-rc7-asahi-next-20220310-5-2-ARCH. But I upgraded by pacman sometimes after

[ivabus]

Also nvme scan command in U-Boot shows no devices

[ivabus]

If necessary, I can provide additional information.

[kettenis]

@marcan so m1n1 works fine with the new firmware? I'll take a look at what m1n1 does different then.

[jannau]

Issue seems to be missing sart support in u-boot. ASC trace stop after sending 0 as crashlog buffer dva.

[gjsman]

I’d like to provide some information. I installed macOS 13 Beta on a separate APFS volume (Apple official instructions) alongside my main macOS 12 Monterey, then ran Asahi Linux install from the Beta. I get this image on boot:

Maybe I read @marcan ’s tweets wrong but I thought firmware wasn’t persistent? Unless it was my mistake to install Asahi from the 13 Beta instead of the Monterey partition.

[gjsman]

Update: I uninstalled Asahi Linux, then reinstalled from my macOS 12 Partition. Same issue.

[marcan]

No need to provide more information, we know this is outright broken. U-Boot's NVMe implementation tried to be too smart for its own good in the name of simplicity and it backfired. m1n1's driver works fine.

Some firmware is systemwide, like NVMe, for obvious reasons. If you install a beta macOS then that updates your systemwide firmware to that version. Systemwide firmware is backwards compatible if the driver is doing things properly, but U-Boot's isn't.

[jannau]

A fix is under development and the proof of concept already restores NVMe function in u-boot. There is still work to be done to make it a proper upstreamable commit.
With u-boot fixed there's another issue though. The kernel SMC driver crashes SMC which will reboot the system after 30 seconds.

[jannau]

u-boot and kernel fixes are tested
u-boot: https://github.com/jannau/u-boot-1/tree/macos13-nvme
kernel smc driver: jannau/linux@6bf3300

Following steps will be required to fix broken installations:

• provide fixed kernel and u-boot packages for AsahiLinux
• prepare a fixed m1n1/boot.bin with updated u-boot and a device tree without SMC

On user side them1n1/boot.bin on the EFI system partition needs to be replaced. That will result in bootable AsahiLinux installation with reduced functionality (no poweroff, no battery info or charger control, …). User need to upgrade the system with pacman -Syu an reboot. After the reboot the system is back to full functionality.

[stevena1844]

@jannau are we able to just download a fixed boot.bin then mount the EFI partition and copy over then reboot and upgrade using pacman? or do we need to somehow create a new boot.bin using the u-boot from git? If the latter how do we do that please? I really appreciate your help I had Asahi going for only two days or so then decided to put Ventura on here and messed it up.

[jannau]

@stevena1844 fixing this is unfortunately a little bit more complicated since there are two issues. We will provide the necessary binaries and a description to restore affected systems when the fixes are in place. I can't give an ETA because @marcan needs to handle most of it.

The second issue is that the SMC driver in our current kernel causes the macOS 13 beta SMC firmware to crash. This results a reboot after a minute or so. We can disable SMC easily with a modified device tree but SMC is required for turning the wifi device on. Devices which require wifi for connectivity will not be able to download fixed u-boot and kernel packages. Packages have to be downloaded beforehand and installed manually.

If you don't need wifi for connectivity https://www.jannau.net/asahi/boot.bin_2022-06-12_macos13_nvme_no_smc is a preview of the fixup boot.bin. Replace the existing m1n1/boot.bin on the EFI system partition (FAT32). This should be possible from macOS or macOS Recovery. The system will have reduced HW support: no wifi, no power-off, no RTC, no battery info or charger control, …

[sassa7777]

@jannau When do you think the fix will be completed?

[gjsman]

Curious question: How would said firmware that is persistent be downgraded and where is it stored?

[marcan]

System firmware is stored in NOR flash and possibly the iBootSystemContainer partition. Downgrading requires a full DFU erase restore (i.e. complete machine wipe). For fairly obvious reasons, even if you could downgrade iBoot et al somehow (which you can't), SSD/NVMe firmware in particular is impossible to downgrade without a full wipe, on any SSD, for the same reason you can't downgrade a database server or an Android version and keep your data (without the chance of everything exploding).

[ubenmackin]

On user side them1n1/boot.bin on the EFI system partition needs to be replaced. That will result in bootable AsahiLinux installation with reduced functionality (no poweroff, no battery info or charger control, …). User need to upgrade the system with pacman -Syu an reboot. After the reboot the system is back to full functionality.

I just gave this a spin.

1. Did a fresh install of Asahi Linux to my M1 MacBook Air.
2. Copied the boot.bin from https://www.jannau.net/asahi/boot.bin_2022-06-12_macos13_nvme_no_smc to my EFI partition, replacing the existing one.
3. I rebooted into Asahi Linux, which worked as expected, with the missing features.
4. While plugged into Ethernet, I ran the pacman command to update things (there was 770 updates to apply)
5. On the next reboot, I got the failed to boot error again.

So maybe there is more to it, or I misunderstood what the steps to do would entail.

[marcan]

Package updates have not been pushed yet.

[7urtles]

Same issue. Updated to Ventura, wiped all volumes/containers, re-installed Monterey.
I've read all the above. Downloaded the above boot.bin. Waiting for pushed update confirmation to test.

[jannau]

I posted the fixup instructions into the slightly wrong ticket, see AsahiLinux/asahi-installer#100 (comment)

[jannau]

Updated packages are pushed to the asahi-dev package repository

[7urtles]

I posted the fixup instructions into the slightly wrong ticket, see AsahiLinux/asahi-installer#100 (comment)

Worked. Thank you

[kettenis]

The fixes are now in mainline U-Boot and will be in U-Boot 2022.07.

[ivabus]

Was the installer rebuilt after fixes?
Because I need to fix m1n1/boot.bin every time I reinstall.

[jannau]

@ivabus the installer has not yet been refreshed.

[jannau]

The installer has now fixed packages and the asahi repo has fixed packages as well. Who switched to asahi-dev can now switch back to asahi.