How to boot from external drive?

[moham96]

Hi,
I want to boot from my external SSD, I tried to install asahi as UEFI-only to the internal nvme drive, then on my external SSD I created a fat32 partition and set it as boot and a linux partition then unpacked the asahi-plasma-20220321-1.zip archive and copied the boot folder to the fat32 partition and used dd to burn the root.img file to the linux partition, but I'm still unable to boot from external (u-boot doesn't automatically try to boot from external SSD) I tried to manually load the bootaa64.efi file and bootefi it but it just drops me to grub rescue terminal.

Regards

[leifliddy]

As long as your fat32 partition contains /efi/boot/bootaa64.efi (case insensitive) it should work.

That's what u-boot is looking for. So you could either boot a usb stick containing a single fat32 partition (which contains /efi/boot/bootaa64.efi)
Or better yet, just create a usb stick with a small fat32 efi partition (containing /efi/boot/bootaa64.efi) and then create a separate ext4/xfs/fs2fs...etc root (and perhaps a boot) partition as well.

I have a project that will create a bootable usb drive for the M1.
https://github.com/leifliddy/asahi-fedora-usb

There's other projects out there like this one that show you how to make a bootable m1 usb stick.
https://git.zerfleddert.de/cgi-bin/gitweb.cgi/m1-debian/
**too much manual work IMO and it only creates a read-only usb drive.
**however, it's definitely educational....that's how I learned how it worked.

Anyways, once you have your usb drive ready, interrupt u-boot and enter the following command to boot it.
run bootcmd_usb0

[moham96]

@leifliddy well, as i said in the OP i have made a bootable usb drive using the official asahi rootfs
But grub fails to start and tries to load from a hardcoded uuid

[leifliddy]

@moham96 Can you post the content of grub.cfg on the external drive?
I "think" Asahi Linux stores it under /boot/grub/grub.cfg

A fat32 partition only contains 8 hex characters, so if you have a single fat32 partition on that drive then grub shouldn't be looking for a uuid containing 33 hex characters.

Also, if you boot to Asahi Linux and then plug that external drive in, could you please post the output of

lsblk /dev/sda
blkid /dev/sda*

said in the OP i have made a bootable usb drive using the official asahi rootfs

I understand...but knowing how a usb drive boots with the M1 system and creating one from scratch will help you understand things better.

[moham96]

@moham96 Can you post the content of grub.cfg on the external drive? I "think" Asahi Linux stores it under /boot/grub/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod ext2
search --no-floppy --fs-uuid --set=root 725346d2-f127-47bc-b464-9dd46155e8d6
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-/home/marcan/asahi/git/asahi-alarm-builder/images/asahi-plasma/root.img' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod ext2
	search --no-floppy --fs-uuid --set=root 725346d2-f127-47bc-b464-9dd46155e8d6
	echo	'Loading Linux linux-asahi ...'
	linux	/boot/vmlinuz-linux-asahi root=UUID=725346d2-f127-47bc-b464-9dd46155e8d6 rw  loglevel=3 quiet
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initramfs-linux-asahi.img
}
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-/home/marcan/asahi/git/asahi-alarm-builder/images/asahi-plasma/root.img' {
	menuentry 'Arch Linux, with Linux linux-asahi' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-asahi-advanced-/home/marcan/asahi/git/asahi-alarm-builder/images/asahi-plasma/root.img' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 725346d2-f127-47bc-b464-9dd46155e8d6
		echo	'Loading Linux linux-asahi ...'
		linux	/boot/vmlinuz-linux-asahi root=UUID=725346d2-f127-47bc-b464-9dd46155e8d6 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-asahi.img
	}
	menuentry 'Arch Linux, with Linux linux-asahi (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-asahi-fallback-/home/marcan/asahi/git/asahi-alarm-builder/images/asahi-plasma/root.img' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod ext2
		search --no-floppy --fs-uuid --set=root 725346d2-f127-47bc-b464-9dd46155e8d6
		echo	'Loading Linux linux-asahi ...'
		linux	/boot/vmlinuz-linux-asahi root=UUID=725346d2-f127-47bc-b464-9dd46155e8d6 rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-asahi-fallback.img
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

note that the boot process is stopping before loading the grub.cfg so it's probably not the culprit

A fat32 partition only contains 8 hex characters, so if you have a single fat32 partition on that drive then grub shouldn't be looking for a uuid containing 33 hex characters.

The uuid grub is searching for is not for the fat32 partition but the asahi rootfs partition ext4 (/dev/sdc2 below)

Also, if you boot to Asahi Linux and then plug that external drive in, could you please post the output of

lsblk /dev/sda
blkid /dev/sda*

This is the output from an x86 machine
blkid

/dev/sdc: PTUUID="6c01a31c-cd6e-45a6-80b4-2fccdff95f7d" PTTYPE="gpt"
/dev/sdc1: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="E96A-4C7D" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="efi" PARTUUID="725346d2-f127-47bc-b464-9dd46155e8d6"
/dev/sdc2: LABEL="asahi-root" UUID="725346d2-f127-47bc-b464-9dd46155e8d6" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="asahi" PARTUUID="ceee9388-cf4f-481b-83b7-f5a1ef03e76b"
/dev/sdc3: LABEL="misc" UUID="63D7-7EA3" BLOCK_SIZE="512" TYPE="exfat" PTTYPE="dos" PARTLABEL="misc" PARTUUID="4ecb2fab-ee82-4940-8e6e-7e05099fa25c"
/dev/sdc4: LABEL="misc2" BLOCK_SIZE="512" UUID="21FCF7D944E34400" TYPE="ntfs" PARTUUID="6c8c8c2c-ec03-45f4-b79d-483c063d0650"

You can see that /dev/sdc2 has the correct uuid that grub is searching for but unfortunately it can't find it for some reason
lsblk:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sdc      8:32   0 931.5G  0 disk 
├─sdc1   8:33   0   100M  0 part /media/EFI
├─sdc2   8:34   0  19.5G  0 part /media/asahi-root
├─sdc3   8:35   0  48.8G  0 part /media/misc
└─sdc4   8:36   0 863.1G  0 part /media/misc2

said in the OP i have made a bootable usb drive using the official asahi rootfs

I understand...but knowing how a usb drive boots with the M1 system and creating one from scratch will help you understand things better.

[leifliddy]

I'm gone for the weekend, but I can try to replicate what you did early next week.
But in the meantime I would try this.
Create a mount directory ie mnt_drive
then

mount /dev/sdc2 mnt_drive
mount /dev/sdc1 mnt_drive/boot/efi
arch-chroot mnt_drive
rm -rf /boot/efi/m1n1 (you don't need m1n1 on an external drive)
/usr/bin/update-grub
exit (or ctrl+d)
umount  mnt_drive/boot/efi
umount  mnt_drive

and see if that works.

note that the boot process is stopping before loading the grub.cfg so it's probably not the culprit

That fact that it's looking for the uuid of the root partition means that grub is loading.
Plus you can see on your screenshot that grub is loading, by the welcome to grub prompt output.
Actually, that brings up a good point
At the grub rescue prompt you could try to manually boot that drive with:

ls 
# this should list the partitions that grub sees for your device, do you see (hd0,gpt2) in the output? 
set root=(hd0,gpt2)
linux /boot/initramfs-linux-asahi.img
initrd /boot/vmlinuz-linux-asahi
boot 

You can use the grub rescue shell to diagnose quite a few different boot related issues. As I said, I can help you further with that next week.

[leifliddy]

/usr/bin/update-grub "should" work because you're mounting an actual disk partition, it if doesn't complete try running this script instead to update grub.
https://raw.githubusercontent.com/AsahiLinux/asahi-alarm-builder/main/scripts/base/30-grub.sh

Also, make sure that your /etc/fstab has that E96A-4C7D uuid for the efi partition.

[moham96]

Ok so i solved the problem by running usb reset in the u-boot prompt before running run bootcmd_usb.
But the issue is sporadic and unpredictable, sometimes i don't need to reset the usb and u-boot boots directly to grub even without interrupting it's normal sequence, sometimes i need to run usb reset once to solve the issue, sometimes the usb reset command needs to be run multiple times (three times at least in my testing) to make grub capable of finding the partition.
I don't know if i should close this issue or leave it open to track the usb reset issue.
Maybe @marcan can enlighten us a bit about the this.
Regards

[JeremiahCornelius]

After diligent efforts, I made a USB-3 drive containing my Asahi/Debian install. I also have a minimal root install on the NVMe of my Macbook Pro 16 M1, with working uBoot and GRUB.

I successfully added the USB install to my grub configuration, but this fails to boot during initramfs. After a lot of investigation, the UUID of the root filesystem can't be located. This is because USB block storage is not loaded by initramfs. Confirmed with blkid not seeing this, and no /dev/sda present.

I chrooted from the NVMe install, and added all the right usb and scsi modules to an updated initramfs. This hasn't fixed the issue, despite confirming modules loaded with cat /proc/modules from the (initramfs) busybox prompt.

Yes, I earlier tried an fstab on the USB drive, with both LABEL and /dev/sda1 for the root mount. This was before understanding the cause was a failure to start the device and register /dev entries.

Is there any reasonable solution to this, that I haven't yet tried? I believe this would be immensely useful to others, if it culd be made workable.

[leifliddy]

@JeremiahCornelius I have a project that builds a bootable usb drive
https://github.com/leifliddy/asahi-fedora-usb

You could build it on either a x86_64 or aarch Fedora system/VM. For testing purposes it might be worth your time to build that drive and see if boots. That would help isolate the issue a bit more.

I include these modules in initramfs with my project:
https://github.com/leifliddy/asahi-fedora-usb/blob/main/mkosi.skeleton/etc/dracut.conf.d/initial-boot.conf

Interestingly enough I've never tried adding a grub entry (on the internal drive) for the usb drive.
I'm away from home atm (without my m1 macbook), but I can try that out tomorrow and see if it would boot like that.

I would try running these uboot cmds (with the usb drive plugged in obviously)

usb reset
usb info
usb storage

....to ensure that uboot is recognizing the usb drive.

If the device is recognized, then try booting it directly with

env set boot_efi_bootmgr
run usb_boot

Also, how is your usb drive partitioned? And what filesystem(s) are you using?

Actually, it might be helpful if you listed the instructions on how you created the drive.
If you did that -- I could you troubleshoot the issue.

[JeremiahCornelius]

@leifliddy Thanks for these hints. I will probably try the build scripts. I'd love to use the native speed of the M1 versus cross compiling.

My build steps were pretty simple. I made an install on my Mac NVMe partition per https://git.zerfleddert.de/cgi-bin/gitweb.cgi/m1-debian/

Then I formatted a USB-3 2TB as GPT, single part Ext4 FS and tar copied the contents over. Chrooted into it, changing the fstab and fixing up /etc/default grub, updating grub (for completeness) and building a proper update-initramfs -k all -u. On the host I ran os-prober and updated the grub on the host. I've done similar with X86_64 and numerous aarch64 boards many, many times.

I may try adding a little FAT partition for the EFI bits, and loading straight from uBoot to the native grub on the USB, but that's got a manual intervention for every boot, and the uBoot console is HORRIBLE for picking up keystrokes, with a minuscule typeface!

[leifliddy]

Agreed, I'd be more convenient to boot from the main grub menu. I don't see why that wouldn't work. I'll report back on that tomorrow. Someone should really create an automated USB installer for Debian so you can easily create (reproducible) builds vs manually typing in a ton of commands.

[leifliddy]

Booting the usb drive from the main grub menu works perfectly for me.
I've never really used Debian before -- if someone could automate the creation of a Debian (rw) usb drive, then I could help troubleshoot what the issue is.

I don't like these instructions at all: https://git.zerfleddert.de/cgi-bin/gitweb.cgi/m1-debian/
You're basically create a a usb drive with a 2GB vfat partition and then untarring a debian live rootfs on top of that. I'm wondering whether you could even create persistent changes to a drive containing a live filesystem like that.

Surely there must be someone who's following this project who knows Debian well enough to automate the creation of a usb drive that contains an efi + boot + root partition -- that you could use and interact with on the same level as the internal drive.

[JeremiahCornelius]

SUCCESS.
In uBoot the storage device is correctly ID'd with all:
usb reset
usb info
usb storage

Invisible to the initramfs. I built a new one, adding modules for phy_apple_atc and typec, which seemed relevant.
This also did not work.

After messing around with system-trace to try and find the relevant module, I just went brute-force, and built an initrd with all the loaded modules from my working volume install. :-)

The compressed initrd image is about 64 Mb, and I have 2TB to burn. This booted and brought up the graphical target, from the regular grub! Unfortunately, the Broadcom firmware modules didn't have necessary firmware available in initramfs, so network doesn't come up. I will exclude all network device modules in a new initrd build, and expect success on that front.

[leifliddy]

The only modules you should need for a usb drive boot are the ones listed here:
https://github.com/AsahiLinux/asahi-scripts/blob/main/dracut/dracut.conf.d/10-asahi.conf
and vfat loop uas usb-storage

Do you not have the all_firmware.tar.gz file on your internal esp partition?
That contains the broadcom firmware.

asahi scripts will extract the firmware from the esp
https://github.com/AsahiLinux/asahi-scripts/blob/main/asahi-fwupdate

and will use a dracut module to load the firmware at boot time
https://github.com/AsahiLinux/asahi-scripts/blob/main/dracut/modules.d/99asahi-firmware/load-asahi-firmware.sh

[JeremiahCornelius]

These are all helpful information, but I have been unsuccessful with a minimal module set for a working USB-C boot. That's OK. I did trim the Broadcom modules from my initrd, and have functioning network with Network Manager under Gnome 44 on Debian Sid.

Emboldened, I converted the root volume to btrfs with separate @ and @home subvols. This requires some hand editing of /boot/grub/grub.conf to work while keeping a minimal Asahi Debian on the nvme as a rescue partition with chroot script. LUKS is a bridge too far at this stage, but I use ecryptfs for my home directory on other portables, and was able to get this working in short order.

I exported my dpkg list from a UTM VM, along with a script to reproduce all installed Flatpak applications. With software in place, I untarred my home from the UTM backup, and replicated full user environment.

Overall I'm very happy with the result. I added an Asahi-styled Plymoth splash, suppressed messages on my Linux cmdline, and fixed up Grub with the highly graphical Tela theme that ties in nicely.
https://github.com/chadmed/asahi-plymouth
https://github.com/vinceliuice/grub2-themes

Asahi is a work-in-progress, but I now have a 2TB machine, with minimal impact to my running Sonoma system.