"macOS style" bootable ISOs can't boot in UTM virtual machines

[steven-michaud]

I can't get "macOS style" bootable ISOs to boot in any kind of UTM virtual machine -- "macOS 12+" or "Other".

In a macOS 12.3.1 VM (created and run on an Apple Silicon Mac Mini running macOS 12.3.1) I add a bootable ISO as an external drive, then "move it up" (I assume this is to put the drive earlier in the boot order). Then when I start the VM it chews CPU indefinitely, without ever actually booting.

On an Intel machine (a MacBook Pro) running macOS 12.3.1 I attempt to create a VM of type "Other" and set the "Boot ISO Image" to a bootable ISO. When I start the VM it fails to boot from the ISO -- it falls through to the "UEFI Interactive Shell".

This is using UTM 3.1.5, downloaded from https://mac.getutm.app/.

Is this a bug? By design? Apple's fault? Is there anything UTM can do about it?

For reference, this is how I create my "macOS style" bootable ISO. I can use VMware Fusion (on an Intel Mac) to create a VM from it, so I know it really is bootable.

1. Download the full installer:

   softwareupdate --fetch-full-installer --full-installer-version 12.3.1

2. Create a sparse image to work with:

   hdiutil create -o blah -size 15G -volname blah -layout SPUD -type SPARSE -fs HFS+J

3. Mount this sparse image:

   hdiutil attach blah.sparseimage -noverify -mountpoint /Volumes/blah

4. Use the full installer's createinstallmedia to write a bootable installer to the sparse image:

   sudo /Applications/Install\ macOS\ Monterey.app/Contents/Resources/createinstallmedia --volume /Volumes/blah

5. Detach the bootable installer:

   hdiutil detach -force /Volumes/Install\ macOS\ Monterey

6. Create the bootable ISO:

   hdiutil convert blah.sparseimage -format UDTO -o macOS_Monterey_12.3.1_Installer

7. Rename the ISO:

   mv macOS_Monterey_12.3.1_Installer.cdr macOS_Monterey_12.3.1_Installer.iso

[watersb]

I can't get "macOS style" bootable ISOs to boot in any kind of UTM virtual machine -- "macOS 12+" or "Other".

@steven-michaud - Thanks for your very interesting HookCase tool!

The articles on Eclectic Light regarding Apple Silicon Boot might be of interest here.

Back to UTM, there is some interesting weirdness here when I tried to add a disk image as an External Drive. I kept the image format as SPARSE, and it appeared an an internal drive to the macOS guest VM, but it was not recognizable in that guest.

Converting it to another format, such as UDRW or UDTO, the disk image did not appear at all.

Just for fun, I wondered what qemu-img would think about these disk images. The disk image used by UTM itself for the macOS VM is not exactly a standard diskarbitrationd style image, although it does mount via hdiutil:

% hdiutil attach /Users/bwaters/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img 
There may be a problem with this disk image. Are you sure you want to open it?
[Opening this disk image may make your computer less secure or cause other problems.]
Attach? (Y/N) y

You get the standard Apple Silicon disk volume layout, but I didn't poke around on this image, given the warnings...

% hdiutil info
framework       : 623.100.1
driver          : 623.100.1
images          : 1
================================================
image-path      : /Users/me/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img
image-alias     : /Users/me/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img
shadow-path     : <none>
icon-path       : /System/Library/PrivateFrameworks/DiskImages.framework/Resources/CDiskImage.icns
image-type      : read/write
system-image    : false
blockcount      : 134217728
blocksize       : 512
writeable       : TRUE
autodiskmount   : TRUE
removable       : TRUE
image-encrypted : false
mounting user   : me
mounting mode   : <unknown>
process ID      : 45057
framework name  : DiskImages
/dev/disk4	GUID_partition_scheme	
/dev/disk4s1	69646961-6700-11AA-AA11-00306543ECAC	
/dev/disk6	EF57347C-0000-11AA-AA11-00306543ECAC	
/dev/disk6s1	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/iSCPreboot
/dev/disk6s2	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/xART
/dev/disk6s3	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Hardware
/dev/disk6s4	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Recovery
/dev/disk4s2	7C3457EF-0000-11AA-AA11-00306543ECAC	
/dev/disk8	EF57347C-0000-11AA-AA11-00306543ECAC	
/dev/disk8s1	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Macintosh HD
/dev/disk8s2	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Preboot
/dev/disk8s3	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Recovery 2
/dev/disk8s4	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Update 1
/dev/disk8s5	41504653-0000-11AA-AA11-00306543ECAC	
/dev/disk8s6	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/VM
/dev/disk4s3	52637672-7900-11AA-AA11-00306543ECAC	
/dev/disk7	EF57347C-0000-11AA-AA11-00306543ECAC	
/dev/disk7s1	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Recovery 1
/dev/disk7s2	41504653-0000-11AA-AA11-00306543ECAC	/Volumes/Update

Finally, I wanted to see what UTM's qemu-image utility would think about this image. I realize that the macOS guest virtualization is handled by virtualizationFramework, and isn't emulated by QEMU...

But I hit another, unexpected bump:

% /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img
zsh: exec format error: /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img

Yow. "exec format error"?

% file /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img 
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64
- Mach-O 64-bit dynamically linked shared library x86_64] [arm64]
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

Ok, sorry, not helpful.

But in summary, getting disk images attached to a UTM macOS guest VM is not (yet?) working as I might expect from the other path via QEMU.

[steven-michaud]

I have no trouble adding my bootable ISO as an "external drive" to my M1 macOS 12.3.1 guest running on an M1 macOS 12.3.1 host. It shows up (like you said) as an internal drive. But I'm able to navigate it, and even to start the installer on it (though of course I quit before anything's actually installed). "New External Drive", for UTM, seems to mean "attach a drive image from somewhere outside the default location of ~/Library/Containers/com.utmapp.UTM". My problem is that I can't boot from it.

The tests you performed on your sparse image may have damaged it before you converted it to UDTO format. That may explain the troubles you had with your ISO.

Thanks for your link to those Eclectic Light articles. I may find in them some way to trick the boot loader into failing over to the recovery partition. Assuming, of course, that the guest actually has one ...

The whole reason I want to boot from an Apple bootable installer is to load its recovery partition, so that I can turn off SIP in the guest, so that I can use it to work on porting HookCase to Apple Silicon. I'll be happy if I can find another way to do this.

@steven-michaud - Thanks for your very interesting HookCase tool!

You're most welcome :-)

[steven-michaud]

I may find in them some way to trick the boot loader into failing over to the recovery partition.

I did it! It was absurdly simple!

1. Create a new drive and start the guest VM.

2. Just after you log in, you'll be warned that "the disk you attached is not readable by this computer". Choose to initialize it, then choose "Erase". As best I can tell, it doesn't matter which format you choose.

3. Shut down the guest, then "Move Up" the new drive (so that it's the first in the boot order).

4. Restart the guest VM. It will fail to boot.

5. Close the guest VM's window, which will kill the VM.

6. Restore the original boot order (and possibly delete the drive you created in step 1), then restart the guest VM.

7. As it starts up, you'll be told that "authentication is required to verify startup disk", and prompted for a password.

8. Choose "Cancel", then choose "Startup Disk".

9. Quit "Startup Disk". You'll end up in the Recovery utility!

This resolves my problem, but doesn't resolve this bug. So I'll leave it open. But fixing it has now become much less urgent, at least for me.

[grnch]

Thank you so much @steven-michaud! I was stuck in a similar situation and these steps were the only thing that worked!

In my case I didn't even need to boot up the guest VM to initialize the new disk. Simply creating a disk in the UTM settings and moving it to the top of the disk order did the trick.

[steven-michaud]

You're most welcome.

But I just retested, and found that I had to go through the whole sequence (with an existing extra drive). It wasn't enough to just move the extra drive to the top, save the settings, move the original (bootable) drive back up, and save the settings again. Of course my "extra drive" isn't bootable. I don't know if yours is.

[grnch]

It was definitely a non-bootable disk. I simply hit "New Disk" from the UTM settings dialog and moved it to the top. So it was a brand new uninitialized virtual disk.

But I just realized the VM I was trying to recover was a bit older actually. It had Mac OS 12.1 on it, definitely not the most recent, so maybe that accounts for the difference?

Sorry for the confusion, I should have looked up and posted the exact MacOS version I was dealing with. UTM version was 3.2.4 (58), the most recent stable release as of writing.

[steven-michaud]

I've done some more testing, with a LaCie 1TB external SSD that has a Thunderbolt 3 connection.

First I used createinstallmedia to install a macOS 12.3.1 bootable installer on it. I found it would only boot on Intel Macs -- not on Apple Silicon ones. createinstallmedia only works with drives formatted with the HFS+J file system. I suspect this is what makes my external SSD's bootable installer unbootable on Apple Silicon machines.

Then I used macOS 12.3.1 recovery partitions on an Apple Silicon Mac and an Intel Mac to install the macOS 12.3.1 OS on my external SSD. (I did this once on the Apple Silicon Mac, once again on the Intel Mac, and once yet again on the Apple Silicon Mac.) At the end of this, my external SSD was bootable on both my Apple Silicon Mac and my Intel Mac. On installing macOS 10.15, 11 or 12, the target drive/partition gets reformatted to APFS, if need be. This, I suspect, is what made it bootable on both kinds of hardware.

This doesn't entirely explain the problems I reported in my first comment. UTM can't boot from my bootable ISO on either kind of hardware. But it may provide a partial explanation.

[watersb]

Wow, very nice! Thanks!

…

----

On Apr 17, 2022, at 11:59 AM, Steven Michaud ***@***.***> wrote:  I may find in them some way to trick the boot loader into failing over to the recovery partition. I did it! It was absurdly simple! Create a new drive and start the guest VM. Just after you log in, you'll be warned that "the disk you attached is not readable by this computer". Choose to initialize it, then choose "Erase". As best I can tell, it doesn't matter which format you choose. Shut down the guest, then "Move Up" the new drive (so that it's the first in the boot order). Restart the guest VM. It will fail to boot. Close the guest VM's window, which will kill the VM. Restore the original boot order (and possibly delete the drive you created in step 1), then restart the guest VM. As it starts up, you'll be told that "authentication is required to verify startup disk", and prompted for a password. Choose "Cancel", then choose "Startup Disk". Quit "Startup Disk". You'll end up in the Recovery utility! This resolves my problem, but doesn't resolve this bug. So I'll leave it open. But fixing it has now become much less urgent, at least for me. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[adespoton]

I may have some information to help explain this (but it won't help fix the "bug" here). Because AARCH64 uses the internal nVME SSD to store the firmware as well as the EFI and regular disk partitions, a properly formatted disk image needs to be available to check the firmware partition, and the virtual machine needs to think it's the internal one. This same issue exists on bare metal hardware: if you don't have an internal firmware partition available, an external ISO image won't boot on an M1 device.

What you've done here is successfully partition an image so that it's got the required partitions to read/write, and then you can boot the iso because it has somewhere to write data to during boot.

[steven-michaud]

@adespoton, are you saying that unless an external image/drive has a particular partition stored on it, it won't boot on an Apple Silicon machine? And is that partition the "nVME SSD" partition?

And if so, does only the APFS file system have "room" to store this partition ("nVME SSD" or something else)? As opposed to the HFS+J file system?

Edit: Actually, "nVME" seems to be 1) the NVM Express specification and 2) the (physical) logic to implement this specification, which is stored on SSD drives that support NVM Express. If so, I don't see the connection between "nVME" and my test results with my 1TB LaCie external SSD. It presumably does support the NVM Express specification. But it's bootable on both Apple Silicon and Intel hardware if it has a copy of macOS on an APFS partition, and only bootable on Intel hardware if it has a bootable macOS installer on an HFS+J partition.

[osy]

ISOs are not supported by Apple's Virtualization.framework and for QEMU you need to go through OpenCore and follow some of the rather complicated guides. I also am pretty sure QEMU doesn't support ARM64 macOS at all. Apple's supported method is to use an IPSW.

[steven-michaud]

See above :-) Then, of course, you'll need to run the Terminal app (from the Recovery utility) and do csrutil disable.

[dariaphoebe]

@steven-michaud, even from Terminal I get a "Failed to create local security policy" error from csrutil; the startup disk security option throws SDErrorDomain error 104. As it might be particular to macOS versions, I assume you used Monterey? (Given you mention 12.3.1 in your external disk exercises)

[dariaphoebe]

Ah, i see from steven-michaud/HookCase#19 that your aims are the same as mine and I'm going to end up stuck anyway.

[steven-michaud]

I've since upgraded both host and guest to macOS 12.4 (using Software Update in both cases). But I'm still able to use my hack to get into the Recovery Utility and successfully run csrutil disable (in Terminal). I just tried it again. In case it's relevant, I'm still running UTM 3.1.5.

But no, that doesn't allow me to load a third party (non-Apple) kernel extension -- a plain-vanilla kext made using XCode's sample code.

[dariaphoebe]

csrutil disable fails on Ventura beta, but there's no point in me rolling backwards if i'm not going to be able to load a kext. so, that's useful to know. thanks!

[frenkie1521]

Okej I managed to do it on my own using some tips.

Install .iso file from vulnhub and use qemu-img to convert it into qcow2 if you dont know how google it its simple.
Boot the device in UTM as you would boot qcow2 file. (no boot option, pci440fx-6.2)
It will show you no boot device when you start it and then close it and go back add new disk with iso file and start it.

It loaded normal for me I downloaded damn_vulnorable_linux.iso and did all those steps.

Hope it helps!