OpenBSD 7.0 Guest crashes QEMULauncher

[AwlsomeAlex]

Describe the issue
OpenBSD 7.0 crashes UTM/QEMULauncher

Configuration

• UTM Version: 2.3.0 Beta
• OS Version: macOS Big Sur 11.6
• Intel or Apple Silicon? Apple Silicon

Went to run VM via miniroot.img, got to boot loader, typed in 'set tty fb0', then VM crashes.

Crash log
QEMULauncher.log

Debug log
debug.log

Upload VM
config.plist.txt

[osy]

@agraf this seems to be an assertion failure on https://github.com/qemu/qemu/blob/master/target/arm/hvf/hvf.c#L1159

Have you hit this before? Can you explain the reasoning for that assertion? Thanks.

[agraf]

@agraf this seems to be an assertion failure on https://github.com/qemu/qemu/blob/master/target/arm/hvf/hvf.c#L1159

Have you hit this before? Can you explain the reasoning for that assertion? Thanks.

This means that an instruction was accessing MMIO space that is not predecoded by hardware. Neither KVM not QEMU's hvf implementation emulate instructions for MMIO exits; they merely take the hardware exit info and act accordingly.

does the same guest run fine on KVM?

[AwlsomeAlex]

@agraf No issues on my x86 virtualization host (running KVM).

[osy]

I think we need ARM64 KVM to test. Unfortunately I don’t have easy access to hardware (haven’t tried AsahiLinux so idk if it supports KVM).

[agraf]

This is an ARM specific issue. Please check on aarch64 KVM (Raspberry Pi 4 for example).

[agraf]

Looking at the latest from here: http://ftp.luis.uni-hannover.de/openbsd/snapshots/arm64/

$ ./build/qemu-system-aarch64 -M virt,highmem=no -bios build/pc-bios/edk2-aarch64-code.fd -m 4G -smp 2 -nographic -drive file=~/Downloads/miniroot70.img,if=none,id=d,snapshot=on -net none -cpu host -accel hvf -device nvme,drive=d,serial=1234 -smp 2 -trace '*hvf*' -D log

[...]
hvf_data_abort data abort: [pc=0xffffff8000342794 va=0xffffff8002421000 pa=0x0000000009000000 isv=0 iswrite=1 s1ptw=0 len=1 srt=0]

(gdb) x/i $pc
=> 0xffffff8000342794:	dc	civac, x0

(qemu) info mtree
[...]
    0000000009000000-0000000009000fff (prio 0, i/o): pl011
[...]

So from what I can see, OpenBSD is trying to issue a "cache line clear" instruction (memset(0) optimization) on the serial port registers. That is not a good idea ... :).

[AwlsomeAlex]

So is this an OpenBSD issue? @agraf

[kettenis]

Yes, I've seen that. I actually think this is a cache line clean and invalidate instruction. And my reading of the ARMv8 ARM says that it should be perfectly fine to issue those on device memory.

May still not be a good idea, but our kernel cleans any page that is mapped non-cachable because it might have been mapped cachable in the past. It works on real hardware though.

[agraf]

I posted a possible workaround for this in QEMU: https://patchwork.kernel.org/project/qemu-devel/patch/20211025191349.52992-1-agraf@csgraf.de/

[agraf]

v2: https://patchwork.kernel.org/project/qemu-devel/patch/20211026071241.74889-1-agraf@csgraf.de/