Xen general protection fault during startup when executed from qemu OVMF UEFI firmware

[osresearch]

Qubes OS version
qubes-4.1 source

Affected component(s) or functionality
qubes-vmm-xen

Brief summary
Booting the Qubes-patched Xen 4.14.0 hypervisor directly from the qemu UEFI firmware (without grub) causes a GPF when setting up memory maps. The clean version of 4.14.0 does not show this problem.

How Reproducible
100% of the time.

To Reproduce

Steps to reproduce the behavior:

1. Extract the boot/efi/EFI/qubes/xen-4.14.0.efi from https://gitlab.com/QubesOS/qubes-vmm-xen/-/jobs/891462871/artifacts/file/qubes-packages-mirror-repo/dom0-fc32/rpm/xen-hypervisor-4.14.0-8.fc32.x86_64.rpm
2. Build an EFI System Partion with xen-4.14.0.efi as EFI/BOOT/BOOTX64.EFI, a kernel in EFI/BOOT/vmlinuz, a ramdisk in EFI/BOOT/initrd.img and a simplified config file as EFI/BOOT/BOOTX64.CFG:

[global]
default=qubes-verbose
[qubes-verbose]
options=console=com1 efi=attr=uc no-real-mode
noexitboot=1
mapbs=1
kernel=vmlinuz console=ttyS0 earlyprintk=ttyS0
ramdisk=initrd.img

3. Make a disk image (I use the safeboot/sbin/mkesp and safeboot/sbin/mkgpt to build the filesystems):

~/safeboot/sbin/mkfat esp.bin ./esp/
~/safeboot/sbin/mkgpt hda.bin esp.bin

4. Run qemu with the OVMF firmware and serial console:

qemu-system-x86_64 \
  -M q35,accel=kvm \
  -m 4G \
  -bios /usr/share/OVMF/OVMF_CODE.fd  \
  -hda hda.bin \
  -serial stdio

Expected behavior

Xen 4.14.0 (c/s ) EFI loader
Using configuration file 'BOOTX64.cfg'
vmlinuz: 0x000000007c684000-0x000000007ccd3ca0
initrd.img: 0x000000007ab69000-0x000000007bf35b1c
 __  __            _  _    _ _  _    ___
 \ \/ /___ _ __   | || |  / | || |  / _ \
  \  // _ \ '_ \  | || |_ | | || |_| | | |
  /  \  __/ | | | |__   _|| |__   _| |_| |
 /_/\_\___|_| |_|    |_|(_)_|  |_|(_)___/

(XEN) Xen version 4.14.0 (hudson@) (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0) debug=n  Mon Dec 21 17:21:37 CET 2020
(XEN) Latest ChangeSet: 
(XEN) Bootloader: EFI
(XEN) Command line: console=com1 efi=attr=uc no-real-mode
(XEN) Xen image load base address: 0x7ce00000
....
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000174000000->0000000178000000 (979288 pages to be allocated)
(XEN)  Init. ramdisk: 000000017ec33000->000000017ffffb1c
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff82e2c000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->00000080007a2928
(XEN)  Start info:    ffffffff82e2c000->ffffffff82e2c4b8
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff82e2d000->ffffffff82e48000
(XEN)  Boot stack:    ffffffff82e48000->ffffffff82e49000
(XEN)  TOTAL:         ffffffff80000000->ffffffff83000000
(XEN)  ENTRY ADDRESS: ffffffff82735180
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 2048kB init memory
mapping kernel into physical memory
about to get started...
[    0.000000] Linux version 4.19.94-1.pvops.qubes.x86_64 (user@build-fedora4) (gcc version 6.4.1 20170727 (Red Hat 6.4.1-1) (GCC)) #1 SMP Sun Jan 12 05:45:26 UTC 2020
[    0.000000] Command line: console=ttyS0 earlyprintk=ttyS0
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] Released 0 page(s)
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000000007fffff] usable
[    0.000000] Xen: [mem 0x0000000000800000-0x0000000000807fff] ACPI NVS
[    0.000000] Xen: [mem 0x0000000000808000-0x000000000080ffff] usable
[    0.000000] Xen: [mem 0x0000000000810000-0x00000000008fffff] ACPI NVS
[    0.000000] Xen: [mem 0x0000000000900000-0x000000007ed3efff] usable
[    0.000000] Xen: [mem 0x000000007ed3f000-0x000000007edfffff] reserved
[    0.000000] Xen: [mem 0x000000007ee00000-0x000000007f8eefff] usable
[    0.000000] Xen: [mem 0x000000007f8ef000-0x000000007fb6efff] reserved
[    0.000000] Xen: [mem 0x000000007fb6f000-0x000000007fb7efff] ACPI data
[    0.000000] Xen: [mem 0x000000007fb7f000-0x000000007fbfefff] ACPI NVS
[    0.000000] Xen: [mem 0x000000007fbff000-0x000000007ff57fff] usable
[    0.000000] Xen: [mem 0x000000007ff58000-0x000000007ff77fff] reserved
[    0.000000] Xen: [mem 0x000000007ff78000-0x000000007fffffff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
[    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x000000017fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi:  SMBIOS=0x7f91b000  ACPI=0x7fb7e000  ACPI 2.0=0x7fb7e014  MEMATTR=0x7e36f018 

Actual behavior

Xen 4.14.0 (c/s ) EFI loader
Using configuration file 'BOOTX64.cfg'
vmlinuz: 0x000000007c684000-0x000000007ccd3ca0
initrd.img: 0x000000007ab69000-0x000000007bf35b1c
!!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000
RIP  - 000000007D021B25, CS  - 0000000000000038, RFLAGS - 0000000000010016
RAX  - 2494894D4FFC4D17, RCX - 000000009F850FD2, RDX - 000000007FF27FFF
RBX  - 000000007D448A50, RSP - 000000007FF24628, RBP - 000000007D730360
RSI  - 000000007D448A50, RDI - 000000007D618698
R8   - 0000000000000001, R9  - 000000007D62E0B0, R10 - 000000007FF246D0
R11  - 000000007FF24470, R12 - 000000007FF24670, R13 - 0000000000000206
R14  - 000000007D21D654, R15 - 000000007FF24630
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 000000007FC01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000007F9EEA98 0000000000000047, LDTR - 0000000000000000
IDTR - 000000007F4D4018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000007FF24280
!!!! Find image based on IP(0x7D021B25)  (ImageBase=000000007CE00000, EntryPoint=000000007D43E830) !!!!

Additional context
This was found while testing the unified hypervisor patches QubesOS/qubes-vmm-xen#89, although the crash happens even on a Qubes hypervisor build without the unified patches applied.

OVMF is installed from Ubuntu 18.04 repo:

ovmf/now 2020.05-3~bpo+1 all [installed,local]
  UEFI firmware for 64-bit x86 virtual machines

Solutions you've tried
If Xen's .config is changed to turn off CONFIG_EFI_SET_VIRTUAL_ADDRESS_MAP the hypervisor starts-up safely, but halts when initializing the dom0 kernel:

....
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000174000000->0000000178000000 (969546 pages to be allocated)
(XEN)  Init. ramdisk: 000000017ec33000->000000017ffffb1c
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff82e2c000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->000000800078f8b8
(XEN)  Start info:    ffffffff82e2c000->ffffffff82e2c4b8
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff82e2d000->ffffffff82e48000
(XEN)  Boot stack:    ffffffff82e48000->ffffffff82e49000
(XEN)  TOTAL:         ffffffff80000000->ffffffff83000000
(XEN)  ENTRY ADDRESS: ffffffff82735180
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 2048kB init memory
mapping kernel into physical memory
about to get started...
Xen hypervisor allocated kernel memory conflicts with E820 map
(XEN) Hardware Dom0 halted: halting machine

Relevant documentation you've consulted

Related, non-duplicate issues

[osresearch]

Updating to a newer OVMF (edk2-stable202008) seems to do away with the GPF when setting the virtual address map, so that might be an OVMF bug if there are too many memory descriptors? Logging the passes through the loop in efi_exit_boot() shows there are 132 with the older OVMF, 101 with the newer. Crossing 128 is suspicious...

Although even with the updated OVMF, Xen still halts with the E820 overlap error. Unfortunately debugging around the memory map codes is a challenge since it also sometimes results in KVM errors...

KVM internal error. Suberror: 1
emulation failure
RAX=00000000fffffc00 RBX=000000007ea36130 RCX=0000000000000010 RDX=000000007d24889a
RSI=000000007d24889a RDI=0000000000000000 RBP=8000000000000003 RSP=000000007fe876c8
R8 =000000007ea36180 R9 =0000000000000000 R10=000000007fe87200 R11=00000000000012f0
R12=000000007d24889a R13=000000007fed1000 R14=0000000000000000 R15=0000000000000000
RIP=00000000000affff RFL=00010046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0038 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT=     000000007f9eea98 00000047
IDT=     000000007f423018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=000000007fc01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d00
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

[marmarek]

With CONFIG_EFI_SET_VIRTUAL_ADDRESS_MAP set, noexitboot=1 in xen.cfg should not be needed anymore. Depending on the firmware, mapbs=1 may be unnecessary too.

[marmarek]

I found some notes about mapbs=1 workaround incompatibility with OVMF, I guess this may be exactly the issue you see.

[osresearch]

Removing both mapbs=1 and noexitboot=1 from the xen config allowed the normal and unified Xen to boot, with both OVMF 202005 from the ubuntu 18.04 package or the built-from-source stable-202008 release.

• noexitboot=1 causes the GPF under OVMF during EFI set virtual address map.
• mapbs=1 causes the e820 map overlap error when starting the dom0 kernel.