`lxc-exec --setup-hyperlight` fails

[Skyb0rg007]

Relevant area(s)

Linux

Brief description of your issue

lxc-exec --setup-hyperlight fails

Steps to reproduce

$ lxc-exec --setup-hyperlight
pyhl:   pull ghcr.io/hyperlight-dev/hyperlight-unikraft/python-agent-driver-kernel:v0.9.0
pyhl:   pull ghcr.io/hyperlight-dev/hyperlight-unikraft/python-agent-driver-initrd:v0.9.0
hyperlight setup failed: hyperlight install: Internal Hyperlight VM error: Map region error: VM map memory error: Hypervisor error: KVM error: File exists (os error 17): Map region error: VM map memory error: Hypervisor error: KVM error: File exists (os error 17): VM map memory error: Hypervisor error: KVM error: File exists (os error 17): Hypervisor error: KVM error: File exists (os error 17)

Expected behavior

lxc-exec --setup-hyperlight shouldn't fail

Actual behavior

lxc-exec --setup-hyperlight fails

Additional details:

Running the code under strace shows the following:

...
openat(AT_FDCWD, "/dev/mshv", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/kvm", O_RDWR|O_CLOEXEC) = 3
ioctl(3, KVM_GET_API_VERSION, 0)        = 12
ioctl(3, KVM_CHECK_EXTENSION, KVM_CAP_USER_MEMORY) = 1
close(3)                                = 0
openat(AT_FDCWD, "/dev/kvm", O_RDWR|O_CLOEXEC) = 3
ioctl(3, KVM_CREATE_VM, 0)              = 4
ioctl(3, KVM_GET_VCPU_MMAP_SIZE, 0)     = 12288
ioctl(4, KVM_CREATE_IRQCHIP, 0)         = 0
eventfd2(0, 0)                          = 5
ioctl(4, KVM_IRQFD, 0x7ffed67b0108)     = 0
ioctl(4, KVM_CREATE_VCPU, 0)            = 6
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x7433f6cbd000
ioctl(3, KVM_GET_SUPPORTED_CPUID, {nent=58, entries=[...]}) = 0
ioctl(6, KVM_SET_CPUID2, {nent=58, entries=[...]}) = 0
ioctl(6, KVM_SET_SREGS, {cs={base=0, limit=0, selector=0, type=11, present=1, dpl=0, db=0, s=1, l=1, g=0, avl=0}, ...}) = 0
ioctl(4, KVM_SET_USER_MEMORY_REGION, {slot=0, flags=KVM_MEM_READONLY, guest_phys_addr=0x1000, memory_size=2692313088, userspace_addr=0x74316f482000}) = 0
ioctl(4, KVM_SET_USER_MEMORY_REGION, {slot=1, flags=0, guest_phys_addr=0xfd7af0000, memory_size=676397056, userspace_addr=0x7433c7aef000}) = 0
rt_sigaction(SIGRT_2, {sa_handler=0x64bdc7b2aef0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7433f6a42790}, NULL, 8) = 0
ioctl(4, KVM_SET_USER_MEMORY_REGION, {slot=2, flags=KVM_MEM_READONLY, guest_phys_addr=0xc0000000, memory_size=1123610624, userspace_addr=0x743384a00000}) = -1 EEXIST (File exists)

[jibin7jose]

Hi @Skyb0rg007,

Root Cause:
The EEXIST (os error 17) during KVM initialization occurs because of a memory mapping overlap with the hypervisor's reserved address space.

In the v0.9.0 release of hyperlight-unikraft, the python-agent-driver-initrd image grew significantly (to ~1.046 GB) due to the addition of many new pip packages. The hyperlight-host library maps this initrd into the guest's physical memory starting at address 0xc0000000 (the 3 GB mark).

Because the initrd is now over 1 GB, its memory region (0xc0000000 + 1.046 GB = 0x102f8e000) crosses the 4 GB address boundary. In KVM for x86_64, the Local APIC memory is hard-reserved at physical address 0xfee00000. The new, larger initrd mapping overlaps directly with this APIC region, causing KVM to reject the KVM_SET_USER_MEMORY_REGION mapping.

The Fix:
Until the upstream hyperlight-unikraft library corrects its memory mapping layout to accommodate images over 1 GB (or trims its default image size), we need to downgrade the hyperlight-unikraft-host dependency and the GHCR image tag to v0.8.0 where the initrd was smaller and didn't cross the APIC memory hole.

[MGudgin]

@danbugs Is this issue on your radar?

[danbugs]

Hey--just saw this, @MGudgin. Looking into it.

[danbugs]

I couldn't replicate this issue w/ my kernel version, but, regardless, just dropped a PR that should fix it. @MGudgin , would appreciate a review. @Skyb0rg007 & @jibin7jose if you could test your workload again from #534 that would be great. Thank you!!

[Skyb0rg007]

Seems like it works!

$ lxc-exec --setup-hyperlight
pyhl:   pull ghcr.io/hyperlight-dev/hyperlight-unikraft/python-agent-driver-kernel:v0.10.0
pyhl:   pull ghcr.io/hyperlight-dev/hyperlight-unikraft/python-agent-driver-initrd:v0.10.0
  sparsified: 656 MiB on disk (punched 609727 zero pages)
hyperlight setup: snapshot ready at "/home/skye/.local/share/pyhl/snapshot.hls"

I'm on NixOS, so I'm not directly building from source (my build script is here, and I'm on the 7.0.10 kernel if you want to replicate).