Inability to boot with memory below 1074mb

[ggoodman]

Note: I'm using a bit of a weird setup, where I'm statically linking libkrun and embedding the kernel using #include_bytes! in my host CLI instead of using libkrunfw. I'll add the relevant patches below as an appendix in case this is a case of me breaking things.

Test scenarios

Scenario 1: --memory=1073mb

Here is the full log. No evidence of my init booting. Exit code: 0.

[2025-11-10T14:22:31Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x300000000 guest_addr=40000000 len=3ff00000
[2025-11-10T14:22:31Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x130000000 guest_addr=80000000 len=20000000
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 3
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 3
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 3
[2025-11-10T14:22:31Z DEBUG devices::legacy::aarch64::gpio] SET_IRQ_LINE (GPIO)=33
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 33
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 33
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 33
[2025-11-10T14:22:31Z DEBUG devices::virtio::mmio[balloon]] set_irq_line: 34
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 49
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 49
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 49
[2025-11-10T14:22:31Z DEBUG devices::virtio::mmio[rng]] set_irq_line: 35
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 68
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 70
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 66
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 68
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 68
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 70
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 70
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 66
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 66
[2025-11-10T14:22:31Z DEBUG devices::virtio::mmio[console]] set_irq_line: 36
[2025-11-10T14:22:31Z DEBUG devices::virtio::mmio[fs]] set_irq_line: 37
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 13
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 13
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 13
[2025-11-10T14:22:31Z DEBUG devices::virtio::mmio[vsock]] set_irq_line: 38
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent new: 15
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] EpollEvent data: 15
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] add fd in: 15
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] kevs len: 128
[2025-11-10T14:22:31Z DEBUG vmm::macos::vstate] vCPU 0 PSCI
[2025-11-10T14:22:31Z DEBUG vmm::macos::vstate] vCPU 0 PSCI
[2025-11-10T14:22:31Z INFO  vmm::macos::vstate] vCPU 0 received shutdown signal
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] ret: 1
[2025-11-10T14:22:31Z DEBUG utils::macos::epoll] kev: { ident: 15, data: 8 }
[2025-11-10T14:22:31Z DEBUG vmm] using vcpu exit code: 0
[2025-11-10T14:22:31Z INFO  vmm] Vmm is stopping.
[2025-11-10T14:22:31Z TRACE devices::virtio::console::device] Console on_vmm_exit finished

Scenario 2: --memory=1074mb

Exit code: 134.

thread '<unnamed>' (274019004) panicked at src/vmm/src/builder.rs:1288:18:
called `Result::unwrap()` on an `Err` value: InvalidGuestAddress(GuestAddress(2147483648))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 3, aborting
zsh: abort      ./build/dist/vibebox run --rootfs rootfs --init-log-level=5 --log-level=5 

Scenario 3: --memory=1075mb

In this scenario, libkrun shows epoll logs but doesn't respond to SIGINT or exit.

[2025-11-10T14:24:19Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x103c3c000 guest_addr=40000000 len=40100000
[2025-11-10T14:24:19Z DEBUG vmm::macos::vstate] Guest memory host_addr=0x300000000 guest_addr=c0000000 len=20000000
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 3
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 3
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 3
[2025-11-10T14:24:19Z DEBUG devices::legacy::aarch64::gpio] SET_IRQ_LINE (GPIO)=33
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 33
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 33
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 33
[2025-11-10T14:24:19Z DEBUG devices::virtio::mmio[balloon]] set_irq_line: 34
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 49
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 49
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 49
[2025-11-10T14:24:19Z DEBUG devices::virtio::mmio[rng]] set_irq_line: 35
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 68
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 70
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 66
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 68
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 68
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 70
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 70
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 66
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 66
[2025-11-10T14:24:19Z DEBUG devices::virtio::mmio[console]] set_irq_line: 36
[2025-11-10T14:24:19Z DEBUG devices::virtio::mmio[fs]] set_irq_line: 37
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 13
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 13
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] add fd in: 13
[2025-11-10T14:24:19Z DEBUG devices::virtio::mmio[vsock]] set_irq_line: 38
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent new: 15

thread 'fc_vcpu 0' (274021389) panicked at src/hvf/src/lib.rs:728:18:
unexpected exception: 0x20
[2025-11-10T14:24:19Z DEBUG utils::macos::epoll] EpollEvent data: 15
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Scenario 4: --memory=1097mb (and above, AFAICT)

Works as expected.

Appendix

lib.rs.patch

Allow us to read embedded kernel using custom embed.

diff --git a/src/libkrun/src/lib.rs b/src/libkrun/src/lib.rs
index 3a5c577..4d37cd7 100644
--- a/src/libkrun/src/lib.rs
+++ b/src/libkrun/src/lib.rs
@@ -68,6 +68,9 @@ use krun_input::{InputConfigBackend, InputEventProviderBackend};
 #[cfg(feature = "nitro")]
 use nitro_enclaves::launch::StartFlags;
 
+// Module for embedded kernel (static linking support)
+mod embedded_kernel;
+
 // Value returned on success. We use libc's errors otherwise.
 const KRUN_SUCCESS: i32 = 0;
 // Maximum number of arguments/environment variables we allow
@@ -2030,6 +2033,36 @@ unsafe fn load_krunfw_payload(
     Ok(())
 }
 
+/// Load kernel from embedded binary (for static linking)
+/// This replaces load_krunfw_payload when using embedded kernel
+unsafe fn load_embedded_kernel_payload(vmr: &mut VmResources) -> Result<(), &'static str> {
+    let mut kernel_guest_addr: u64 = 0;
+    let mut kernel_entry_addr: u64 = 0;
+    let mut kernel_size: usize = 0;
+
+    let kernel_host_addr = embedded_kernel::embedded_get_kernel(
+        &mut kernel_guest_addr as *mut u64,
+        &mut kernel_entry_addr as *mut u64,
+        &mut kernel_size as *mut usize,
+    );
+
+    if kernel_host_addr.is_null() || kernel_size == 0 {
+        return Err("Failed to load embedded kernel");
+    }
+
+    let kernel_bundle = KernelBundle {
+        host_addr: kernel_host_addr as u64,
+        guest_addr: kernel_guest_addr,
+        entry_addr: kernel_entry_addr,
+        size: kernel_size,
+    };
+
+    vmr.set_kernel_bundle(kernel_bundle)
+        .map_err(|_| "Failed to set kernel bundle")?;
+
+    Ok(())
+}
+
 #[no_mangle]
 pub extern "C" fn krun_setuid(ctx_id: u32, uid: libc::uid_t) -> i32 {
     match CTX_MAP.lock().unwrap().entry(ctx_id) {
@@ -2401,14 +2434,24 @@ pub extern "C" fn krun_start_enter(ctx_id: u32) -> i32 {
         && ctx_cfg.vmr.firmware_config.is_none()
         && cfg!(not(feature = "efi"))
     {
-        if let Some(ref krunfw) = ctx_cfg.krunfw {
-            if let Err(err) = unsafe { load_krunfw_payload(krunfw, &mut ctx_cfg.vmr) } {
-                eprintln!("Can't load libkrunfw symbols: {err}");
+        // Try embedded kernel first (for static linking)
+        let embedded_result = unsafe { load_embedded_kernel_payload(&mut ctx_cfg.vmr) };
+
+        if let Err(embedded_err) = embedded_result {
+            // Embedded kernel failed, try dynamic libkrunfw
+            debug!("Embedded kernel not available ({}), trying dynamic libkrunfw", embedded_err);
+
+            if let Some(ref krunfw) = ctx_cfg.krunfw {
+                if let Err(err) = unsafe { load_krunfw_payload(krunfw, &mut ctx_cfg.vmr) } {
+                    eprintln!("Can't load libkrunfw symbols: {err}");
+                    return -libc::ENOENT;
+                }
+            } else {
+                eprintln!("Couldn't find or load {KRUNFW_NAME} and embedded kernel not available");
                 return -libc::ENOENT;
             }
         } else {
-            eprintln!("Couldn't find or load {KRUNFW_NAME}");
-            return -libc::ENOENT;
+            debug!("Using embedded kernel (static linking)");
         }
     }

embedded_kernel.rs

// Embedded kernel binary for static linking
// This replaces the dynamic loading of libkrunfw.dylib

use std::ffi::c_char;

// Embed the kernel binary at compile time
// Path is relative to the libkrun/ directory root
static KERNEL_BINARY: &[u8] = include_bytes!("../../../kernel.bin");

// Guest load address for ARM64
const KERNEL_LOAD_ADDR: u64 = 0x80000000;
// Entry point address for ARM64
const KERNEL_ENTRY_ADDR: u64 = 0x80000000;

/// Get the embedded kernel binary
/// This function has the same signature as krunfw_get_kernel from libkrunfw
///
/// # Arguments
/// * `load_addr` - Output: guest physical address where kernel should be loaded
/// * `entry_addr` - Output: guest physical address of kernel entry point
/// * `size` - Output: size of kernel binary in bytes
///
/// # Returns
/// Pointer to kernel binary data
#[no_mangle]
pub extern "C" fn embedded_get_kernel(
    load_addr: *mut u64,
    entry_addr: *mut u64,
    size: *mut usize,
) -> *const c_char {
    unsafe {
        if !load_addr.is_null() {
            *load_addr = KERNEL_LOAD_ADDR;
        }
        if !entry_addr.is_null() {
            *entry_addr = KERNEL_ENTRY_ADDR;
        }
        if !size.is_null() {
            *size = KERNEL_BINARY.len();
        }
    }
    KERNEL_BINARY.as_ptr() as *const c_char
}

/// Rust-friendly function to get kernel metadata
#[allow(dead_code)]
pub fn get_kernel_info() -> (u64, u64, usize) {
    (KERNEL_LOAD_ADDR, KERNEL_ENTRY_ADDR, KERNEL_BINARY.len())
}

/// Rust-friendly function to get kernel data slice
#[allow(dead_code)]
pub fn get_kernel_data() -> &'static [u8] {
    KERNEL_BINARY
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_kernel_embedded() {
        assert!(KERNEL_BINARY.len() > 0);
        // ARM64 kernel images start with specific magic bytes
        // ARM64 Image format has a header, let's just verify we have data
        assert!(KERNEL_BINARY.len() > 1024);
    }

    #[test]
    fn test_get_kernel_info() {
        let (load_addr, entry_addr, size) = get_kernel_info();
        assert_eq!(load_addr, 0x80000000);
        assert_eq!(entry_addr, 0x80000000);
        assert_eq!(size, KERNEL_BINARY.len());
    }

    #[test]
    fn test_embedded_get_kernel() {
        let mut load_addr: u64 = 0;
        let mut entry_addr: u64 = 0;
        let mut size: usize = 0;

        let ptr = embedded_get_kernel(&mut load_addr, &mut entry_addr, &mut size);

        assert!(!ptr.is_null());
        assert_eq!(load_addr, 0x80000000);
        assert_eq!(entry_addr, 0x80000000);
        assert_eq!(size, KERNEL_BINARY.len());
    }
}

Note that libkrun's cargo.toml is also patched to add "staticlib" to crate-type.

[valpackett]

I'm seeing this in a less-weird setup (Linux/aarch64, muvm, only commented out check_extension(GuestMemfd) to always fail since I'm getting EINVAL when the memfd is assigned — debugging now)..

❯ PATH=$PWD/target/release:$PATH RUST_LOG=debug muvm --mem=128 -it bash
[2025-11-27T23:23:11Z DEBUG arch::aarch64] ArchMemoryInfo { ram_last_addr: 1207959552, shm_start_addr: 2147483648, page_size: 4096, initrd_addr: 1205862400, firmware_addr: 0 } // my print
[..]
[2025-11-27T23:23:11Z INFO  vmm::linux::vstate] Received KVM_SYSTEM_EVENT_RESET
[2025-11-27T23:23:11Z DEBUG vmm] using vcpu exit code: 0
[2025-11-27T23:23:11Z INFO  vmm] Vmm is stopping.

❯ PATH=$PWD/target/release:$PATH muvm --mem=1024 -it bash
No IPv6 nameserver available for NDP/DHCPv6

thread '<unnamed>' panicked at src/vmm/src/builder.rs:1288:18:
called `Result::unwrap()` on an `Err` value: InvalidGuestAddress(GuestAddress(2147483648))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at library/core/src/panicking.rs:225:5:
panic in a function that cannot unwind
stack backtrace: [..]
   9:     0xfffff7a538f4 - core::panicking::panic_cannot_unwind::hbc3b8f86f705b9ed
  10:     0xfffff7a7869c - krun_start_enter

❯ PATH=$PWD/target/release:$PATH muvm --mem=1097 -it bash
No IPv6 nameserver available for NDP/DHCPv6
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
bash-5.3$

[slp]

I can confirm this problem here too on aarch64. Let's fix it.

[slp]

With #476 I'm able to boot VMs as small as 128MB.