[Bug]: Improper emulation of vmmcall instructions, leading to a full BSOD in the entire VM

[NotRequiem]

Version

7.2.6

Host OS Type

Windows

Host OS name + version

Windows 11 Pro 25H2 26200.8039

Host Architecture

x86

Guest OS Type

Windows

Guest Architecture

x86

Guest OS name + version

Windows 10 Home 22H2 19045.3803

Component

Unspecified

What happened?

When reading the source code, I noticed a bug in the emulator.

When VT-x support is not enabled because of Hyper-V, and an application issues the vmmcall instruction (0f 01 d9) when not executing under an AMD CPU, I can cause the whole system to crash.

VBox.log

Applications might want to issue this instruction to detect virtualization, verify the CPU vendor, etc. In any case, a CPL3 context should not be able to bring down the entire system

How can we reproduce this?

Just compile and run my code in a VM with an Intel CPU:

#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <stdint.h>
#include <string.h>

static PVOID g_veh = NULL;
static volatile LONG g_handled = 0;
static volatile DWORD g_exception_code = 0;

static LONG WINAPI VehHandler(PEXCEPTION_POINTERS ep)
{
    if (!ep || !ep->ExceptionRecord || !ep->ContextRecord) {
        return EXCEPTION_CONTINUE_SEARCH;
    }

    switch (ep->ExceptionRecord->ExceptionCode) {
    case EXCEPTION_PRIV_INSTRUCTION:
    case EXCEPTION_ILLEGAL_INSTRUCTION:
        g_exception_code = ep->ExceptionRecord->ExceptionCode;
        g_handled = 1;

#if defined(_M_X64) || defined(__x86_64__)
        ep->ContextRecord->Rip += 3;
#else
        ep->ContextRecord->Eip += 3;
#endif
        return EXCEPTION_CONTINUE_EXECUTION;

    default:
        return EXCEPTION_CONTINUE_SEARCH;
    }
}

#if defined(_MSC_VER)

static void run_vmmcall(void)
{
    static const unsigned char stub[] = {
        0x0F, 0x01, 0xD9, 0xC3
    };

    void* mem = VirtualAlloc(NULL, sizeof(stub),
        MEM_COMMIT | MEM_RESERVE,
        PAGE_EXECUTE_READWRITE);
    void (*fn)(void);

    if (!mem) {
        return;
    }

    memcpy(mem, stub, sizeof(stub));
    FlushInstructionCache(GetCurrentProcess(), mem, sizeof(stub));

    fn = (void (*)(void))mem;
    fn();

    VirtualFree(mem, 0, MEM_RELEASE);
}

#else

__attribute__((naked, noinline))
static void run_vmmcall(void)
{
    __asm__ __volatile__(
        ".byte 0x0f, 0x01, 0xd9\n\t"
        "ret\n\t"
    );
}

#endif

int main(void)
{
    g_veh = AddVectoredExceptionHandler(1, VehHandler);
    if (!g_veh) {
        return 1;
    }

    run_vmmcall();

    RemoveVectoredExceptionHandler(g_veh);
    g_veh = NULL;

    return g_handled ? 0 : 2;
}

Did you upload all of your necessary log files, screenshots, etc.?

• Yes, I've uploaded all pertinent files to this issue.