vgpu-unlock-blackwell

Research project attempting to enable vGPU on consumer NVIDIA Blackwell GPUs (RTX 5090 / GB202).

Status: Hardware-blocked. The entire software pipeline works, but the GSP firmware crashes when accessing hardware registers that are fused off on consumer silicon. See Findings below.

Key Discovery: NVIDIA's enterprise vGPU (GRID) and consumer GeForce drivers ship identical binaries. The nv-kernel.o_binary and gsp_ga10x.bin firmware are byte-for-byte the same (verified MD5). The entire vGPU software stack -- type registration, mdev, VFIO, BOOTLOAD RPC, GSP plugin -- is present and functional in every consumer driver. The only software differences are a PCI device ID allowlist (trivially patched) and a compile flag (NV_VGPU_KVM_BUILD). NVIDIA's vGPU segmentation between a $2,000 consumer card and a $7,000+ datacenter card comes down to fuse programming in a single VF PRIV register bank (0x111xxx).

Reproduction time: ~12 hours. This entire project -- from zero to BOOTLOAD RPC reaching the GSP firmware -- was completed in a single evening/night session. The methodology is accessible to anyone comfortable with kernel module development and binary patching.

What This Is

A set of binary patches, kernel module modifications, and automation scripts that bypass all CPU-side vGPU validation in NVIDIA's 595.58.03 driver, enabling:

• vGPU type registration (60 PRO 6000 profiles across 3 scheduling modes)
• mdev device creation via VFIO
• Guest VM PCI visibility (QEMU sees the GPU as 10de:2b85)
• Device ID spoofing (0x2b85 consumer -> 0x2bb5 datacenter)
• BOOTLOAD RPC delivery to the GSP firmware with complete, valid params
• GSP firmware accepts and processes the BOOTLOAD for ~4 seconds

The GSP crashes on fused-off VF PRIV registers. See Why Blackwell Is Different for why the vgpu_unlock approach that worked on Turing cannot work here.

Requirements

• NVIDIA RTX 5090 (GB202) or similar Blackwell consumer GPU
• NVIDIA driver 595.58.03 (kernel-open source)
• Linux kernel 6.x with KVM, VFIO, mdev support
• pci=realloc kernel boot parameter

Files

File	Description
nvidia-vgpu-vfio-stub.c	Kernel module: VFIO stub, mdev driver, device ID spoof, BOOTLOAD orchestration (~1770 lines)
apply-binary-patches.py	19 binary patches for nv-kernel.o_binary (self-verifying, no proprietary code)
patch-kbuild.sh	Kbuild modifications to compile the stub into nvidia.ko
vgpu-type-register.c	Userspace tool to register vGPU types via NVA081 API (~590 lines)
setup-vgpu.sh	Full setup: load module, register types, create mdev
teardown-vgpu.sh	Clean teardown with SR-IOV disable and module unload
build-vgpu.sh	Build automation
test-vm-lspci.sh	Boot Alpine Linux VM and scan PCI
boot-vm.sh	QEMU launch script

Setup

# 1. Download NVIDIA 595.58.03 driver and extract kernel-open
# 2. Copy our files into the kernel-open directory
# 3. Apply patches:
python3 apply-binary-patches.py nvidia/nv-kernel.o_binary
bash patch-kbuild.sh
# 4. Set NV_VGPU_KVM_BUILD=1 in conftest:
sed -i 's/#undef NV_VGPU_KVM_BUILD/#define NV_VGPU_KVM_BUILD 1/' conftest/generic.h
# 5. Build:
make modules -j$(nproc) NV_KERNEL_MODULES="nvidia" CONFIG_OBJTOOL=
# 6. Compile type register tool:
gcc -o vgpu-type-register vgpu-type-register.c
# 7. Run setup:
sudo ./setup-vgpu.sh

Binary Patches (19 total)

#	Function	VA	Description
1-4	kvgpumgrPgpuAddVgpuType	0x37caff+	Skip GSP error, PCI ID, flag, init checks
5-7	kvgpumgrCreateRequestVgpu	0x384cb7+	Skip PCI ID, flag, bounds checks
8	kvgpumgrSendAllVgpuTypesToGsp	0x37c684	Stub to return NV_OK
9-10	kvgpumgrGetPgpuIndex	0x37c331+	Skip device ID and flag checks
11-12	BOOTLOAD handler	0x37b1d7+	Stub pre/post-bootload VF resource alloc
13	kfifoChidMgrFreeSystemChids	0x1e5480	Tolerate NULL channel heap
14	destroyKernelHostVgpuDeviceShare	0x379d60	Prevent GSP SHUTDOWN hang on teardown
15-16	BOOTLOAD handler	0x37b068+	Pre-alloc RPC buffer, skip memset
17-18	BOOTLOAD handler	0x37b0a6+	Skip memGetByHandle loop, bounds check
19	BOOTLOAD handler	0x37b0b8	Load FB base, jump to BAR fill

Findings

What works

• Identical binaries: GRID and consumer drivers share the same nv-kernel.o_binary and gsp_ga10x.bin (verified MD5). The vGPU code is present in every consumer driver.
• CPU-side bypass: All 19 patches successfully bypass validation. Device ID spoof (0x2b85 -> 0x2bb5) accepted at every level.
• GSP RPC delivery: BOOTLOAD RPC (cmd 0x20804001) reaches GSP with complete params: dbdf, gfid, vgpuType, numChannels, FB addresses, segment info.
• GSP processes: The GSP accepts the RPC and runs for 4 seconds -- completing significant initialization before hitting the hardware wall.
• Time-sliced mode: SET_VGPU_MIG_TIMESLICE_MODE (cmd 0x2080400f) succeeds on consumer hardware.
• Guest visibility: VMs launched via QEMU see the GPU on their PCI bus with correct vendor/device/class.

What blocks

The GSP firmware crashes (Xid 119) accessing NV_PPRIV registers at 0x111xxx:

REG 0x111424 via RM aperture: 0xbadf1002
REG 0x111424 via raw BAR0:   0xbadf1002
REG 0x000000 via RM aperture: 0x1b2000a1  (BOOT_0 -- works fine)
Error code: NV_PPRIV_SYS_PRI_ERROR_CODE_FECS_PRI_TIMEOUT

These registers are hardware fused off on consumer GPUs. Confirmed by reading through BOTH the RM's privileged IO aperture (osGpuReadReg032) AND raw BAR0 ioremap -- both return identical BADF values. A working register (NV_PMC_BOOT_0 at 0x000000) returns correct values through both paths, proving the read mechanism works and this is not a PRIV firewall issue.

Tested and ruled out

Hypothesis	Test	Result
PRIV firewall (not fuse)	Read via osGpuReadReg032 vs raw BAR0	Both return same BADF
FB address conflict	Tested at 4GB and 16GB VRAM offsets	Crash identical
SR-IOV vs time-sliced scheduling	Tested types with flags 0x008e, 0x0045, 0x0020	All crash
Missing timeslice mode enable	Called cmd 0x2080400f (returns NV_OK)	Still crashes
Device ID mismatch in GSP	Both are GB202 (chip ID 0x1b2 from BOOT_0)	Chip matches
VF present/absent	With and without pci_enable_sriov	Both crash
Incorrect RPC buffer	Dumped all fields, verified complete	Buffer is valid

Why Blackwell Is Different

The original vgpu_unlock project by DualCoder successfully enabled vGPU on Turing consumer GPUs. That approach cannot work on Blackwell due to a fundamental architecture change:

	Turing (vgpu_unlock works)	Blackwell (blocked)
vGPU plugin runs on	CPU (kernel module)	GSP (RISC-V firmware)
Plugin code modifiable	Yes -- binary patches in kernel module	No -- WPR memory + FMC signature verification
Hardware register access	Via patchable OS functions (osGpuReadReg032)	Direct MMIO from RISC-V core
Register access interception	Hook the read function, return spoofed values	Impossible -- firmware has raw hardware access
VF PRIV registers	Not needed (Turing vGPU is time-sliced on PF)	Always accessed during BOOTLOAD, regardless of scheduling mode
SR-IOV requirement	None (Turing consumer has no SR-IOV)	Consumer has SR-IOV (totalvfs=1) but VF PRIV regs are fused off

On Turing, the vGPU plugin ran as CPU code inside the kernel module. When it tried to read a fused-off register, the vgpu_unlock project could intercept the read and return a valid value. On Blackwell, the vGPU plugin runs inside the GSP firmware on a dedicated RISC-V core with direct memory-mapped hardware access. There is no software layer to intercept.

Key Technical Details

• OBJGPU device ID: offset +0x7f6 (16-bit LE field)
• PGPU table: [g_pSys+0x1f8], stride 0xbbb80 per entry, PCI ID at +0x48
• sVgpuUsmTypes: 60 entries for device 0x2bb5 (types 1460-1620, 3 flag groups)
• GSP BOOTLOAD RPC: cmd 0x20804001, params 0x19d8 bytes via rpcRmApiControl_GSP
• NVA084 BOOTLOAD ctrl: cmd 0xa084010d, params 0x648 bytes
• SET_VGPU_MIG_TIMESLICE_MODE: cmd 0x2080400f (succeeds on consumer GPU)
• PCI reset recovery: echo 1 > /sys/bus/pci/devices/.../reset (recovers GPU after GSP crash without reboot)
• Fused registers: 0x111424 (VF PRIV), 0x009080 (PRI timeout), 0x088000 (BIF) -- all return 0xbadf via both RM aperture and raw BAR0

Contributing / Next Steps

The software side is solved. The remaining challenge is hardware. The most actionable paths for continuing this work:

FPGA Interposer (most promising)

An FPGA board placed between the GPU and the PCIe slot could intercept and emulate the fused-off register accesses. When the GSP reads 0x111424 via the PRIV ring and gets a PRI timeout, an FPGA watching the register bus could inject a valid response. This would require:

• An FPGA with PCIe Gen5 capability (or a riser that taps the PRIV ring signals)
• Reverse engineering the expected register values from a datacenter GPU (or from the GSP firmware's initialization code)
• Building a register map of all fused-off addresses the GSP touches during BOOTLOAD

The GSP only runs for ~4 seconds before crashing, suggesting a relatively small number of register accesses need to be emulated.

GSP Firmware Analysis

The GSP crash buffer contains trace entries with timestamps that could be decoded if the GSP firmware's runtime memory layout were understood. The GSP mailbox registers (MAILBOX(0) = crash PC?) could map to specific functions in the gsp_ga10x.bin RISC-V ELF, revealing exactly which register access causes the crash and what value the GSP expects.

Alternative GPU Architectures

Ada Lovelace (RTX 4090) uses a similar GSP architecture but may have different fuse programming. Hopper (H100 vs consumer variants) is another candidate. The methodology in this project (binary patches, device ID spoof, RPC injection) transfers directly -- only the specific patch addresses change.

Driver Version Exploration

Older Blackwell driver versions or beta releases might have different GSP firmware with fewer register dependencies. The patch script's self-verifying approach makes it easy to test: if the original bytes don't match, the patch fails cleanly.

License

Our original code (stub, patches, scripts, tools) is released under MIT License.

This project does NOT include any NVIDIA proprietary binaries. You must obtain the NVIDIA driver separately. The binary patches modify the driver at specific byte offsets -- no proprietary code is distributed.

Acknowledgments

This research builds on the vgpu_unlock project by DualCoder for Turing GPUs, and the broader GPU virtualization research community. The Turing vs Blackwell architecture comparison in this README directly illustrates why that pioneering approach cannot extend to modern NVIDIA GPUs with GSP firmware.