v0.2.1

Since v0.2.0:

• docs: a short README; the detail lives in the other docs (@wanjohiryan)
• docs: a shorter ARCHITECTURE, brought up to date with v0.2 (@wanjohiryan)
• security: a shorter SECURITY, with a reporting address (@wanjohiryan)
• security: any socket other than AF_UNIX (@wanjohiryan)
• ci: send the draft's since date in UTC, so authors resolve to @login (@wanjohiryan)
• driver: avoid recreating the kernel-owned proc driver directory (@wiktorglowacki)
• device: recover SHM capacity after failed window placement (@wiktorglowacki)
• test(device): initialize real GPU mapping fixture (@wiktorglowacki)
• chore: Bump to v0.2.1 (@wanjohiryan)

v0.2.0

TLDR; CUDA in the guest, a VRAM limit, a backend that decides what a guest may reach and more security fixes.

Huge rewrite that closed a lot of security gaps... here is the AI overview:

What changed

	v0.1.1	v0.2.0
CUDA	refused, unless --allow-uvm-unsafe	works, with --caps ...,compute and a VMM that offers the UVM aperture
What a guest is served	everything, less two escape flags	--caps, default graphics,video,utility
Escape flags	--allow-root-unsafe, --allow-uvm-unsafe	removed; no flag turns a check off
RM controls and classes	forwarded	only those RM serves to an unprivileged caller, from RM's own tables per release
Pointers inside RM controls	forwarded	the backend supplies every buffer RM reads through one
Memory registered by CPU address	refused	rebuilt from the guest's own pages
UVM calls	refused	size-checked per release; descriptors and their RM client checked
--vram-limit-mib	did not exist	enforced; the guest's nvidia-smi, Vulkan and CUDA report the limit
Backend process	no root, no capabilities	also seccomp and Landlock, refused at start without them
One RM control, in the guest	35.5 us	19.3 us

Tested on

	RTX 3060, 595.104.02	RTX A2000, 615.71.09
offscreen draw, pixel check	pass	pass
vulkaninfo	pass	pass
vkcube + Vulkan Video H.264 1280x720@60, stream decoded on the host	628 frames	605 frames
CUDA: context, 1 MiB to the device and back	10 of 10	10 of 10
--vram-limit-mib: CUDA and Vulkan stop at the limit with out-of-memory	4096 MiB: pass	2048 MiB: pass
guests encoding 720p60 at once, every stream decoded	12	12

Numbers and method are in BENCHMARKS.md. Each row ran once, on one box.

Upgrading

• Replace --allow-uvm-unsafe with --caps graphics,video,utility,compute. Without compute, /dev/nvidia-uvm is refused and CUDA reports CUDA_ERROR_UNKNOWN.
• --allow-root-unsafe is gone. Run the backend as an unprivileged user that can open /dev/nvidia* and the render node.
• CUDA needs the VMM to offer the UVM aperture, shared-memory region 2. nesbox does from 8087ff9 on dev. With an older VMM, cuCtxCreate fails and graphics are unaffected.
• The backend needs a kernel with seccomp and Landlock.
• --vram-limit-mib on a driver release with no video memory table of its own is refused at start. Tables ship for 535.129.03, 580.178.04, 595.71.05, 595.104.02 and 615.71.09.

Known issues

• The video memory limit does not count memory RM allocates inside other objects, such as channel and context buffers: about 34 MiB for one encoding guest.
• An encode through NVENC counts against the driver's session cap, 12 per card on a GeForce at 595.104.02. Vulkan Video, which the capture layer uses, does not count on that release.
• One backend serves every process in a guest. Processes inside one guest are not isolated from each other.
• Tests that attack the backend on purpose, and fuzzing, are not written yet.

PR: #5

Since v0.1.1:

• style: cargo fmt (rustfmt 1.10) (Wanjohi)
• rig: build on one host, test on another, from a workstation that builds nothing (Wanjohi)
• backend: split nvidia.rs into a module per message kind (Wanjohi)
• rig: build nesbox per host, cargo on PATH, quieter builds (Wanjohi)
• backend: --caps decides what a guest is served; the escape flags go (Wanjohi)
• driver: create device nodes from the caps the backend serves (Wanjohi)
• rig: a probe for the device nodes each capability set should give (Wanjohi)
• gen: rmctrl_extract.py, the RM controls whose parameters carry a user pointer (Wanjohi)
• rig: rmbench, the cost of one forwarded RM control (@wanjohiryan)
• backend: per-call logging off the hot path, guarded buffers reused (@wanjohiryan)
• backend: keep the request and response buffers across chains (@wanjohiryan)
• rig: BACKEND_WRAP runs the backend under another command (@wanjohiryan)
• gen: RM control pointer tables, and the controls the probe was dropping (@wanjohiryan)
• docs: SECURITY.md, what a guest can and cannot reach on v0.2 (@wanjohiryan)
• driver: one completion per request, and the control queue under one lock (@wanjohiryan)
• rig: probes run detached on the GPU host; rmbench-mt (@wanjohiryan)
• backend: no guest pointer reaches RM inside a control's parameters (@wanjohiryan)
• docs: BENCHMARKS.md, the cost of one forwarded RM control (@wanjohiryan)
• driver: carry what an RM control's pointers address, per pointer (@wanjohiryan)
• backend: refuse what an older release's table cannot vouch for (@wanjohiryan)
• driver: spin 10 us for a control-queue answer before sleeping (@wanjohiryan)
• backend: announce the segment encoding, and trust a length by where it came from (@wanjohiryan)
• backend: seccomp, and the sandbox goes on before the first thread (@wanjohiryan)
• docs: SECURITY.md, what the backend's own process can reach (@wanjohiryan)
• backend: every forwarded ioctl goes through one host trait (@wanjohiryan)
• ci: keep a draft release listing every commit since the last one (@wanjohiryan)
• ci: a repo with no release yet drafts from all of history (@wanjohiryan)
• ci: updating the draft keeps its tag (@wanjohiryan)
• abi: the controls and classes RM exports to an unprivileged caller (@wanjohiryan)
• ci: credit commit authors by GitHub login (@wanjohiryan)
• rig: a probe that makes a Vulkan device with each extension a guest is offered (@wanjohiryan)
• abi: the three RM paths that carry no control flag (@wanjohiryan)
• backend: the guest is told what RM sizes an allocation at (@wanjohiryan)
• docs: what the RM allowlist refuses (@wanjohiryan)
• abi: what a video memory limit needs to know about each RM release (@wanjohiryan)
• backend: drop two needless mut in the allocation-size tests (@wanjohiryan)
• gen: read the UVM command table out of each release (@wanjohiryan)
• vklayer: stop offering a guest the Vulkan extensions it cannot create (@wanjohiryan)
• backend: the UVM calls a guest may make, and the descriptors in them (@wanjohiryan)
• ci: take the draft release's version from Cargo.toml (@wanjohiryan)
• ci: keep the draft workflow's script inside its YAML block (@wanjohiryan)
• guest: the UVM calls the host takes, and where their descriptors sit (@wanjohiryan)
• v0.2.0 (@wanjohiryan)
• vklayer: hide only from forwarded NVIDIA devices (@wanjohiryan)
• Cargo.lock: catch up with 0.2.0 (@wanjohiryan)
• backend: a guest does not get to name memory by a CPU address (@wanjohiryan)
• backend: hold the heap discriminator to exactly one function (@wanjohiryan)
• backend: one host address built from a guest's own pages (@wanjohiryan)
• backend: the transport hands over guest RAM (@wanjohiryan)
• backend: the hClient beside a UVM descriptor must be one made on that file (@wanjohiryan)
• protocol: the guest-physical pages behind a registered address (@wanjohiryan)
• backend: tell the guest where each route keeps its address (@wanjohiryan)
• style: cargo fmt (rustfmt 1.10) (@wanjohiryan)
• driver: pin the pages behind a registration and send where they are (@wanjohiryan)
• rig: say so when a test binary dies before it can report (@wanjohiryan)
• device: give RM the guest's own pages instead of refusing it (@wanjohiryan)
• device: -1 in a UVM descriptor field means none, and goes through (@wanjohiryan)
• rig: a CUDA probe, and what it says is still missing (@wanjohiryan)
• uvm: place CUDA semaphore pools in an aperture of their own (@wanjohiryan)
• rig: send nesbox's tracked files only (@wanjohiryan)
• rig: the draw, vulkan and encode probes, and a decoded encode check (@wanjohiryan)
• docs: CUDA works in a guest, and the UVM aperture (@wanjohiryan)
• rig: density, N guests at once on one card (@wanjohiryan)
• backend: resolve a render node against the list the guest was given (@wanjohiryan)
• rig: GPU_VCPUS and GPU_MEM_MIB size the guest (@wanjohiryan)
• rig: host-encode-cap, the driver's encode session cap by API (@wanjohiryan)
• benchmarks: twelve encoding guests per card, and the session cap by API (@wanjohiryan)
• backend: hold a guest to --vram-limit-mib (@wanjohiryan)
• docs: the video memory limit holds, and what it does not count (@wanjohiryan)
• security: UVM calls are size-checked now; drop it from what is open (@wanjohiryan)
• security: list the UVM size check among what is refused (@wanjohiryan)
• docs: per-directory READMEs for v0.2, and today's sections in the house style (@wanjohiryan)
• rig: the vulkan probe takes its paths from guest-nvidia-env.sh (@wanjohiryan)

v0.1.1

Security fixes for v0.1.0. If you run v0.1.0, upgrade.

What changed

	v0.1.0	v0.1.1
Backend run as root	allowed	refused, unless --allow-root-unsafe
Backend capabilities	kept	dropped at start, no_new_privs set
/dev/nvidia-uvm	forwarded	refused, unless --allow-uvm-unsafe
UVM_INITIALIZE flags	the guest's (and wrong)	the backend's
ABI profile	picked from the guest's first version check	picked from the host at start; no profile, no start

• A backend running as root made every guest process an RM administrator. It now refuses to start as root or with CAP_SYS_ADMIN. scripts/run-guest.sh runs it as NVGPU_USER.
• UVM calls went to the host without a size table, and some carry descriptor numbers the backend doesn't translate. UVM is off until v0.2, which brings compute properly.
• The guest set bit 2 on UVM_INITIALIZE for sharing mode. On 610.43.02 and later that bit is DISABLE_PAGEABLE_ACCESS, and older UVM refuses the call outright. The backend now sends sharing mode with HMM and pageable access off.
• Until the guest's first version check, RM ioctls went through unchecked. The profile is now set before the first guest message.

Tested on

	RTX 3060, 595.104.02	RTX A2000, 615.71.09
offscreen draw, pixel check	pass	pass
vulkaninfo	pass	crashes, same as v0.1.0 [1]
vkcube + NVENC H.264 1280x720@60	621 frames in 10 s	vkcube crashes, same as v0.1.0 [1]

[1] That host has nvidia_modeset loaded but no /dev/nvidia-modeset node. v0.1.0 crashes at the same place.

Upgrading

• Run the backend as an unprivileged user that can open /dev/nvidia* and the GPU's render node (the render group).
• Keep the backend binary out of /root so that user can reach it.
• CUDA needs --allow-uvm-unsafe for now. The name is honest about what it costs.

Known issues

• Pointer handling inside RM controls isn't finished. Run only guests you trust.
• No CUDA without --allow-uvm-unsafe.

PR: #4

v0.1.0

Run NVIDIA's own drivers inside a KVM virtual machine, sharing one card with the host and other VMs, at near-native speed. Experimental: you build it yourself.

Features

	v0.1.0
Graphics
Vulkan, rendering on the real GPU	✅
OpenGL / EGL	untested
NVENC video encoding	✅
NVDEC video decoding	untested
Compute
CUDA	⚠️ detected, untested
OpenCL	untested
Sharing
Several VMs on one card	✅ 4 tested
Per-VM video memory limit	—
Virtual machine monitors
nesbox	✅
crosvm	—
Security
Unknown driver calls refused	✅
Driver controls filtered	—
Backend sandboxed	—
Driver releases tested on hardware	595.99.02 (RTX 3060), 615.71.09 (RTX A2000)

✅ works · ⚠️ partly · — not supported · untested: may work, never checked

Speed

On an RTX 3060, a guest renders within 2% of the same machine on bare metal for any frame over ~2 ms, and uses the same CPU. Four guests on one card get an even share each. Details: BENCHMARKS.md.

How it compares

	virtio-nvgpu	VFIO passthrough	NVIDIA vGPU	virtio-gpu (Venus)
Share one card between host and VMs	✅	— (one VM owns it)	✅	✅
Consumer GeForce cards	✅	✅	—	✅
NVIDIA's own drivers in the guest	✅	✅	✅	— (translated Vulkan)
Near-native speed	✅	✅	✅	⚠️
Hardware isolation (IOMMU)	—	✅	✅	—
License needed	—	—	✅	—

Not for mutually untrusted users. The guest talks to the host's NVIDIA driver, with no IOMMU boundary in between. For tenants who don't trust each other, use VFIO or vGPU.

Known issues

• Tested with at most four guests, and only one running a real game.
• On the RTX A2000, one of two game launches lost its video capture; the cause is unknown.

Try it

• Host: cargo build --release -p device --features vhost-user
• Guest: make -C driver KDIR=<guest kernel build tree>

README.md explains the layout, ARCHITECTURE.md the design.