Add an absolute GPU-memory budget for unified-memory systems

[randomvariable]

Objective

Add an optional absolute GPU-memory budget in GiB for unified-memory systems, alongside gpu_memory_utilization.

Target hardware

• NVIDIA DGX Spark / GB10 / SM121 unified memory
• AMD Strix Halo / gfx1151 unified memory

Motivation

A fraction of device-visible memory is unstable when CPU and GPU share one moving memory pool. A fixed GiB ceiling may make vLLM allocation easier to bound under host-memory pressure; this issue must validate whether it improves stability.

The fork currently has fractional gpu_memory_utilization and absolute kv_cache_memory_bytes, but no whole-engine absolute budget. kv_cache_memory_bytes is not equivalent because it only overrides KV-cache sizing; graph reservations, profiling, validation, and OOM diagnostics still reason from the fractional engine budget.

Reference

• https://github.com/eugr/spark-vllm-docker/tree/main/mods/gpu-mem-util-gb
• Related unified-memory startup safety work: [Bug]: gpu_memory_utilization is not respected on GB10 unified-memory systems during startup profiling (host wedges) vllm-project/vllm#46307 and [Bugfix][Worker] Respect gpu_memory_utilization on integrated (unified-memory) GPUs; fail clean instead of wedging the host vllm-project/vllm#49760
• No matching upstream vLLM PR or issue found for an absolute gpu_memory_utilization_gb control as of 2026-07-29.

Proposed work

• Add a nullable absolute-GiB field and CLI argument using vLLM config conventions.
• Define precedence and mutual-exclusion behavior relative to gpu_memory_utilization.
• Thread the effective budget through memory profiling, graph-capture estimates, KV allocation, usage reporting, validation, and OOM errors.
• Keep behavior unchanged when the option is unset.
• Cover CUDA and ROCm integrated-GPU paths without platform-specific branches in generic config code.

Acceptance criteria

• Config and CLI tests cover unset, valid, invalid, and conflicting values.
• Profiling and allocation use the same effective byte budget.
• Error messages report the active absolute budget.
• On a documented model/config, perturb host free memory across a recorded range and report process RSS, device-visible usage, OOM outcome, and peak overshoot above the requested budget.
• No throughput or capacity regression when the feature is unset.

[randomvariable]

Current-model applicability expansion

Absolute budgeting remains model-neutral, but accounting must include more than checkpoint weights and paged KV for current releases. Add explicit estimates or measured reservations for multimodal encoder/projector residency, variable media-token expansion, convolutional or Mamba recurrent state, mixed full/SWA cache retention, and native/external draft caches.

Useful stress rows include Step-3.7-Flash, MiMo-V2.5, Nemotron 3 Nano Omni, Mistral Small 4, and LFM2.5. Their architecture makes omitted state visible; it does not prove current fork support.

[randomvariable]

Status: partial.

216bea62c ships opt-in gpu_memory_utilization_gb, mutual exclusion/validation, and CPU regression coverage for effective budgeting, profiling, KV sizing, graph-estimate failure handling, and startup-plan interaction. The issue remains open because its unified-memory acceptance test is hardware-dependent: perturb host free memory and report requested-budget overshoot, process RSS, device-visible use, and OOM behavior on GB10 and gfx1151. Next gate: publish that measured residency matrix.