fix(deployer): stop granting privilege to plugin-allocated devices

[thxCode]

What

A privileged container receives all device nodes of the host, so it enumerates — and can use — every accelerator on the node, whatever the device plugin allocated to it. On a multi-card node that undoes both slicing modes.

Drop privilege whenever the devices come from a device plugin: any device plugin resource family (nvidia.com/gpu, nvidia.com/gpu.sliced*, nvidia.com/gpu.partitioned*, and the same shapes for every other vendor in GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_CDI), and every mapped device request under the KDP injection policy. Device requests the deployer resolves itself through visible-devices envs keep their privilege, as does a container that asks for no device at all.

Why — measured on an 8×H100 node with MIG enabled on exactly one card

Before, with privileged: true (which every GPUStack inference backend sets unconditionally):

pod	NVIDIA_VISIBLE_DEVICES	nvidia-smi -L
MIG, not privileged	its MIG UUID	GPU 0 + its one MIG device, nothing else
MIG, privileged	its MIG UUID (correct)	all eight cards, plus both MIG devices on card 0 — including another pod's
sliced 50 %, privileged	its whole-card UUID (correct)	all eight cards, plus both MIG devices on card 0

NVIDIA_VISIBLE_DEVICES is correct in all three, so it is not what enforces the bound — it is consumed by the container runtime at creation to decide which device nodes to mount, and a privileged container gets the whole /dev regardless.

The leak was functional, not cosmetic. In the sliced pod HAMi-core set CUDA_DEVICE_MEMORY_LIMIT_0=40779m; reported memory.total was 40779 MiB at index 0 and 81559 MiB uncapped at indexes 1–7. That pod was allocated GPU-a158631e, which was index 1 inside the container — index 0 was the MIG card. Once the device set is not narrowed, a container-local _LIMIT_0 no longer names the assigned card.

After

Same node, two real vLLM model instances (Qwen3.5-0.8B on gpustack/runner:cuda12.8-vllm0.17.1):

	By Ratio 50 % / 50 %	By Profile 3g.40gb
limits	gpu.sliced: 1, sliced.memory-percentage: 50, sliced.cores-percentage: 50	gpu.partitioned: 1, partitioned.mig-3g.40gb: 1
privileged	false	false
nvidia-smi -L	one GPU, its own — the other seven are gone	GPU 0 + exactly one MIG device, its own
memory.total	40779 MiB at index 0, which now is the allocated card	[Insufficient Permissions] (NVML refusing whole-card queries from inside a MIG container)
inference	running, 2+2 → 4	running, 2+2 → 4

Tests

tests/gpustack_runtime/deployer/test_privileged.py — 13 parametrized cases: no execution, privilege not requested, no resources, CPU/memory only, specific whole cards under env and under kdp, all-devices under env, auto-map under kdp, exclusive whole card, soft slice, hard partition, a non-NVIDIA soft slice, and the boundary case nvidia.com/gpu-alike (merely prefixed by a CDI kind — keeps its privilege).

make test: 62 passed, 16 skipped. pre-commit clean on both files.

Not in scope

In env injection mode a privileged container asking for specific devices is deliberately widened to every device and then narrowed at the backend layer via CUDA_VISIBLE_DEVICES; docker.py / podman.py carry the identical shape. That predates this change and leaks the same way on a multi-card host, but it is not the Kubernetes divided-mode path.

[gemini-code-assist]

Code Review

This pull request introduces logic to drop container privileges when devices are allocated by a device plugin, preventing workloads from bypassing slicing limits. It adds helper functions _is_device_plugin_resource and _resolve_privileged in the Kubernetes deployer, along with corresponding unit tests. Feedback was provided regarding a potential TypeError if envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES is None, suggesting a fallback empty dictionary.

[gemini-code-assist]

The envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES environment variable map can potentially be None according to its type definition in envs.py. If it is None, performing an in membership check will raise a TypeError.\n\nTo prevent potential runtime crashes, please use a fallback empty dictionary or {} when performing the membership check, similar to how envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_CDI is handled in _is_device_plugin_resource.

Suggested change

	if _is_device_plugin_resource(r_k) or (
	kdp
	and (
	r_k
	in envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES
	or r_k == envs.GPUSTACK_RUNTIME_DEPLOY_AUTOMAP_RESOURCE_KEY
	)
	):
	if _is_device_plugin_resource(r_k) or (
	kdp
	and (
	r_k
	in (envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES or {})
	or r_k == envs.GPUSTACK_RUNTIME_DEPLOY_AUTOMAP_RESOURCE_KEY
	)
	):

[thxCode]

Not applicable — this map can never be None at runtime.

envs.__getattr__ always evaluates the lambda in variables, and the lambda is to_dict(getenv(...)); to_dict short-circuits to {} on an empty value:

>>> from gpustack_runtime import envs
>>> type(envs.GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES)
<class 'dict'>
>>> from gpustack_runtime.envs import to_dict
>>> to_dict('')
{}

The | None in the module-level annotation is a placeholder for the lazy-attribute declaration, not a reachable value. Six existing call sites read this same map unguarded — deployer/__types__.py:1391, deployer/kuberentes.py:1137 and :1141, deployer/docker.py:1004, deployer/podman.py:1003, deployer/cdi/__types__.py:680 — so a guard here would be the only one of its kind.

You did point at a real inconsistency, though: I had or {} on the CDI map two functions up and not here. Resolved in the other direction — the stray or {} is gone, so both reads now match the idiom used everywhere else in the file. Leaving this thread open since it is the opposite of the suggested change.

[Copilot]

Pull request overview

This PR tightens Kubernetes deployment security by preventing privileged containers from bypassing device-plugin device isolation (e.g., MIG, slicing/partitioning) and unintentionally gaining access to all host accelerators.

Changes:

• Introduces privilege-resolution logic that drops privileged when devices are provided by a device plugin (and for mapped device requests under the KDP injection policy).
• Wires the new privilege resolution into Kubernetes Pod security context creation.
• Adds a parametrized test suite covering key privilege/resource-policy combinations and boundary cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
gpustack_runtime/deployer/kuberentes.py	Adds device-plugin detection and _resolve_privileged(); uses it when building Kubernetes container security contexts.
tests/gpustack_runtime/deployer/test_privileged.py	Adds parametrized unit tests for the new privilege-resolution behavior across env/KDP policies and resource shapes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thxCode]

The stated failure mode does not occur. With an empty CDI kind, resource_key.startswith(f"{cdi}.") is resource_key.startswith(".") — that requires the key to begin with a dot, not to contain one:

>>> "nvidia.com/gpu.sliced".startswith(".")
False
>>> "nvidia.com/gpu.sliced" == ""
False

A Kubernetes resource name is a qualified name and cannot start with ., so an empty CDI kind matches nothing rather than everything. cpu / memory are skipped before this call anyway.

An empty value is reachable in the map (to_dict("nvidia.com/devices=") yields {"nvidia.com/devices": ""}), so the premise is right — but the consequence is a misconfigured entry that matches no key, which is the misconfiguration's own failure, not a misclassification here. Skipping falsy kinds would be a no-op guard, so I have left the code as is.

[thxCode]

Released as gpustack-runtime 0.2.2.post7 (tag v0.2.2post7).

Consumed downstream by:

• refactor: align with the gpustack-operator v0.8.0 partition ledger gpustack#5984 — pins 0.2.2.post7; carries the operator v0.8.0 bump and the partition-ledger schema.
• fix(gpu-service): align partition availability with the operator ledger gpustack-ui#1309 — the matching vGPU form alignment.

On provenance, to be precise: the after-state in the description was measured with 0.2.2.post7.dev1, built from this branch at 0ead756 and force-installed over the GPUStack worker image, driven through the real deployment path (Kubernetes deployer → device manager → vLLM model instance). The only delta between that build and what was released is the removal of an unreachable or {} guard during the review pass — no behaviour change — but the released artifact itself was not re-driven end-to-end.