Repository navigation
Found in the ai-rm end-to-end run (ai-rm#32): two training processes started on a node outside Kubernetes held both
of its RNGD cards, the device plugin and the AcceleratorProfiles reported them free, and a decode pod placed there
failed with EBUSY (#9).
Added
- Card occupancy (
rm_abstract.airm.occupancy): per card, the holders fromfuriosa-smi ps --format json, the
Rebellions driver'sctx_info/rbln-stat --jsoncontexts and/proc/<pid>/fd, each classified by
/proc/<pid>/cgroupask8s(pod) orhost. Published inspec.extraasinUseandholders(pid, executable
basename, managed). A card held by a host process also getshealth: busyandhealthy: "false", which
rm-controller already skips.AIRM_OCCUPANCY=0turns it off; without the host PID namespace no occupancy is
published (with a warning). - Node label
airm.ai-computing.io/host-held=<n>;/metricsseriesrm_accel_accelerators_in_useand
rm_accel_accelerators_host_held. - Chart values
hostPID(defaulttrue) andoccupancy(defaulttrue).
Verified on: furiosa-smi 2026.4.0, rbln-stat 3.0.0, Python 3.10/3.11, cgroup v2 + containerd. Read-only dry-run on two NPU nodes: host-held RNGD cards reported busy, idle cards free; rm-controller excludes the busy cards (#10).
Install: chart deploy/helm/rm-abstract-node 0.6.0 (hostPID: true is needed to see host processes), or pip install https://github.com/ai-computing/rm-accel/archive/refs/tags/v0.5.0.tar.gz.
Limits: GPUs not covered; a busy ATOM and k8s-held cards are verified by fixtures only.