Skip to content

xpustat 0.4.0

Choose a tag to compare

View all tags
@Bekhouche Bekhouche released this 27 Oct 08:51
· 10 commits to main since this release
v0.4.0
d1cee4f

Feature release: the pieces a scheduler or training script needs around the device list, one more NVIDIA form factor, and a way to add vendors from outside the package.

Device selection for the shell

  • xpustat --select N picks the N best devices and prints the assignment that restricts a job to them, one line per vendor: CUDA_VISIBLE_DEVICES=1,2, HIP_VISIBLE_DEVICES=0, ... Use it as export $(xpustat --select 2). --by, --min-free and --uuid refine the pick.
  • xpustat.visible_devices_env(devices, format="index"|"uuid") is the library form.
  • TPU_VISIBLE_CHIPS, TT_VISIBLE_DEVICES and NEURON_RT_VISIBLE_CORES are honoured, matching Ray's accelerator managers. Neuron is handled at core level: ranges such as 0-3 are parsed and a device is visible when any of its cores is.
  • Every device carries ray_resource (GPU, NPU, TPU, neuron_cores, ...) and k8s_resource (nvidia.com/gpu, amd.com/gpu, habana.ai/gaudi, ...). Kubernetes names that could not be confirmed are None.

NVIDIA

  • Jetson (jetson): Nano, Xavier and Orin modules through tegrastats: shared RAM, GPU load, temperature and the power rail the module exposes. Silent when nvidia-smi exists, so a GPU is never listed twice. Parsers are tested on all three line formats; the backend has not yet been run on a board.
  • MIG: on the NVML path a GPU in MIG mode lists its compute instances in mig_instances, with UUID, instance ids, memory and processes. Shown as sub-lines in the CLI and exported as devices to Prometheus. Written from the NVML API; not yet run on a MIG-enabled GPU.

Plugin backends

  • A package can add a vendor through the xpustat.backends entry point, or at runtime with xpustat.register_backend(name, cls). The vendor then appears in query_all, to_dict, the CLI, pick_device and the exporter. Contract and example in docs/contributing.md.

Query cache

  • query_all(max_age=seconds) reuses a recent result; --cache-ttl does the same for --serve-prometheus, for scrapes that arrive faster than the vendor tools answer.

Tested hardware

Vendor Device
Apple Apple M5, 10-core GPU, macOS 26
AMD Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0
NVIDIA 7x NVIDIA L4 (cc8.9), nvidia-smi and nvidia-ml-py paths

Upgrade

pip install -U xpustat
Assets 2
Loading