Skip to content

xpustat 0.3.0

Choose a tag to compare

View all tags
@Bekhouche Bekhouche released this 23 Oct 21:37
· 13 commits to main since this release
v0.3.0
fa28815

Feature release: four new backends, a vendor-neutral device API for training scripts, and a Prometheus exporter.

New backends

  • Apple Silicon (apple). Read in-process through IOKit, with ioreg as fallback; about 2 ms per query against 25 ms for ioreg. Reports the chip name, GPU core count, utilization and the memory held by the GPU driver. Unified memory means mem_total_mb is the machine's RAM. Temperature, power and per-process usage are root-only on macOS and are not reported.
  • AWS Neuron (neuron) through neuron-ls --json-output, Google Cloud TPU (tpu) through tpu-info --format json, and Tenstorrent (tenstorrent) through tt-smi -s. These are written from the tools' documented output and have not yet been run on hardware; see vendor_devices_tests/ to help with that.

Device selection for scripts

  • Every device object now has vendor, device_index, mem_free_mb, framework_index and torch_device.
  • xpustat.pick_device() returns the least-loaded devices and honours CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ASCEND_RT_VISIBLE_DEVICES and the other vendors' variables, so the index it returns is the one the framework will see.
dev = xpustat.pick_device()[0]
model.to(dev.torch_device)     # "cuda:2", "npu:0", "mps", ...

Prometheus

  • xpustat --serve-prometheus 9400 serves /metrics for every vendor from one process; xpustat --prometheus prints the text once; xpustat.to_prometheus(stats) is the library form. No new dependencies.

Fixes

  • Index access on vendor collections. stats.amd[0] raised TypeError; it works as the README has always shown.
  • Faster vendor commands. The two nvidia-smi and three amd-smi invocations run concurrently: 246 ms to 172 ms on a seven-GPU NVIDIA host, 460 ms to 167 ms on ROCm 10.
  • Watch mode no longer swallows exceptions.
  • Process lines render as user:command/pid instead of running user and command together.
  • pip install xpustat[nvidia] now pulls nvidia-ml-py; the pynvml package is deprecated.

Documentation

A docs/ directory with eight pages (installation, CLI, Python API, device selection, Prometheus, backends, contributing) and a mkdocs.yml.

Tested hardware

Vendor Device
Apple Apple M5, 10-core GPU, macOS 26
AMD Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0
NVIDIA 7x NVIDIA L4 (cc8.9), nvidia-smi and nvidia-ml-py paths

Upgrade

pip install -U xpustat
Assets 2
Loading