xpustat 0.7.0

Feature release: xpustat follows a training run from inside the loop, the cluster page streams from every host over one ssh session each, and the server now carries alerts, history and a JSON Schema for its payload.

In a training loop

• xpustat.monitor() samples the devices on a thread while your code runs, as a context manager or an iterator, and reports peaks and means per device (peak("util_pct"), mean("mem_used_mb"), summary()).
• xpustat.callbacks: HFTrainerCallback, LightningCallback and KerasCallback log device metrics to TensorBoard, W&B, MLflow, print or any callable at every logging step. The frameworks are imported only when a callback is constructed; psutil stays the only dependency.

Cluster page

• One streaming ssh session per host: the remote runs xpustat --json --meta -i N and the server reads one JSON line per refresh, so the cost no longer grows with a command per host per refresh. Connections are multiplexed with ControlMaster; a remote older than 0.7.0 falls back to a command per refresh, and --no-stream forces that path.
• --history 15m|2h keeps the sparkline history on the server behind /history, so every tab shows the same history from the moment it opens. --history-file FILE.sqlite writes a replayable log; xpustat --replay FILE.sqlite plays it back.
• xpustat --select N --ssh node1 node2 ... and /free pick the freest devices across hosts (xpustat.pick_remote). /metrics covers every host with a host label plus xpustat_host_up and xpustat_host_stale (xpustat.to_prometheus_hosts).
• Hosts show their real hostname: xpustat --json --meta wraps the devices in hostname, time and version, and --json -i N streams one line per refresh.
• Page: fixed column widths so host tables align, a user filter, CSV export, an older xpustat badge on hosts that cannot stream.

Alerts

• xpustat --check exits 1 when a device is over --max-temp, under --min-free, reports ECC errors, is throttled or unhealthy, or when a host is down; --no-alert-throttle ignores throttling. Works with --hosts and --ssh.
• The page shows an alerts bar and sends desktop notifications; --alert-url posts new alerts to a webhook, --alert-cooldown keeps a repeated condition to one message.

Schema and deployment

• xpustat --schema prints a JSON Schema for the payload and xpustat.validate_payload checks one. A test holds every vendor's keys to it, which showed that Gaudi, Intel, Huawei, Hygon, Cambricon and Moore Threads emit no processes key; the schema says so.
• deploy/: Dockerfile, systemd unit, Kubernetes DaemonSet and a Grafana dashboard, documented on a new Deployment page. CI publishes the docs to GitHub Pages.

Fixes

• The ssh control socket path under the macOS temp directory was longer than a Unix socket allows; the sockets now live under /tmp.
• A non-interactive ssh shell has no ~/.local/bin on its PATH, so an xpustat installed with uv tool or pip install --user was invisible to --ssh; the default remote commands add it.

Tested hardware

Platform	Device
Apple	Apple M5, 10-core GPU, macOS 26
NVIDIA	7x NVIDIA L4 (cc8.9), dual Xeon Silver 4314, Linux
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0, Ryzen 9 9950X3D
CPU only	Intel N200 (Linux)

All four were shown on one cluster page over streaming ssh, with one unreachable host alongside; 144 tests pass on macOS and 141 on Linux with 3 skipped. The Windows host was unreachable during this release, so native Windows and WSL were not run this time; CI covers Windows for the tests that need no hardware.

Upgrade

pip install -U xpustat

xpustat 0.6.0

Feature release: a browser page for one machine or a whole cluster, served by the exporter with no new dependency, and continuous integration on three platforms.

Web UI

• xpustat --ui serves http://127.0.0.1:9400/: one self-contained HTML page, nothing loaded from the network, that shows every -a column (memory bar, utilization, temperature, power, fan, clocks, P-state, health badges), MIG instances under their GPU, the CPU row with --cpu, and two sparklines per device for the time the tab is open.
• Click a device for its detail: UUID, free memory, hotspot and memory temperatures, codec load, ECC counts, throttle reasons, energy, PCIe link, ANE power on Apple, and the processes with PID, user, device memory, SM share, host RSS, CPU share, container or SLURM job and full command line.
• Host, vendor and text filters, a compact mode so forty devices fit on one screen, expand all, pause; /?expand starts with every row open.
• /json, /metrics and the new /cluster answer on the same port, so one process serves the browser, scripts and Prometheus. --port and --host set the bind address; the UI binds to loopback by default because the page shows command lines.

Cluster page

• xpustat --ui --ssh node1 node2 ..., --hosts http://node:9400 ... or --hosts-file lab.txt put every host on one page with a summary bar: hosts up and down, devices and how many are free, device memory used, processes, the hottest device, throttled and faulted devices.
• The server polls the hosts itself on the -i timer, in parallel, and caches the answer: many open tabs cost one ssh fan-out. A host that stops answering keeps its last snapshot, marked stale with the error; a host that never answered is shown as unreachable; the others are unaffected.
• This machine is the first block unless --no-local is given. --ssh-command and --timeout work as for the terminal cluster view.
• From Python: xpustat.ui.serve_ui() and xpustat.ui.ClusterCollector.

Continuous integration and fixes

• GitHub Actions runs the tests on Ubuntu, macOS and Windows with Python 3.9 and 3.13, a CLI smoke run on a machine without accelerators, the sdist and wheel build, twine check and a strict docs build, on every push and pull request.
• Prometheus: a textual P-state such as P2 from a plugin backend is exported as 2, and states that are not numbers are skipped instead of raising.

Tested hardware

Platform	Device
Apple	Apple M5, 10-core GPU, macOS 26
NVIDIA	7x NVIDIA L4 (cc8.9), dual Xeon Silver 4314, Linux
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0, Ryzen 9 9950X3D
CPU only	Intel N200 (Linux)
Windows 11	Ryzen 7 5800X3D, native Python 3.12 and WSL2

All five, plus one unreachable host, were shown on one cluster page polled over ssh; the suite passes on macOS, Linux and native Windows.

Upgrade

pip install -U xpustat

xpustat 0.5.0

Feature release: the host CPU as a device, per-process and health detail on NVIDIA and AMD, Apple Silicon sensors without sudo, a cluster view, diagnostics, recording, two new NPU vendors, and a native Windows run.

Host CPU, opt-in

• xpustat --cpu and query_all(include_cpu=True) add one [CPU] row: model, ISA, cores and threads, 4P+6E layout on Apple Silicon, sockets, NUMA nodes, frequency, load average, RAM and swap, temperature and power where the OS exposes them, cgroup limits in containers, and the top processes by resident memory. The cpu JSON key appears only when asked, so existing output is unchanged.

Processes

• Per-process compute utilization (sm_pct, from NVML samples or nvidia-smi pmon), memory-bandwidth and codec utilization, full command line (--full-cmd), host RSS and CPU.
• Container or job attribution on every process from its cgroup: docker, containerd, podman, Kubernetes pod, SLURM job. Shown with -k, exported as a container label.

Health and hardware columns

• NVIDIA and AMD: fan (-F), SM/graphics and memory clocks (-C), encoder/decoder load (-E), and health (-H): P-state or perf level, active throttle reasons, uncorrected ECC. throttled and healthy on every device, energy counters where available, PCIe link, power limit.
• MIG inventory on the nvidia-smi path; Jetson engines (NVENC, NVDEC, DLA, ...) and nvpmodel mode.
• --id 0,2 to pick devices, --print-completion bash|zsh|fish.

Apple Silicon

• GPU temperature, GPU power and ANE power without sudo, read in-process from the SMC and the IOReport energy counters; the CPU row gets temperature and power too. Free memory is the OS "available" figure.

Several machines, diagnostics, recording

• --hosts http://n1:9400 ... reads other exporters' new /json endpoint; --ssh n1 n2 runs xpustat over ssh. One table with a host column; an unreachable host never hides the others.
• --doctor explains every backend: tool path, probe answer, native binding, env vars, device count, errors.
• --record FILE writes one JSON frame per line (gzip optional); --replay FILE renders it.

New backends and fixes

• Rebellions (rbln-stat --json) and FuriosaAI (furiosa-smi), written from the tools' JSON schemas and awaiting a hardware run; RBLN_DEVICES and FURIOSA_DEVICES honoured.
• Plugin backends through the xpustat.backends entry point or register_backend().
• The optional amdsmi and nvidia-ml-py bindings no longer break import xpustat when they fail to load against the installed driver.
• Windows: the console is switched to UTF-8 so the table renders on cp1252 consoles, and the CPU model is read from the registry.
• Sole author: Salah Eddine Bekhouche; Hichem Telli is acknowledged for testing the Ascend backend.

Tested hardware

Platform	Device
Apple	Apple M5, 10-core GPU, macOS 26
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0, Ryzen 9 9950X3D
NVIDIA	7x NVIDIA L4 (cc8.9), nvidia-smi and nvidia-ml-py paths, dual Xeon Silver 4314
CPU only	Intel N200 (Linux)
Windows 11	Ryzen 7 5800X3D, native Python 3.12: suite, CLI, JSON, doctor

Upgrade

pip install -U xpustat

xpustat 0.4.0

Feature release: the pieces a scheduler or training script needs around the device list, one more NVIDIA form factor, and a way to add vendors from outside the package.

Device selection for the shell

• xpustat --select N picks the N best devices and prints the assignment that restricts a job to them, one line per vendor: CUDA_VISIBLE_DEVICES=1,2, HIP_VISIBLE_DEVICES=0, ... Use it as export $(xpustat --select 2). --by, --min-free and --uuid refine the pick.
• xpustat.visible_devices_env(devices, format="index"|"uuid") is the library form.
• TPU_VISIBLE_CHIPS, TT_VISIBLE_DEVICES and NEURON_RT_VISIBLE_CORES are honoured, matching Ray's accelerator managers. Neuron is handled at core level: ranges such as 0-3 are parsed and a device is visible when any of its cores is.
• Every device carries ray_resource (GPU, NPU, TPU, neuron_cores, ...) and k8s_resource (nvidia.com/gpu, amd.com/gpu, habana.ai/gaudi, ...). Kubernetes names that could not be confirmed are None.

NVIDIA

• Jetson (jetson): Nano, Xavier and Orin modules through tegrastats: shared RAM, GPU load, temperature and the power rail the module exposes. Silent when nvidia-smi exists, so a GPU is never listed twice. Parsers are tested on all three line formats; the backend has not yet been run on a board.
• MIG: on the NVML path a GPU in MIG mode lists its compute instances in mig_instances, with UUID, instance ids, memory and processes. Shown as sub-lines in the CLI and exported as devices to Prometheus. Written from the NVML API; not yet run on a MIG-enabled GPU.

Plugin backends

• A package can add a vendor through the xpustat.backends entry point, or at runtime with xpustat.register_backend(name, cls). The vendor then appears in query_all, to_dict, the CLI, pick_device and the exporter. Contract and example in docs/contributing.md.

Query cache

• query_all(max_age=seconds) reuses a recent result; --cache-ttl does the same for --serve-prometheus, for scrapes that arrive faster than the vendor tools answer.

Tested hardware

Vendor	Device
Apple	Apple M5, 10-core GPU, macOS 26
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0
NVIDIA	7x NVIDIA L4 (cc8.9), nvidia-smi and nvidia-ml-py paths

Upgrade

pip install -U xpustat

xpustat 0.3.0

Feature release: four new backends, a vendor-neutral device API for training scripts, and a Prometheus exporter.

New backends

• Apple Silicon (apple). Read in-process through IOKit, with ioreg as fallback; about 2 ms per query against 25 ms for ioreg. Reports the chip name, GPU core count, utilization and the memory held by the GPU driver. Unified memory means mem_total_mb is the machine's RAM. Temperature, power and per-process usage are root-only on macOS and are not reported.
• AWS Neuron (neuron) through neuron-ls --json-output, Google Cloud TPU (tpu) through tpu-info --format json, and Tenstorrent (tenstorrent) through tt-smi -s. These are written from the tools' documented output and have not yet been run on hardware; see vendor_devices_tests/ to help with that.

Device selection for scripts

• Every device object now has vendor, device_index, mem_free_mb, framework_index and torch_device.
• xpustat.pick_device() returns the least-loaded devices and honours CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ASCEND_RT_VISIBLE_DEVICES and the other vendors' variables, so the index it returns is the one the framework will see.

dev = xpustat.pick_device()[0]
model.to(dev.torch_device)     # "cuda:2", "npu:0", "mps", ...

Prometheus

• xpustat --serve-prometheus 9400 serves /metrics for every vendor from one process; xpustat --prometheus prints the text once; xpustat.to_prometheus(stats) is the library form. No new dependencies.

Fixes

• Index access on vendor collections. stats.amd[0] raised TypeError; it works as the README has always shown.
• Faster vendor commands. The two nvidia-smi and three amd-smi invocations run concurrently: 246 ms to 172 ms on a seven-GPU NVIDIA host, 460 ms to 167 ms on ROCm 10.
• Watch mode no longer swallows exceptions.
• Process lines render as user:command/pid instead of running user and command together.
• pip install xpustat[nvidia] now pulls nvidia-ml-py; the pynvml package is deprecated.

Documentation

A docs/ directory with eight pages (installation, CLI, Python API, device selection, Prometheus, backends, contributing) and a mkdocs.yml.

Tested hardware

Vendor	Device
Apple	Apple M5, 10-core GPU, macOS 26
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0
NVIDIA	7x NVIDIA L4 (cc8.9), nvidia-smi and nvidia-ml-py paths

Upgrade

pip install -U xpustat

xpustat 0.2.0

Bug-fix release: AMD GPUs detected again on ROCm 10, and a watch mode that no longer flickers.

Fixes

• AMD detection on ROCm 10. When amd-smi reported a section as "N/A", or an idle GPU returned its process list as a string, the AMD query failed silently and xpustat printed "no accelerators detected". This hit machines with an iGPU next to the discrete card. Both cases are handled now, and each GPU shows whatever fields it reports.
• Flicker-free watch mode (-i). Each frame is rendered in memory and drawn over the previous one in place, on the terminal's alternate screen, so the screen no longer blanks on every refresh. Your terminal is restored on exit. Piped output is unchanged.

Tested hardware

Vendor	Device
AMD	Radeon AI PRO R9700 (gfx1201), ROCm 10.0 / amd-smi 27.0

Upgrade

pip install -U xpustat

Full changelog: v0.1.0...v0.2.0

xpustat 0.1.0

First release of xpustat — a unified accelerator stat tool in the spirit of gpustat, supporting every major AI accelerator vendor from a single command.

Supported hardware

Vendor	Device	Status
NVIDIA	L4 (cc8.9), TITAN V (cc7.0)	✅ Tested
AMD	Radeon RX 7900 XTX (gfx1100)	✅ Tested
Huawei	Ascend 910B2	✅ Tested
Intel	Gaudi / Gaudi 2 / Gaudi 3	🔲 Implemented
Intel	Arc / Data Center GPU	🔲 Implemented
Hygon	DCU	🔲 Implemented
Cambricon	MLU	🔲 Implemented
Moore Threads	MTT GPU	🔲 Implemented

Install

pip install xpustat
# or
conda install -c conda-forge xpustat
CLI

xpustat          # snapshot of all detected devices
xpustat -a       # add util, temp, power, arch, processes
xpustat -i 1     # watch mode, refresh every second
xpustat --json   # raw JSON output
Python API

import xpustat
stats = xpustat.query_all()
print(stats.amd[0].mem_used_mb)