Skip to content

xpustat 0.7.0

Latest

Choose a tag to compare

@Bekhouche Bekhouche released this 03 Nov 20:30
· 2 commits to main since this release

Feature release: xpustat follows a training run from inside the loop, the cluster page streams from every host over one ssh session each, and the server now carries alerts, history and a JSON Schema for its payload.

In a training loop

  • xpustat.monitor() samples the devices on a thread while your code runs, as a context manager or an iterator, and reports peaks and means per device (peak("util_pct"), mean("mem_used_mb"), summary()).
  • xpustat.callbacks: HFTrainerCallback, LightningCallback and KerasCallback log device metrics to TensorBoard, W&B, MLflow, print or any callable at every logging step. The frameworks are imported only when a callback is constructed; psutil stays the only dependency.

Cluster page

  • One streaming ssh session per host: the remote runs xpustat --json --meta -i N and the server reads one JSON line per refresh, so the cost no longer grows with a command per host per refresh. Connections are multiplexed with ControlMaster; a remote older than 0.7.0 falls back to a command per refresh, and --no-stream forces that path.
  • --history 15m|2h keeps the sparkline history on the server behind /history, so every tab shows the same history from the moment it opens. --history-file FILE.sqlite writes a replayable log; xpustat --replay FILE.sqlite plays it back.
  • xpustat --select N --ssh node1 node2 ... and /free pick the freest devices across hosts (xpustat.pick_remote). /metrics covers every host with a host label plus xpustat_host_up and xpustat_host_stale (xpustat.to_prometheus_hosts).
  • Hosts show their real hostname: xpustat --json --meta wraps the devices in hostname, time and version, and --json -i N streams one line per refresh.
  • Page: fixed column widths so host tables align, a user filter, CSV export, an older xpustat badge on hosts that cannot stream.

Alerts

  • xpustat --check exits 1 when a device is over --max-temp, under --min-free, reports ECC errors, is throttled or unhealthy, or when a host is down; --no-alert-throttle ignores throttling. Works with --hosts and --ssh.
  • The page shows an alerts bar and sends desktop notifications; --alert-url posts new alerts to a webhook, --alert-cooldown keeps a repeated condition to one message.

Schema and deployment

  • xpustat --schema prints a JSON Schema for the payload and xpustat.validate_payload checks one. A test holds every vendor's keys to it, which showed that Gaudi, Intel, Huawei, Hygon, Cambricon and Moore Threads emit no processes key; the schema says so.
  • deploy/: Dockerfile, systemd unit, Kubernetes DaemonSet and a Grafana dashboard, documented on a new Deployment page. CI publishes the docs to GitHub Pages.

Fixes

  • The ssh control socket path under the macOS temp directory was longer than a Unix socket allows; the sockets now live under /tmp.
  • A non-interactive ssh shell has no ~/.local/bin on its PATH, so an xpustat installed with uv tool or pip install --user was invisible to --ssh; the default remote commands add it.

Tested hardware

Platform Device
Apple Apple M5, 10-core GPU, macOS 26
NVIDIA 7x NVIDIA L4 (cc8.9), dual Xeon Silver 4314, Linux
AMD Radeon AI PRO R9700 (gfx1201), ROCm 10.0, Ryzen 9 9950X3D
CPU only Intel N200 (Linux)

All four were shown on one cluster page over streaming ssh, with one unreachable host alongside; 144 tests pass on macOS and 141 on Linux with 3 skipped. The Windows host was unreachable during this release, so native Windows and WSL were not run this time; CI covers Windows for the tests that need no hardware.

Upgrade

pip install -U xpustat