Repository navigation
Feature release: xpustat follows a training run from inside the loop, the cluster page streams from every host over one ssh session each, and the server now carries alerts, history and a JSON Schema for its payload.
In a training loop
xpustat.monitor()samples the devices on a thread while your code runs, as a context manager or an iterator, and reports peaks and means per device (peak("util_pct"),mean("mem_used_mb"),summary()).xpustat.callbacks:HFTrainerCallback,LightningCallbackandKerasCallbacklog device metrics to TensorBoard, W&B, MLflow,printor any callable at every logging step. The frameworks are imported only when a callback is constructed; psutil stays the only dependency.
Cluster page
- One streaming ssh session per host: the remote runs
xpustat --json --meta -i Nand the server reads one JSON line per refresh, so the cost no longer grows with a command per host per refresh. Connections are multiplexed withControlMaster; a remote older than 0.7.0 falls back to a command per refresh, and--no-streamforces that path. --history 15m|2hkeeps the sparkline history on the server behind/history, so every tab shows the same history from the moment it opens.--history-file FILE.sqlitewrites a replayable log;xpustat --replay FILE.sqliteplays it back.xpustat --select N --ssh node1 node2 ...and/freepick the freest devices across hosts (xpustat.pick_remote)./metricscovers every host with ahostlabel plusxpustat_host_upandxpustat_host_stale(xpustat.to_prometheus_hosts).- Hosts show their real hostname:
xpustat --json --metawraps the devices in hostname, time and version, and--json -i Nstreams one line per refresh. - Page: fixed column widths so host tables align, a user filter, CSV export, an
older xpustatbadge on hosts that cannot stream.
Alerts
xpustat --checkexits 1 when a device is over--max-temp, under--min-free, reports ECC errors, is throttled or unhealthy, or when a host is down;--no-alert-throttleignores throttling. Works with--hostsand--ssh.- The page shows an alerts bar and sends desktop notifications;
--alert-urlposts new alerts to a webhook,--alert-cooldownkeeps a repeated condition to one message.
Schema and deployment
xpustat --schemaprints a JSON Schema for the payload andxpustat.validate_payloadchecks one. A test holds every vendor's keys to it, which showed that Gaudi, Intel, Huawei, Hygon, Cambricon and Moore Threads emit noprocesseskey; the schema says so.deploy/: Dockerfile, systemd unit, Kubernetes DaemonSet and a Grafana dashboard, documented on a new Deployment page. CI publishes the docs to GitHub Pages.
Fixes
- The ssh control socket path under the macOS temp directory was longer than a Unix socket allows; the sockets now live under
/tmp. - A non-interactive ssh shell has no
~/.local/binon its PATH, so an xpustat installed withuv toolorpip install --userwas invisible to--ssh; the default remote commands add it.
Tested hardware
| Platform | Device |
|---|---|
| Apple | Apple M5, 10-core GPU, macOS 26 |
| NVIDIA | 7x NVIDIA L4 (cc8.9), dual Xeon Silver 4314, Linux |
| AMD | Radeon AI PRO R9700 (gfx1201), ROCm 10.0, Ryzen 9 9950X3D |
| CPU only | Intel N200 (Linux) |
All four were shown on one cluster page over streaming ssh, with one unreachable host alongside; 144 tests pass on macOS and 141 on Linux with 3 skipped. The Windows host was unreachable during this release, so native Windows and WSL were not run this time; CI covers Windows for the tests that need no hardware.
Upgrade
pip install -U xpustat