all-smi v0.28.1-metrum.3

all-smi v0.28.1-metrum.3

What's new

• Windows NPU discovery and telemetry via DXCore and Windows performance counters, with best-effort AMD XRT identity and a windows.npu doctor check.
• Windows iGPU performance-counter gauge and support for numbered compute and 3D engines.
• Windows users build from source for now; this release does not include a Windows binary.

Hardware validation

A Ryzen AI 9 HX PRO 475 test laptop with an XDNA 2 NPU and Radeon 890M was exercised at idle and under NPU, iGPU and CPU workloads. The report TECHNICAL_REPORTS/436-windows-npu-dxcore-pdh-20261030.en.md records the time-paired counter comparisons and sampling limits.

NVIDIA validation used an RTX PRO 6000 Blackwell Server Edition with driver 580.126.09 and CUDA 13.0: release workspace build and tests, GPM and DCGM doctor checks, and live GPM/DCGM metrics passed. NVIDIA GPU metric schemas matched the previous Metrum release on the same system. AMD Instinct was not rerun because this change does not modify the discrete AMD device path.

Developed by Metrum AI.

all-smi v0.28.1-metrum.2

This release improves AMD telemetry and scripting while retaining the fork's NVIDIA monitoring capabilities.

• Recording (#29): sub-second cadence, recording metadata and units, gap reporting, and CSV and optional Parquet export. Legacy recordings preserve unknown gap state.
• API binding (#30): choose the listen address with --bind or --listen; the existing default remains unchanged.
• ROCm detection (#31): recognize additional ROCm installations and report runtime version information more accurately.
• Non-root local mode (#33): allow AMD monitoring when the user has appropriate render-node access.
• APU GPU telemetry (#26, partial): expose shared GTT memory separately from VRAM, distinguish graphics and socket power, and add supported clock, video-engine and raw throttle-residency readings.
• XDNA NPU (#27, partial): discover supported devices and report identity, firmware and available client-memory information.
• SoC and bandwidth (#28, partial): avoid counting APU socket power as discrete GPU chassis power, explain unavailable privileged counters, and expose available APU system DRAM read/write source metrics.
• AMD plugin robustness (#32): handle nullable and non-finite process samples safely and improve diagnostics. The original root-mode failure has not been reproduced and remains open.

Validated on a Strix Halo (Radeon 8060S) test system, NVIDIA RTX PRO 6000 Blackwell Server Edition (driver 580.126.09, CUDA compatibility 13.0), and an AMD Instinct MI325X SR-IOV guest (ROCm 7.14.0). NVIDIA release tests and live GPM/DCGM scrapes passed; the previous NVIDIA binary could not be compared on the test guest because its glibc was too old. The XDNA checks did not include an active NPU workload. Validation results and limitations are summarized below; known issues are tracked in the linked issues.

• #48: on the MI325X SR-IOV guest, GPU utilization was reported as 0% under load and an unavailable edge temperature appeared as 0 °C. Both were also observed with the prior release.
• #49: a cold one-shot CPU snapshot can underreport utilization; warmed samples matched the reference. This predates the release.

Remaining work includes per-process APU utilization and allocation attribution, DCLK and broader validation (#26); active XDNA workloads and unsupported NPU metrics (#27); additional SoC energy, performance counters and bandwidth accuracy (#28); reproduction of the root-mode plugin error (#32); service/configuration bind-address support (#40); and XDNA process-scan efficiency and deduplication (#45).

Linux glibc builds include the AMD and DCGM runtime plugins. See the platform archive and matching SHA256 file for your system.

all-smi v0.28.1-metrum.1 — Metrum AI

all-smi v0.28.1-metrum.1

Maintained, integrated and validated by Metrum AI, based on the upstream v0.28.1 release tag. Upstream authorship and licenses are preserved.

Highlights

• Upstream NVIDIA RTX Spark N1X/SoC and DLA telemetry, Windows ARM64 support, REBEL100 support, corrected Gaudi detection, frozen Intel Xe counter handling, and russh/luwen updates.
• Retains Metrum AI NVIDIA GPM, NVML extras, deep telemetry and DCGM runtime enrichment. Main, AMD and DCGM crate versions are consistently 0.28.1.
• Includes previously unreleased Metrum fixes from main: crate-private GPM sampling helpers under the shared process-wide NVML lock, reduced XID watcher lock contention/fairness, DCGM reconnect after sampling failures, graceful missing-sudo handling, manual Shadeform validation, and pinned ARC CI.
• Preserves process-lifetime NVML ownership and serialization. DCGM uses an external hostengine only; no embedded hostengine fallback.
• Corrects ARC x86_64 → Linux ARM64 release cross-builds: host-architecture protoc, isolated target libdrm sysroot, explicit pkg-config dependency, contained linker symlinks, target pkg-config settings and AArch64 ELF checks. ARM64 CC/AR/Cargo linker configuration is retained.

API note: direct low-level GPM sampling helpers are now crate-private to prevent bypassing NVML serialization. Consumers should use the supported AllSmi APIs.

Validation

Both physical GPUs used rustc 1.99.1 (58cf8f9c3 2026-10-13) and DCGM 4.7.0. CUDA below is the nvidia-smi driver-reported compatibility version, not a claim about the installed toolkit/PyTorch wheel version:

GPU	Driver	CUDA	Workspace / release tests	Default-parallel regression	Doctor	Fresh live source series
RTX PRO 6000 Blackwell Server Edition	580.126.09	13.0	Passed	10 × 26/26	4/4	20 GPM default / 13 DCGM fallback
NVIDIA H100 PCIe	570.195.03	12.8	Passed	10 × 26/26	4/4	20 GPM default / 13 DCGM fallback

• cargo build --release --workspace built main, liball_smi_amd.so and liball_smi_dcgm.so on both GPUs.
• cargo test --release passed including executed doctests on both GPUs and in the container. Library: 1,960 passed / 1 ignored; binary: 2,157 passed / 2 ignored; all integration targets; doctests: 23 passed / 13 ignored. Library/binary counts include overlapping module coverage, not unique tests.
• Ten additional cargo test --release --test library_api_test runs per GPU used default parallelism, with no serialization/exclusions. NVIDIA doctor passed --only nvidia.gpm,nvidia.dcgm; AMD plugin tests passed 12/12, AMD doctor had 3 passes / 5 expected skips.
• Fresh /metrics scrapes showed source="gpm" by default and source="dcgm" with ALL_SMI_NVIDIA_DISABLE_GPM=1. nvidia-smi stayed responsive before and after validation.
• GPU-tested telemetry implementation: merge 8d5c9be. Subsequent changes are release workflow fixes, docs/evidence, nonfunctional copyright headers and the document-title test guard; no telemetry behavior changed. Final local workspace build, full release tests, formatting and Python script tests passed. One repeated run exposed an unchanged timing-sensitive SSE lag-event test; subsequent complete normal-parallel runs passed. A document-title guard was adapted only to allow the required exact leading Metrum copyright notice before the original title/history assertions.

CI and build evidence

• PR #25, final reviewed head 53b30a7d567a2e2459d45125eddbff1d46c5da66.
• Completed SUCCESS: Final-head CI 40226340436, including ARC Linux Test Suite and Build Check, macOS, Windows ARM64, packaging, systemd and launchd smoke tests.
• Completed SUCCESS: Final-head Documentation 40226340416.
• Completed SUCCESS: Corrected final-head Linux release preflight 40226335412, including actual ARM64 main/AMD/DCGM builds and ELF checks.
• Earlier all-platform preflight 40224846945 passed macOS ARM64 and Linux x86_64 glibc/musl; its ARM64 missing-pkg-config failure was diagnosed and corrected before publication. Superseded/canceled runs are not counted as green release gates.
• Detailed report: TECHNICAL_REPORTS/deep-telemetry-progress.md.
• Sanitized dual-GPU raw evidence: artifact snapshot 40448ee6, tags all-smi / v0.28.1-metrum.1 / gpu-validation.

Coverage limitations

Server-GPU testing covers Blackwell and Hopper discrete NVIDIA paths, not physical N1X Windows ARM64/SoC/DLA hardware. Platform CI and pure helper tests do not replace N1X hardware validation. No destructive XID storm was induced. Capture-based GPM parity tests used historical fixtures, not a newly generated workload parity experiment; fresh hardware evidence is diagnostics, concurrency regressions and live source-labelled scrapes.

Rental accounting and verified cleanup

GPU	Rate	Create-to-verified-delete elapsed	Conservative pro-rata estimate
RTX PRO 6000 Blackwell Server Edition	$1.95/hour	978.5 seconds	$0.5300
H100 PCIe	$2.50/hour	992.1 seconds	$0.6890

Combined conservative estimate $1.2190, including provisioning and deletion polling. Individual deleted-instance API cost_estimate scalars were 0.4564289223 and 0.5378233534 (sum 0.9942522757); these are provider estimates, not final invoices or billing-rounding guarantees.

Both run-owned GPU instances were deleted and separately confirmed state=deleted by individual-ID lookups. The run-owned registered SSH key was deleted (HTTP 200), then its individual info endpoint returned HTTP 404. Local key material was removed; the independent finite-deadline watcher was stopped only after verified cleanup. No account SSH-key listing, managed key or unrelated resources were used.

Release provenance

PR #25 was merged with a merge commit: e80f490fcf4102b2f7f8a85115eba73c2df40dba. Its tree matches the validated final PR head. Annotated tag v0.28.1-metrum.1 is tagged by Chetan Gadgil chetan@metrum.ai with these Metrum AI release notes.

Download assets

Four platform archives, each with its .sha256 file (8 assets total). Windows ARM64 support is covered by source/platform CI, not a Windows download in this fork's asset matrix:

• all-smi-linux-x86_64.tar.gz — glibc; includes main, AMD and DCGM runtime plugins and all-smi.1.
• all-smi-linux-x86_64-musl.tar.gz.
• all-smi-linux-aarch64.tar.gz.
• all-smi-macos-aarch64.zip — unsigned.

Prefer the x86_64 glibc archive on NVIDIA GPU servers. Announcement graphics are handled separately by Metrum AI and are not GitHub release assets.

v0.27.1-metrum.1

Sync of lablup/all-smi through v0.27.1, plus a fix for NVML crashes when the library is used from several threads. The fork keeps NVIDIA GPM, NVML extras, and the DCGM plugin. Crate version is 0.27.1. Prepared by Metrum AI.

Upstream v0.27.1:

• Display-only --hide-storage for local and view modes (lablup#455)
• Fixed false remote host disconnects and remote chassis energy and cost totals; bounded remote HTTP buffering; timed-out remote fetches are cancelled (lablup#454)

Fork fixes:

• NVML is now one process-wide handle that is never shut down, and NVML/GPM calls are serialized process-wide. Before this, creating and dropping several AllSmi instances concurrently segfaulted inside libnvidia-ml.so, and on one RTX PRO 6000 it wedged the NVIDIA kernel driver
• Behaviour change: the DCGM plugin no longer starts an embedded hostengine, because its in-process NVML threads raced with GPM sampling. Run the nvidia-dcgm service for DCGM metrics. Without it, NVML/GPM collection continues, and the plugin retries the connection with backoff (1 s up to 60 s)
• DCGM plugin crate version moves to 0.27.1 with the main binary
• Install docs still point deep-telemetry binaries at Metrum Releases. cargo install all-smi remains the upstream crates.io crate

Validated on Shadeform H100 PCIe (driver 580.126.09 and 570.195.03, DCGM 4.7.0): full release suite passes; library_api_test passes 10/10 default-parallel runs, with a new 12-thread create/drop regression; doctor GPM/DCGM 4 pass; default scrape source="gpm", GPM disabled plus the DCGM plugin source="dcgm"; AMD plugin tests 12 pass; no new kernel NVRM errors.

v0.27.0-metrum.1

Sync of lablup/all-smi through v0.27.0, plus the commits after that tag. The fork keeps NVIDIA GPM, NVML extras, and the DCGM plugin. Crate version is 0.27.0.

Upstream

• AWS Neuron (Trainium and Inferentia) monitoring
• Lower per-tick collection cost on macOS and Linux
• Apple Silicon and Rebellions power fixes, and stable Prometheus info labels
• AMD GPU power reported in watts (no longer 1000 times too low on the hwmon path)
• README split into task guides, Launchpad builds the AMD companion, and debug-pid-mapping stays out of a default cargo install

Fork

• DCGM plugin crate version moves to 0.27.0 with the main binary
• Install docs still point deep-telemetry binaries at Metrum Releases. cargo install all-smi remains the upstream crates.io crate

Verification

Shadeform H100 PCIe (driver 570.148.08, DCGM 4.7.0): doctor GPM/DCGM 4 pass, default scrape source="gpm", GPM disabled plus the DCGM plugin source="dcgm", AMD plugin tests 12 pass.

v0.26.3-metrum.4

Fork release on top of v0.26.3-metrum.3 after syncing lablup/main into the fork:

• Upstream sync: AWS Neuron support (lablup#417); Rebellions load/power fixes (lablup#416, lablup#423); Tenstorrent vendor metrics re-key (lablup#439)
• Upstream sync: volatile detail keys / dedicated gauges (lablup#435, lablup#441, lablup#442); Linux process CPU sampler (lablup#443); macOS proc_pidinfo sampling and collection-cost work (lablup#426, lablup#432)
• Upstream sync: drop dead generic all_smi_npu_* metrics (lablup#440); Prometheus parse hardening (lablup#438); macOS CI clippy gate (lablup#444)
• Merge fix: initialize fork deep-telemetry GpuInfo fields on new upstream Neuron/NPU/test constructors

Crate version remains 0.26.3.

Download the asset for your platform from this release. Linux x86_64 example:

curl -fsSL -o all-smi-linux-x86_64.tar.gz \
  https://github.com/chetan-metrum-ai/all-smi/releases/download/v0.26.3-metrum.4/all-smi-linux-x86_64.tar.gz
tar -xzf all-smi-linux-x86_64.tar.gz
./all-smi --version

• Binaries are attached by the Metrum Release workflow after this release is published.
• DCGM plugin requires libdcgm.so.4 on the host at runtime.

v0.26.3-metrum.3

Fork release on top of v0.26.3-metrum.2:

• GPM doctor: two-sample activity probe; WARN when NVML gpm_support() is a false positive (issue #5 / PR #6)
• Snapshot: first GPM collector poll self-primes so snapshot --samples 1 can populate source=gpm
• Support bundle / CI: enabled_features() records the no-op dcgm cargo feature

Crate version remains 0.26.3.

Download the asset for your platform from this release. Linux x86_64 example:

curl -fsSL -o all-smi-linux-x86_64.tar.gz \
  https://github.com/chetan-metrum-ai/all-smi/releases/download/v0.26.3-metrum.3/all-smi-linux-x86_64.tar.gz
tar -xzf all-smi-linux-x86_64.tar.gz
./all-smi --version

• Binaries are attached by the Metrum Release workflow after this release is published.
• DCGM plugin requires libdcgm.so.4 on the host at runtime.

v0.26.3-metrum.2

Metrum deep-telemetry (P0–P4)

Fork release completing the deep-telemetry track:

• P0–P2: GPM collector, NVML extras, doctor checks (also in v0.26.3-metrum.1)
• P3: Derived hollow utilization + alert rules (hollow / no-tensor / memory-bound / remap / XID / throttle) and ALL_SMI_MOCK_HOLLOW
• P4: liball_smi_dcgm.so DCGM plugin (dlopen); fills only None GPM fields; GPM wins when present

Install

Download the asset for your platform from this release. Linux x86_64 example:

curl -fsSL -o all-smi-linux-x86_64.tar.gz \
  https://github.com/chetan-metrum-ai/all-smi/releases/download/v0.26.3-metrum.2/all-smi-linux-x86_64.tar.gz
tar -xzf all-smi-linux-x86_64.tar.gz
./all-smi --version
# Optional: place liball_smi_dcgm.so beside the binary (glibc Linux builds)

Notes

• Binaries are attached by the Metrum Release workflow after this release is published.
• DCGM plugin requires libdcgm.so.4 on the host at runtime.

all-smi v0.26.3-metrum.1 (Metrum deep-telemetry)

Metrum deep-telemetry preview (P0–P2)

Fork release of all-smi with NVIDIA deep telemetry:

• P0: Shadeform harness, doctor GPM/DCGM checks
• P1: Real NVML GPM two-sample collector (Hopper+) with full field set and source= labels
• P2: Throttle reasons, per-process SM util, PCIe fallback, hardware energy counter, remapped rows, NVLink errors, utilization sample summaries, XID watcher

Install

Download the asset for your platform from this release (built by metrum-release.yml). Linux x86_64 example:

curl -fsSL -o all-smi-linux-x86_64.tar.gz \
  https://github.com/chetan-metrum-ai/all-smi/releases/download/v0.26.3-metrum.1/all-smi-linux-x86_64.tar.gz
tar -xzf all-smi-linux-x86_64.tar.gz
./all-smi --version

Notes

• P3 (derived hollow util / alerts) and P4 (DCGM plugin for Ampere/Ada) are not in this tag yet.
• Binaries are attached by the Metrum Release workflow after this release is published.