fix(npu): probe AIE columns via temp file, not -o /dev/stdout#893
Merged
Conversation
The NPU-occupancy card showed "column probe unavailable" because the xrt-smi probe always failed. The live xrt-smi build rejects `-o /dev/stdout` with "output file already exists" (exit 1), and with `--force` it interleaves its human-readable console report onto stdout alongside the JSON, so json.loads chokes. Either way read_aie_columns returned None and the route degraded to the single-tenant "all 8 columns" fallback. Run the probe through `sh -c` instead: write JSON to a private per-exec temp file (`$$`), cat it back (JSON only), then remove it. A failed probe propagates a non-zero exit so the existing returncode check still degrades cleanly. Verified end-to-end against hal0-slot-npu on CT 105. Add a regression test pinning the `sh -c` + --force + temp-file shape so nobody reverts to `-o /dev/stdout`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
thinmintdev
enabled auto-merge (squash)
June 19, 2026 05:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The NPU occupancy dashboard card always showed
column probe unavailableinstead ofN/8 columns claimed. Root cause: the AIE-column probe (xrt-smi examine -r aie-partitions) always failed against the live FLM image:-o /dev/stdout) →ERROR: output file '/dev/stdout' already exists→ exit 1 →read_aie_columnsreturnsNone.--force, xrt-smi writes JSON to the-otarget and prints its human-readable console report to stdout — with-o /dev/stdoutthe two interleave, sojson.loadschokes on the leading banner →None.Either way
columns_availablestayedFalse, and the route silently degraded to the single-tenant "all 8 columns" fallback. The gauge looked plausible (8/8 is right for one slot), so the honest multi-tenant path never actually ran.Fix
Run the probe through
sh -cinside the container: write JSON to a private per-exec temp file ($$),catit back (JSON only), then remove it. A failed probe propagates a non-zero exit so the existing returncode check degrades cleanly.Verified end-to-end against
hal0-slot-npuon CT 105 — returns clean JSON,partitions: [(0, 8)], exit 0.Tests
test_read_uses_tempfile_shell_not_dev_stdoutpins thesh -c+--force+ temp-file shape and asserts/dev/stdoutnever reappears.🤖 Generated with Claude Code