NVMAI 3.8

NVMAI 3.8 — performance investigation release

A measured investigation into decode throughput. One real fix, the instrumentation
that made it findable, and a documented ceiling so the next attempt starts from
evidence rather than assumption.

Faster

Decode is ~7% faster. The shared dense MLP was encoded after the router
readback, so the GPU sat idle through the entire round trip before that work was
submitted — it depends only on the post-attention norm and could always have been
queued earlier.

	before	after
attn_tail_router → shared_expert gap	7.88 ms/token	0.02
total GPU idle	19.9 ms/token	16.7
occupancy	57.9%	61.1%

Output is byte-identical to golden reference on 4-bit — this reorders submission,
not arithmetic.

New instrumentation

• NVMAI_KERNEL_STATS reports true GPU occupancy by merging overlapping
  command-buffer intervals. The previous metric summed them and could exceed 100%,
  which made it useless for deciding whether the GPU was the bottleneck.
• Per-transition gap attribution — every interval between one buffer ending
  and the next starting, charged to the pair of roles it separates. This is what
  located the fix above.
• TURBO_FIELDFARE_PHASES adds active experts per layer.
• NVMAI_ROUTE_TRACE=<path> dumps real per-layer routing (position layer e0..e7).
• NVMAI_MTP_EXPERT_SLOTS makes the MTP sidecar's expert cache tunable. Default
  unchanged at 8 — an interleaved A/B found no gain from raising it.

New C target

NVMAIKernelsC holds hot loops where Swift's vector types do not lower well. Its
int4 affine GEMV is 3.4× faster than the Swift version it replaced, validated
against an independent reference. Not on the decode path — see the ceiling.

The measured ceiling

Batch-1 decode moves ~1.8 GB per token and this hardware sustains ~64 GB/s, so
~36 tok/s is the maximum for 4-bit on an M3. That is measured from four
directions, all converging: eliminating GPU idle, running entirely on the ANE,
running entirely on the CPU, and the bandwidth arithmetic itself.

Adding compute units makes it worse, because every unit draws on one memory
controller — CPU load raises GPU-busy 45%, ANE load 89%. Nine approaches
measured out negative, including command-buffer merging, spin-waiting, weight
relayout, lossless compression (the payload is at 93% of its entropy limit) and
the fused greedy head.

Known limitation

On a 24 GB machine the 6-bit and 8-bit models exceed RAM and page continuously:
6.7 and 1.6 tok/s against 4-bit's 18.8. Prefer 4-bit unless the machine has
comfortable headroom over the model size.

Fixed

• tools/release.sh fails when release notes quote the wrong archive checksum,
  and now fills the checksum in rather than requiring notes to predict it.

689 tests / 124 suites. Full method, including every negative result:
docs/cpu-coexecution-plan.md

Binaries

macOS 26+, Apple Silicon only. Not code-signed or notarized — verify the checksum,
then clear quarantine:

xattr -dr com.apple.quarantine /path/to/nvmai-3.8-macos-arm64

nvmai-3.8-macos-arm64.tar.gz sha256:

938394658480877e3fbb1a013f17b94b0f1d075b6ad4f631de1c75ebf30c3b07