test(metal): pin the S5 reference-tier contract, and record the MLX provider A/B

`test_metal_backend.cpp` still asserted the PRE-S5 contract: that
`GetOp(kMoeCombine, kMETAL)` throws for an op Metal has no native kernel for.
Since af0b21b that is wrong. Metal is unified-memory, so `ReferenceTierEligible`
holds and the portable CPU tier installs lazily on the miss, and `GetOp` returns
a working (correct but slow) kernel instead of throwing. The suite was red on any
Apple host: 19361/19362 assertions, one failure, unrelated to MLX (it reproduces
with `VT_OP_PROVIDER_DISABLE=mlx`).

Assert the CURRENT contract instead of deleting the check, and keep the intent
the old assertion had. `OpRegistered` stays false, because it means a NATIVE
kernel exists and the tier is a fallback, so a later row still cannot quietly
claim more than it implements. Added on top: the selection is the reference tier
BY NAME and `GetReferenceTierHits()` moved, so a Metal op silently running on the
CPU path can never pass for a native kernel (Risk 7). `> 0` rather than a strict
per-call increment is deliberate: Resolve() caches the winner in the slot and
ResetOpProviderStats does not clear that cache, so a second resolution of the
same (op, device) never re-counts.

Verified on the M4 (Apple M4, 16 GiB, macOS 26.5.2): 17/17 test cases,
19367/19367 assertions, and the `[vt reference-tier] op=15 device=2` line
confirms the portable path is the one that served it.

Public checkpoint, per the same-change obligation: docs/BENCHMARKS.md gains the
first measurement of what the optional MLX GEMM provider is WORTH, which is the
question study §5.4 left open. One binary, arms toggled by
`VT_OP_PROVIDER_DISABLE=mlx`, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order
alternated. MLX wins every axis: 1.5x to 2.2x aggregate throughput, 2x to 3x
faster TTFT, no memory cost, and NMSE-identical (in fact bit-identical) output on
all six parity shapes. It does NOT close BACKEND-GATE-METAL-MLXLM: still ~4.8x to
~5.5x behind the mlx_lm floor, which is what M3c and a simdgroup GEMM are for.
INDICATIVE, not binding: the three actions.runner LaunchAgents were verified
idle, booted out and restored, and the whole A/B ran inside one lockf /tmp/gpu,
but com.localai.worker and the aerial wallpaper stayed up (no passwordless sudo).

FOLLOWING_AGENTS_PROTOCOL
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

Author: mudler

README.md

@@ -309,7 +309,7 @@ For C++ consumers, the higher-level surface lives under [`include/vllm/`](includ
 | CUDA | Consumer Blackwell, sm_120a | Build-supported (compiles, emits real sm_120a code, all fast paths resolve) but not runtime-proven here (no such card) |
 | CUDA | Hopper, sm_90a | Build-supported, portable-kernels-only (accelerated paths disabled, no Hopper kernel bodies); not runtime-proven here |
 | CUDA | Ampere/Ada (sm_80/86/87/89), datacenter Blackwell (sm_100a/103a), sm_110 | Build-supported, portable-kernels-only (accelerated paths disabled, no fast-path bodies for these families); sm_80/sm_100a/sm_110 compiled `-Werror` clean on dgx as per-major representatives, siblings share the bodies; not runtime-proven here (no such boards). sm_70/sm_75 are not build-supported (no bf16 tensor cores) |
-| Metal | Apple Silicon | Two models run end to end and pass correctness (OPT-125m, Qwen3-0.6B); 18 of 75 ops native, the rest fall back to CPU on unified memory. No general model support yet; MLX is the named speed floor |
+| Metal | Apple Silicon | Two models run end to end and pass correctness (OPT-125m, Qwen3-0.6B); 18 of 75 ops native, the rest fall back to CPU on unified memory. No general model support yet; MLX is the named speed floor. The optional MLX GEMM provider (`-DVLLM_CPP_MLX=ON`) measures 1.5x to 2.2x throughput over native MSL, indicative ([BENCHMARKS](docs/BENCHMARKS.md)) |
 | Vulkan | Portable GPU | Skeleton: 8 ops plus the fusion catalogue run and cross-check against CPU and CUDA. No model runs yet; off unless `-DVLLM_CPP_VULKAN=ON` |
 | Intel XPU | Intel GPUs | Spiked, hardware-blocked |
 | ROCm / ANE | AMD GPUs / Apple Neural Engine | Post-parity roadmap |

docs/BENCHMARKS.md

@@ -2230,6 +2230,79 @@ in that evidence document before publishing these values as binding for the
 development branch.
 This 4B diagnostic does not establish 27B/35B support.
 
+## MLX GEMM provider A/B on Apple M4 - Qwen3-1.7B (2026-07-27, M5) - INDICATIVE / partially quieted
+
+**The first measurement of what the optional MLX provider is WORTH, and the
+answer to the question §5.4 of `metal-mlx-reuse-study.md` left open: does
+delegating the dense GEMM to MLX earn its 105 MB dependency.** This is an
+ours-vs-ours A/B, NOT a competitor comparison (for that see the M3b table below).
+
+**Method - one binary, no rebuild between arms.** Built ONCE with
+`-DVLLM_CPP_MLX=ON`; the arms are `VT_OP_PROVIDER_DISABLE=mlx` (native MSL) versus
+unset (MLX). A rebuild-free A/B is what the provider seam exists for, and it
+removes any build-difference confound. Commit `9e1c9025`; `git log
+9e1c9025..main -- src/vt/metal/ src/vt/op_provider.cpp include/vt/op_provider.h`
+is EMPTY, so these numbers apply unchanged to `main`. Model
+`mlx-community/Qwen3-1.7B-bf16` rev `9cd6692855d3e06772228e9a962b2606359b2d24`,
+p=512 g=128, b in {1,2,4,8,16}, 2 reps, arm order ALTERNATED per rep so thermal
+drift cannot favour one arm. Peak memory via `/usr/bin/time -l`.
+
+| B | MLX agg tok/s | native agg tok/s | speedup | MLX TTFT | native TTFT | MLX decode tok/s/stream | native decode tok/s/stream |
+|--:|--:|--:|--:|--:|--:|--:|--:|
+| 1 | 5.79 | 3.08 | 1.88x | 3.32 s | 7.68 s | 6.76 | 3.75 |
+| 2 | 10.19 | 6.08 | 1.68x | 5.09 s | 11.68 s | 6.34 | 4.17 |
+| 4 | 15.75 | 10.24 | 1.54x | 9.63 s | 18.77 s | 5.58 | 4.09 |
+| 8 | 25.70 | 13.69 | 1.88x | 13.95 s | 34.38 s | 4.94 | 3.16 |
+| 16 | 38.65 | 17.69 | 2.19x | 18.33 s | 54.48 s | 3.69 | 2.09 |
+
+**MLX wins every axis: 1.5x to 2.2x aggregate throughput, 2x to 3x faster TTFT,
+at no memory cost** (peak RSS 6.65 to 7.50 GB in BOTH arms; one MLX b=16 rep read
+9.20 GB). Exactly ONE op is delegated: `VT_OP_PROVIDER_STATS=1` shows `op=65
+selected=mlx priority=100` and everything else `vt-native`. `kPagedAttention`
+stays ours, as study §5.3 requires.
+
+**And it is free of accuracy cost.** `tests/vt/test_metal_backend.cpp` three-way
+parity (bar NMSE <= 5e-4), same box, MLX build:
+
+| Shape | msl-vs-cpu | mlx-vs-cpu | mlx-vs-msl |
+|---|--:|--:|--:|
+| decode bf16 1x2048x2048 | 2.80e-06 | 2.80e-06 | 0 |
+| decode bf16, B transposed | 2.76e-06 | 2.76e-06 | 0 |
+| prefill bf16 32x2048x6144 | 2.75e-06 | 2.75e-06 | 0 |
+| prefill bf16, B transposed | 2.74e-06 | 2.74e-06 | 0 |
+| square f32 128x512x512 | 3.81e-14 | 3.81e-14 | 0 |
+| square f32, B transposed | 3.73e-14 | 3.73e-14 | 0 |
+
+The exact zeros are REAL, not a stale buffer: the harness allocates fresh input
+and output buffers per call, `mlx_declines == 0` rules out a silent forward down
+to MSL, and the 1.5x to 2.2x throughput delta proves a different kernel ran. Same
+accumulation order, different scheduling. Bit-exactness across providers is still
+NOT claimed as a contract (study §5.3); it is what these six shapes measured.
+
+**This does NOT close `BACKEND-GATE-METAL-MLXLM`.** Against the mlx_lm floor
+below (27.57 agg tok/s at b=1, 213.39 at b=16, itself INDICATIVE and not re-run
+today) the MLX-provider arm is still ~4.8x to ~5.5x behind. The provider
+accelerates the dense GEMM only; the rest of the deficit is what `M3c` (batched
+encoders) and a simdgroup-matrix GEMM are for.
+
+**Status: INDICATIVE, NOT BINDING - partially quieted.** Better isolation than the
+M3b run but still short of the standing bar. DONE: the three `actions.runner`
+LaunchAgents (dante-desktop, platform, wingman) were verified job-idle, booted
+out for the sweep and restored after; the WHOLE A/B ran inside one
+`/usr/bin/lockf -k /tmp/gpu` so nothing could interleave between arms. NOT done:
+`com.localai.worker` and the aerial wallpaper stayed up (still no passwordless
+sudo). Rep spread reached 9.4% (MLX b=16) against mlx_lm's 0.12 to 0.63%, on 2
+reps, so read the multipliers as +/-10%; the gap far exceeds the noise.
+**Repro (M4, `~/vllmcpp-mlx-ab`):** configure `-DVLLM_CPP_METAL=ON
+-DVLLM_CPP_MLX=ON -DMLX_ROOT=$HOME/mlx-venv/lib/python3.9/site-packages/mlx`,
+then per arm `/usr/bin/lockf -k /tmp/gpu env [VT_OP_PROVIDER_DISABLE=mlx]
+/usr/bin/time -l ./build/examples/vllm-bench --model <snapshot> --num-prompts <B>
+--input-len 512 --output-len 128 --concurrency <B> --seed 0`. To make it binding,
+add `sudo launchctl bootout system/com.localai.worker` plus the wallpaper
+disable, and restore both after.
+
+---
+
 ## Ours-vs-MLX on Apple M4 - Qwen3-1.7B (2026-07-23, M3b) - INDICATIVE / `BLOCKED-ON-SUDO`
 
 **The first ours-vs-MLX comparison. NOT binding.** Qwen3-dense now runs on Metal

tests/vt/test_metal_backend.cpp

@@ -198,14 +198,31 @@ TEST_CASE("Metal registers the W0 op set and NOT the unimplemented rest") {
   }
   // Still stubbed, and asserted so a later row cannot quietly claim more than it
   // implements: the quant tier, the GDN/MoE families, and every sampler op except
-  // greedy argmax. A partial backend is a supported state: vt::GetOp throws on
-  // lookup.
+  // greedy argmax. `OpRegistered` means a NATIVE kernel exists, so it stays false
+  // regardless of what the portable reference tier installs underneath.
   for (vt::OpId op : {vt::OpId::kRandomSample, vt::OpId::kComputeProbs,
                       vt::OpId::kMoeCombine, vt::OpId::kGdnStateGather}) {
     CHECK_FALSE(vt::OpRegistered(op, DeviceType::kMETAL));
   }
-  CHECK_THROWS_AS(vt::GetOp(vt::OpId::kMoeCombine, DeviceType::kMETAL),
-                  std::runtime_error);
+  // A partial backend is still a supported state, but since S5 that no longer
+  // means "GetOp throws". Metal is unified-memory, so ReferenceTierEligible holds
+  // and the portable CPU tier installs LAZILY on the miss: GetOp returns a
+  // working (correct but slow) kernel instead. This asserts the CURRENT contract
+  // plus the two facts that keep it honest — the selection is the reference tier
+  // BY NAME, and the observability counter moves — so a Metal op silently running
+  // on the CPU path can never masquerade as a native kernel (Risk 7).
+  REQUIRE(vt::ReferenceTierEligible(DeviceType::kMETAL));
+  void* moe_combine = nullptr;
+  CHECK_NOTHROW(moe_combine = vt::GetOp(vt::OpId::kMoeCombine, DeviceType::kMETAL));
+  CHECK(moe_combine != nullptr);
+  const auto moe_stats = vt::GetOpProviderStats(vt::OpId::kMoeCombine, DeviceType::kMETAL);
+  REQUIRE(moe_stats.last_selected != nullptr);
+  CHECK(std::string(moe_stats.last_selected) == std::string(vt::kReferenceProviderName));
+  // Deliberately `> 0` and not a strict per-call increment: Resolve() caches the
+  // winner in the slot, and ResetOpProviderStats does not clear that cache, so a
+  // second resolution of the same (op, device) never re-counts. `> 0` is the
+  // order-independent form of "the portable path announced itself".
+  CHECK(vt::GetReferenceTierHits() > 0);
 }
 
 // ===========================================================================