Streaming-video VLMs in MLX are LM-prefill-bound — what does M5's Neural Accelerator do to the frontier?

[katlun-lgtm]

I've been building a bounded-memory streaming-video path for VLMs in mlx-vlm (ties into #492): ingest frames one at a time into a persistent KV cache, keep an attention-sink + the most-recent frames, evict the oldest vision spans, and re-anchor survivors to contiguous RoPE positions — instead of load_video()'s pre-sample-all-frames + O(T) KV growth. It runs on 1-D-RoPE models (SmolVLM / Idefics3) with no model changes, and the re-anchor is numerically exact vs a fresh prefill (max|Δ| = 0).

Building it, I measured where the per-frame time actually goes on an M3 Max — and I think it points at a nice M5 question, so sharing here.

Where the time goes (per frame, M3 Max)

model	vision-encode ms	LM-prefill ms	KV/frame	recency-recall (full ctx)
FastVLM-0.5B	74.6	27.5	3.0 MiB	(refuses)
FastVLM-1.5B	~73	73.3	7.0 MiB	~0.31
FastVLM-7B	~73	309	14.0 MiB	engages
Idefics3-8B	72.8	244	21.9 MiB	0.79

Two things fall out:

1. Vision encode is ~constant (~75 ms/frame) and encoder-independent — optimizing the vision tower doesn't move the needle.
2. The useful models are the capable (large) ones, and their per-frame cost is dominated by LM prefill (matmul) — Idefics3-8B at 244 ms/frame is the one that actually answers "what were the last N scenes" (0.79 recall); the tiny fast models have a low capability ceiling no window size fixes.

So the streaming frontier is gated by per-frame LM-prefill GEMM on a large model.

The M5 question

M5's GPU adds a Neural Accelerator (matrix unit) per core — aimed exactly at that matmul inner loop, which streaming prefill is nothing but, run once per frame. So the falsifiable hypothesis is that M5 shifts the streaming Pareto frontier: at a fixed per-frame latency budget you could run a more capable model (higher accuracy) or a higher frame rate than M3 Max allows.

The thing I'd genuinely like to know from folks closer to the Metal backend:

• Do MLX's current qmv/qmm/GEMM kernels already dispatch to M5's Neural Accelerators, or is there headroom the kernels don't yet capture? If there's a gap, that seems like a concrete kernel opportunity, and I'd be happy to help chase it.

What I'd measure (harness already exists)

Same code, cross-generation (M3 Max → M4 → M5), reusing the streaming benchmark I've already written: per-frame encode/prefill split, prefill tok/s on the qmv/GEMM microbench, answer TTFT, and the money curve — accuracy at equal latency budget (the largest model whose per-frame prefill fits budget B, M3 Max vs M5). KV memory stays bounded/flat regardless of chip (algorithmic, a good null control).

I can run the M3 Max numbers now. If anyone has M5 access or wants specific configs in the comparison, I'm glad to fold them in.

Code + harness (fork): the streaming path and the full benchmark harness + raw JSON live on feat/streaming-video-kv — examples/phase2b_scaled.py, fastvlm_size_sweep.py, phase2c_equal_memory.py, streaming_video_demo.py (+ the committed *_results.json).

Curious whether this matches what others are seeing on M5, and whether the "does MLX already use the Neural Accelerators" question has a known answer. 🙏