Show and tell: auditable KV/VRAM math for SWA, hybrid-linear and MLA models (npx fitllm)

[click6067-ship-it]

Disclosure: I wrote the tool below. I am posting here because the final question needs measurements from llama.cpp users, and because the KV arithmetic is the part worth checking.

Many quick “will it fit?” estimates start with the textbook KV formula:

KV ≈ 2 × n_layer × n_kv_head × head_dim × n_ctx × bytes

That assumes every layer keeps a full-context cache with one uniform head shape. It works for full-attention dense models, but not for several architectures people run in llama.cpp now:

Model and context	What the textbook formula misses	Naive KV	Modelled KV	Ratio
Gemma 4 31B @ 131K, KV 8-bit	50/60 layers are sliding-window (1,024 tokens); 10 global layers use 4 KV heads × 512 rather than 16 × 256	60 GiB	5.39 GiB	11.1×
Qwen 3.8 27B @ 262K, KV F16	KV exists on 16/64 layers; 48 layers use linear attention with a fixed recurrent state	64 GiB	16 GiB (+0.14 GiB state)	4×
GLM-4.7-Flash @ 131K, KV BF16	MLA caches one shared 512+64 latent rather than per-head K/V	117.5 GiB	6.61 GiB	17.8×
Llama-3 dense full-attention	Nothing	Same	Same	1×

An 11× error can flip a capacity verdict and make a runnable configuration look impossible.

I put the memory calculation in a single MIT-licensed file with zero runtime dependencies. Built-in architecture fields are pinned to official model configs, the public Hugging Face ID path reads config/evidence directly, and 30 byte-exact conformance vectors anchor the math. Unsupported architectures fail closed: the CLI refuses to answer rather than borrowing a nearby shape or guessing an unknown quantization width.

npx fitllm "Gemma 4 12b" --detect            # exit 0 fits/tight, 1 won't fit, 2 invalid
npx fitllm "gpt-oss-120b" --mac 64 --why     # prints assumptions and limitations

Source and arithmetic: https://github.com/click6067-ship-it/fitllm-engine

The llama.cpp-specific question: for Gemma 4 e2b/e4b, FitLLM currently subtracts the per_layer_token_embd (PLE) tensor from discrete-GPU VRAM residency, based on default llama.cpp GGUF behavior keeping it in system RAM rather than GPU VRAM. It still counts the full model on Apple unified memory. For e2b, counting all 5.1B parameters against VRAM instead would overstate resident weights by roughly 1.9× and can flip small-card verdicts. The premise is documented and conditional, but the existing tracker still lacks a direct Gemma 4 e2b/e4b GGUF measurement: click6067-ship-it/fitllm-engine#7

If you have run one of those GGUFs, a nvidia-smi peak plus llama.cpp build/commit, quant, context, and launch flags would settle the premise either way. I will publish a correction with credit if the measurement disproves it. Two other open gaps where raw measurements would help are Qwen3.8-27B on RTX 4090/5090 (click6067-ship-it/fitllm-engine#111) and system-total peaks on 16–32GB Apple Silicon (click6067-ship-it/fitllm-engine#8).

This predicts memory capacity, not tokens/second. SSD/NVMe streaming and expert paging are separate deployment modes; they are not treated as ordinary RAM-resident fits.

Corrections to the math are welcome—that is the useful outcome of posting it here.