Show and tell: an interactive visualization of the layer-streaming memory/time tradeoff

[iamdgarcia]

I kept explaining airllm's tradeoff to colleagues ("the VRAM you need is one layer's, the price is time") and the numbers argued better than I did, so I built a small interactive visualization of it, anchored on the numbers this repo publishes:

https://iamdgarcia-one-layer-at-a-time.static.hf.space
(code: https://github.com/iamdgarcia/one-layer-at-a-time)

What it does: a canvas shows the layer stack on disk and the GPU slot; pick a GPU (4 to 24 GB), a model (8B / 70B / 405B / 671B MoE with only the routed experts streaming), a disk (NVMe / SATA / HDD) and optional 4bit weights, then generate a 16-token sentence at the modeled speed while meters count tokens, disk traffic and layer trips.

The declared model: time/token = (non-resident layers) x (layer GB / disk GB/s) + (all layers) x (layer GB / 1000 GB/s GPU ingest). Anchors: the README VRAM table (70B ~4 GB, 27B 3.33 GB, 180B MoE 5.95 GB, 671B ~12 GB) and the real reports in #364 (28.6 s/token on a 3B, Windows) and #374 (30 min/query on 20B); the full constant table is in the page's "how this works" dialog.

The number that lands hardest: at 4 GB with the 70B fp16, 1 of 80 layers is resident, ~55 s/token, and one 16-token sentence reads ~2.2 TB off the disk. The all-resident bound (a 142 GB card) is 0.14 s/token: about 400x faster, same model. 4bit weights cut the load ~4x (~13 s/token), consistent with the 3x end-to-end the README reports.

Two things it makes visible that seemed useful here:

• the case for Feature: Adaptive GPU loading — fill VRAM to maximize speed (full-load bypass v1 PR ready) #325: every GB of VRAM filled is one fewer disk round-trip per token;
• disk choice matters as much as GPU choice (HDD is ~21x slower on travel than NVMe).

It is a simulation with measured anchors, not a running model; your SSD, your GPU and the framework's prefetching shift the constant, not the shape. If there are profiling_mode logs or a benchmark table that would make the anchors sharper, I would happily re-fit the constants.