Skip to content

Longer prefixes separate prefill loss from decode savings

Pre-release
Pre-release

Choose a tag to compare

@displague displague released this 15 Sep 21:50
· 2 commits to main since this release

v0.41.0 - Longer prefixes separate prefill loss from decode savings

Fifth delivery, following the owner's continuation request. The 49.901-second
CPU-first OPT-2.7B FP16 screen uses 512-token prefixes and 16-token continuations.
All checked logits are exact, including warmups; 48 scored tokens per condition
match. Both controls fit 4800 MiB, peak sampled GPU 4,763,783,168 B (about 4543 MiB).

Prefill packets use 68.9268% of dense-stream outgoing bytes and 146.3295% of its
forward-call time: BOTH prefill gates fail. Decode uses 3.3287% of bytes and 21.3284%
of call time, passing both decode gates. Whole packet episodes save 92.5714% H2D
and 62.8169% wall (4.187751 s versus 11.262525 s), but this does not waive the prefill
failures. Stop uniform long-prefix expansion; preserve the phase distinction.

Protocol,
results and inspirations,
compact receipts,
source freeze.

Known archived WikiText concatenations, not a new holdout. v0.40 used different
texts, so this is not a paired context-only causal comparison. Same-precision
streamed reference, no full model GPU preload, no FP32-quality/Q4/small-card OOM
claim. Setup, warmups, host duplication, 527-position KV and records are charged.
Cold shared-worker-to-first DENSE logit 17.201395 s is not packet-first cold TTFT.

657 CPU tests, exact historical replays, source review and repeat raw audit pass.
Raw context-v041-raw.zip: 64,918,727 B, 142 members, SHA256
190a02821b42d59a53a706d0045f667caae841124d178095849c8e9ae53df270.
Archive restored/hash checked; final clean-tip full-suite receipt attaches separately.
No model weights redistributed.

Complete #56, not
milestone 10.
The new, unmeasured hypothesis is phase-switched grain: dense prefill transfers,
exact scalar-decode packets, same banks/KV. That requires a NEW short paired
protocol, not optimistic recombination of these timings. No long matrix/native
pivot; optimized low-bit competition remains open.