Repository navigation
capy-flash v0.8.2 — opt-in YaRN long context and a portable benchmark
Two community contributions on top of v0.8.1, both opt-in: long context with YaRN, and a one-command benchmark. Nothing changes at the default — the model answers token for token as v0.8.1 — so updating is safe for everyone, but there is no reason to rush it.
New
-
Long context with YaRN (
--rope-yarn-factor, issue #28, PR #29). The implementation is @splickz's; @Hazelhof ported it and measured it on Flash-Next TP2.--rope-yarn-factor F(1 to 4) raises the accepted--max-contextto native × F — up to 1,048,576 tokens — with the YaRN NTK-by-parts frequency table. The native window stays 262,144. On two 16 GB cards 1M needs--kv-dtype nvfp4(int8 KV runs out of expert slots first). It is opt-in and experimental: 4× YaRN costs perplexity, and the quality at 1M is not something we benchmarked. Factor 1, the default, installs nothing and leaves the output byte-identical — we checked, 8 prompts out of 8 greedy-identical to v0.8.1 on the 27B. -
A portable benchmark (
bench/capy_flash/ladder.py, issue #30, requested by @Zeppe79). One command runs a fixed set of context sizes against a running server and reports the median cold-prefill and decode rates from the server's owntimings, using deterministic prompts so there is no corpus or tokenizer to download. It records the machine too — GPUs, driver, PCIe link width, WSL2, CPU, RAM — and, from the startup log, the P2P state and the expanded launch line, because those are what decide whether two people's numbers mean the same thing.check_result.pyvalidates a row before it is shared.
Fixed
- The decode attention kernels read the page table directly instead of a fixed 64-entry staging array, so a key range past about 500K no longer overruns it. This is part of the YaRN work and is active at every context length; it is what the greedy-identity check above covers.
Checks
- Build and ctest pass, 213/213 (the new
ninfer_yarn_testgates the frequency table: factor 1 must reproduce the shipped constant exactly). - Same output as v0.8.1: 27B on two GPUs, MTP, greedy, 8 prompts, every token identical.
- Live: Flash-Next on two GPUs with
--rope-yarn-factor 4 --kv-dtype nvfp4 --max-context 1048576starts and answers correctly. - The benchmark ran on our 2× RTX 5070 Ti: prompt-token counts land on the targets, the environment and the server's P2P line are recorded, and
check_result.pyaccepts the row. - Package: the three launchers from the release tarball each answer correctly.