Skip to content

capy-flash v0.8.2 — opt-in YaRN long context and a portable benchmark

Choose a tag to compare

@ValerioDolci ValerioDolci released this 07 Nov 22:26
· 21 commits to main since this release

Two community contributions on top of v0.8.1, both opt-in: long context with YaRN, and a one-command benchmark. Nothing changes at the default — the model answers token for token as v0.8.1 — so updating is safe for everyone, but there is no reason to rush it.

New

  • Long context with YaRN (--rope-yarn-factor, issue #28, PR #29). The implementation is @splickz's; @Hazelhof ported it and measured it on Flash-Next TP2. --rope-yarn-factor F (1 to 4) raises the accepted --max-context to native × F — up to 1,048,576 tokens — with the YaRN NTK-by-parts frequency table. The native window stays 262,144. On two 16 GB cards 1M needs --kv-dtype nvfp4 (int8 KV runs out of expert slots first). It is opt-in and experimental: 4× YaRN costs perplexity, and the quality at 1M is not something we benchmarked. Factor 1, the default, installs nothing and leaves the output byte-identical — we checked, 8 prompts out of 8 greedy-identical to v0.8.1 on the 27B.

  • A portable benchmark (bench/capy_flash/ladder.py, issue #30, requested by @Zeppe79). One command runs a fixed set of context sizes against a running server and reports the median cold-prefill and decode rates from the server's own timings, using deterministic prompts so there is no corpus or tokenizer to download. It records the machine too — GPUs, driver, PCIe link width, WSL2, CPU, RAM — and, from the startup log, the P2P state and the expanded launch line, because those are what decide whether two people's numbers mean the same thing. check_result.py validates a row before it is shared.

Fixed

  • The decode attention kernels read the page table directly instead of a fixed 64-entry staging array, so a key range past about 500K no longer overruns it. This is part of the YaRN work and is active at every context length; it is what the greedy-identity check above covers.

Checks

  • Build and ctest pass, 213/213 (the new ninfer_yarn_test gates the frequency table: factor 1 must reproduce the shipped constant exactly).
  • Same output as v0.8.1: 27B on two GPUs, MTP, greedy, 8 prompts, every token identical.
  • Live: Flash-Next on two GPUs with --rope-yarn-factor 4 --kv-dtype nvfp4 --max-context 1048576 starts and answers correctly.
  • The benchmark ran on our 2× RTX 5070 Ti: prompt-token counts land on the targets, the environment and the server's P2P line are recorded, and check_result.py accepts the row.
  • Package: the three launchers from the release tarball each answer correctly.