Skip to content

v0.1.6 — the warm-ahead fix

Latest
Latest

Choose a tag to compare

View all tags
@swellweb swellweb released this 26 Jul 23:59
v0.1.6
93f1e75

Fix you probably want

Warm-ahead was silently dead whenever speculative decoding was on — which is the default. The decoder branch returned before the shared-prefix cache was consulted, so every POST /v1/warm wrote KV snapshots that nothing ever read.

Measured on a free-tier ARM box, same cache, same request: 90.5s with speculation on vs 16.7s with it off. After the fix the two features compose, and a repeated query against a warm prefix answers in 2.3s with speculation enabled (#6).

If you run Reame with a disk cache, upgrade.

New

  • POST /v1/warm — pre-digest a document before anyone asks about it. TTFT 20.6s → 3.4s on the free tier.
  • bench/ — the reproducible fact-extraction exam behind the headline claim: one page in three input forms, 20 adversarial questions, and a scorer that runs against any OpenAI-compatible server. 
    python3 bench/run_fact_exam.py --server http://127.0.0.1:8080 --model reame

Docs, including the parts that make us look worse

  • Published the measurement behind the headline: same page, same model, ~6× faster time to first token (5.9–6.5× depending on where you measure) and a better score — 19/20 → 20/20 — by rewriting the input as label: value, not by changing the model.
  • Retracted our own claim. The README said a 1B-active MoE matched much larger models at extraction with '100% accuracy'. True on an easy needle test; false on an adversarial one (14/20). Both are now in the benchmarks, with an attention dump showing why: the model sees the right number but cannot bind it to the question.
  • Four negative results added: 4-bit Q4_0 (+37% prefill, −5 facts, rejected), halved experts during prefill (+44% prefill, corrupted KV), MoE expert prefetching (routing is flat by design), and a year of upstream kernel work (0% on this quant).

Fixes

  • build.sh: --no-tests, --avx512 and --no-server passed CMake variable names that no longer existed, so the flags did nothing.
  • config/reame.conf pointed at a 30B GGUF that does not exist — the from-source quickstart could not start.
  • Test count in the README said 220+; the suite has 261.
Assets 5
Loading