Skip to content

v0.3.2 — 453.70 tok/s: 24% past llama.cpp + trust-hardened eval pipeline

Choose a tag to compare

@ai-hpc ai-hpc released this 30 Jun 03:14

The lead over llama.cpp doubles to ~24%, and the evaluation that proves it is substantially hardened.

🏁 Performance — frontier 410.85 → 453.70 tok/s (+10.4%), now 24% past llama.cpp

Same RTX 5090, same Q4_K_M GGUF, warm & interleaved vs llama-bench:

decode length sparkinfer llama.cpp vs v0.3.1
128 tok 453.70 365.85 +24.0% was +12.1%
256 tok 443.53 364.90 +21.6% was +10.0%
512 tok 425.23 361.64 +17.6% was +6.7%

The lead grew at every length — still a pure kernel win (same model/precision/greedy decode, no speculative decoding).

  • #89 — Q/K/V projections on concurrent CUDA streams, byte-identical (@James-CUDA)
  • #86single-pass MoE top-k (parallel rank-count) + fused RoPE/KV-append (@fansilas)

🔒 Trust-hardened evaluation pipeline (#102)

Closes the remaining gaming/poisoning vectors from the eval trust-model audit:

  • Held-out prompts (H1) — a fresh unpredictable per-seed prompt each eval, so a PR can't overfit the gate; seed logged for reproduction.
  • Reference quarantine (C2) — model weights (sha256-pinned) + llama.cpp (commit-pinned) verified/rebuilt each run; a tampered baseline can't skew a verdict.
  • Clock record (M1) — graphics clock pinned where allowed and always recorded, so the absolute tok/s is reproducible.
  • Immutable frontier ledger (H2) — every advance appends a GitHub-timestamped (date, PR, author, commit, Δ%, prev→new) line; auditable line-by-line.
  • Provenance (clock, seed, reference pins) in every verdict + immutable log.

✅ KL metric fixed (honest, strict gate kept)

A held-out KL of 0.27 turned out to be a measurement artifact (shallow top-k dump floored llama's tail). The corrected matched-depth metric shows the true divergence is ~0.02 (top-1 0.97). Proven honest — a broken build reads KL 18; a 12-prompt sweep holds 0.007–0.022. So the strict KL ≤ 0.20 gate is kept, not loosened.

Verified: RTX 5090 453.70 tok/s, top-1 0.97 vs a fully-built CUDA llama.cpp.

Full notes: CHANGELOG.md.