Skip to content

History / Benchmark Configs Registry

Revisions

  • BR-7: correct PPL attribution — 16.40 is NF4-from-F16 (verified quant_kind=1), not uniform-from-Q8; note per-channel-scale dominates the +54% vs int8

    @mafischer mafischer committed Aug 7, 2026
  • Benchmark-Configs-Registry: add BR-7 — 1.7B int4 (W4A8-inflate) NF4-from-F16 vs uniform-from-Q8 (P=128: 215/6.77 vs 172/2.96; recommended = F16-source NF4)

    @mafischer mafischer committed Aug 7, 2026
  • bench registry: record v1.0.7 no-regression validation (keep original rows) BR-2 (int8, Qwen3-1.7B-Q8_0, llama-bench direct): add 2026-08-06 / ork-driver v1.0.7 row — 226.10 pp512 / 13.25 tg128 (NEW record; 178->226 prefill = the intervening #37/#38/#39/#43 levers, decode 12.8->13.25 still ahead of librkllmrt 11.71). #45 mcworker removal is perf-neutral for int8. Note the rkllm 1.7B side was not re-measured (only board .rkllm is 7B, toolkit-1.2.3 blocked by rknpu 0.9.8 kernel); the 184/11.71 reference kept. Original 2026-06-13 row retained for the progression. BR-6 (fp16, qwen3-1.7b-f16, llama-bench direct): add 2026-08-06 / v1.0.7 reproduction (NOT a record) — 222.17 pp512 / 7.43 tg128, within noise of the 225.5/7.48 fresh-boot record (prefill -1.5%, decode -0.7% on a non-fresh boot). fp16 colsplit default path is byte-identical post-#45 (only the dead mcworker fallback was removed); the 225.5/7.48 record stays. Both rows kept.

    @mafischer mafischer committed Aug 6, 2026
  • Benchmark registry: direct is the default transport again (ORK_DIRECT removed, ORK_USE_ORKD opt-in)

    @mafischer mafischer committed Aug 6, 2026
  • fp16 colsplit CONTIG: drop root-cause + prevention (2026-08-05) NPU-Quirks: the fp16 3-core cross-buffer fetch wild (Kslice=1 boundary), why it's unrecoverable (abort domain-put-before-reset + global-domain-refcount double-put), CONTIG prevention, the is_valid_dma_addr Bbc blind-spot, recoverable-vs-wild taxonomy. Experiment-Log: new 2026-08-05 entry. New Exp-2026-08-05-fp16-Colsplit-CONTIG page. Benchmark-Configs-Registry: BR-6 (matmul A/B colsplit vs mcworker + end-to-end qwen3-1.7b-f16 llama-bench, ORK_DIRECT=1 gotcha, prefill-parity finding).

    @mafischer mafischer committed Aug 6, 2026