v0.9.0

Changes since v0.8.0:

Features

• R-B W4A8-FP8 tile — scale-folded e4m3 LUT loader, k32 f8f6f4 vec_dot (one MMA where int8 issues two), plain process_tile + kernel + launcher; BW24_MMQ_F8F4 seam + f8f4-check gate bin
• directory bulk-extraction mode + bf16 hidden dumps
• p3-agentic-long-v3 — real well-formed payload (hqmtp pe3: CPython selectors.py + real traceback) replaces the malformed-glue v1; 9B sampled audit coherent (58% acc, 158 tok/s @0.7)
• sampled mode + FRSPEC_TRIM — d2t scatter residual, trimmed-idx q gathers
• sampled speculative decoding — rejection-sampling verify (eager path), gates green
• BW24_REPLAY_HDUMP — stream pre-norm trunk hiddens + bg meta
• sampled-spec piece A complete — Engine methods + sample-check gate ALL GREEN
• sampled-spec piece A — Philox4x32-10 + Gumbel-max perturb, softmax-prob gather, residual/categorical sampler kernels (fatbin registered)
• f8f4 tile piece 1 — e4m3 activation quantize (d4-twin, cvt.e4m3x2) + plain-kind f8f6f4 MMA primitive + ABI size helpers
• mixed e4m3xe2m1 f8f6f4 probe BIT-EXACT at full 381 TFLOP/s — Probe 0 closed, R-A GO
• BW24_CHAT=1 chat-template wrapping (run-gen contract) — needed for the p3 re-measurement protocol
• mixed e4m3xe2m1 mxf8f6f4 probe (assembles; rate+correctness pending GPU) + design route ladder from form-matrix

Documentation

• sampled-spec implementation map — code anchors, Gumbel-max via device-argmax reuse, residual-sampling plan, gate ladder
• speculative decoding evaluation protocol survey (greedy degeneration)
• sampled-spec arc — owner decision, scope + new gate class
• piece-2 vec_dot/loader twin analysis — MMA halving, dA deletion, remaining port list
• route decision — R-B primary (8.5% adjacent-scale equality kills R-A's fast path; fold keeps per-16 granularity, simpler tile)
• R-A implementation spec — in-loop container build (VRAM law), cvt.e2m3x2 scale-fold recode, plain f8f6f4 e2m1xe4m3 MMA, gate plan
• FLAGS — KV_K fp8 flip-blocked (flat e2e + 9B ST spec self-consistency FAIL)
• prefill arc design — W4A8-FP8 via mxf8f6f4 block-scale MMA (native e2m1 weights x e4m3 acts), probe-0 plan
• board caveat — p3 spec column degeneration (symmetric both engines), per text audit
• HANDOVER — measurement protocol shift: llama win marked for Qwen models, internal A/B from here

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.8.0

Changes since v0.7.0:

Performance

• BW24_PP_PIPE=2 — 2-CTA/SM occupancy variant of the W4A8 MMQ pipe (WASH, default OFF)

Features

• load safetensors HF checkpoints (directory model paths)
• per-burst spec acceptance telemetry
• accept HF safetensors dirs + NV-27B ST config sweep
• teacher-forced replay acceptance — fixed-corpus MTP head metric (hqmtp)
• CORPUS_DIR knob — save every prompt payload sent during acceptance runs
• compressed-tensors NVFP4 arm + modelopt gates green
• full-precision loader mode + oracle-path spec + acceptance battery
• plain-decode forward wiring + mmap spill tier for the Q4_K repack dir

Fixes

• gate hard-fails only on bg/pos mismatch; slot diff = reported noise floor
• strip diff3 base markers leaked into rig5090.jsonl by merge resolution
• accept HF safetensors dirs — skip GGUF weight-oracle sections instead of crashing
• gate semantics — bg exact, drafts <=1% chunk-FP-order slack
• BF16 embed device-gather + chunked bf16 linear — the spec path fits 24GB

Documentation

• Hy3 phase-2 spill-tier tuning plan (lever list + measurement order)
• FLAGS.md PP_FP8 row — local verification landed (+5.3% pp1855 NV-27B)
• HANDOVER — ST spec lanes merged, disputed OOM claim flagged, board pending
• FLAGS.md default-flip sync (SPEC_M2/FUSED_T/FA_V3 to rollback seams) + evidence discipline in CLAUDE.md
• quant scope law (NVFP4 + full-prec only) + board policy (best-vs-best head-to-head, both containers until the choice)
• readability + consistency pass across public docs
• de-duplicate — numbers live once in the generated tables; prose explains without re-quoting; known-gaps and safetensors sections condensed
• format direction — both GGUF+ST tuned, ST preferred (gguf = extra step, not default)
• dual-shape directive — research platform; MTP-heal protocol queued next (9B bf16 acceptance ceiling vs NVFP4 hit)
• box retired after w4a8v2 lands — local-only from here

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.7.0

Changes since v0.6.0:

Performance

• same-session sustained-load board — ALL plain cells above llama (1.02-1.12x); FA_V3 default; thermal-regime board law
• FA v3 dp4a-K hybrid behind BW24_FA_V3 (default OFF) — full battery green, docs+JSONL
• v3 rev-4b — funnel-shift aligned K words + paired B3 smem reads; gsub killed (negative); micro 35B 46.9/74.5 @6257/12288 vs v2 59.7/97.0

Features

• add Hy3 repack loader glue
• BW24_ST_E4M3 — F8-origin ST projections decode DIRECTLY from raw e4m3 (one weight copy)

Fixes

• resolve qmatvec.cu both-added seam (FUSED_T batched twins + E4M3 section), restore fused3_b4 closing brace

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.5.0

Changes since v0.4.0:

Performance

• BW24_PP_PIPE default ON — cp.async pipeline, 27B pp +4.7% / 9B +5.6%, bit-identical
• cp.async multi-stage pipeline for the NVFP4 W4A8 prefill GEMM (BW24_PP_PIPE, default OFF)

Documentation

• roadmap order — finish Qwen ST path first, then Gemma-4 + Hy3 REAP50 (disk tier is the specialty, majority-resident minority-spilled)

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.4.0

Changes since v0.3.0:

Performance

• 35B spec K=3 — 249/232/192, p1+p2 at margin (1.16x/1.11x); spec board 8/9 above llama
• BW24_SPEC_M2 small-m batched verify (t=2 linear arm + MoE dev-rows), default OFF

Configuration

• BW24_SPEC_M2 default ON — verify m-curve flattened (launch structure: t=2 batched arm + MoE dev-rows), 35B spec optimum shifts to K=3: 249/232/192 vs llama 215/208.5/201.7

Documentation

• Hy3 REAP75 assessed as the next spilled-MoE target (42GB fits VRAM+RAM, no disk tier) — replaces M3's roadmap slot

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.3.0

Changes since v0.2.0:

Performance

• full board 2026-07-09 — 35B spec p1/p2 flip positive (225/212 vs 215/208), spec board 8/9 above llama, plain 27B 1.10x/9B 1.08x/35B 1.02x
• vendor llama Q8_0 int8-MMA MMQ (BW24_PP_Q8MMQ, 35B pp 1.30x)

Configuration

• BW24_PP_Q8MMQ default ON — 35B pp 2456->3069 (+25%), 9B +12%; battery green (35B p1-p3 + 9B p2/p3 argmax MATCH, p4 OOM pre-existing both arms)

Documentation

• scope law — this box only; M3 counts as its local number, other-box gains out of scope

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.2.0

Changes since v0.1.0:

Features

• compile-gate CI + tag-triggered releases with auto-drafted changelogs

Fixes

• libcublas-dev for the cuBLASLt FP8 prefill headers

Configuration

• kill the concluded flag zoo — 18 flags + their dispatch arms/kernels; docs/FLAGS.md full catalog

Documentation

• fact-check fixes — intro board current (1.10/1.07/1.02), V cache is q5_1 not q4_0, PP_ONLY prefill numbers, 158.5 depth residual, CUDA 13.1 requirement truth, FLAGS.md stub
• release policy — tag per board-moving change

Boards + reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench · full experiment log in research/tune-data/

v0.1.0 — all three models above llama.cpp at short context

First tagged release. bw24 is a from-scratch Rust+CUDA LLM inference engine built for one machine — RTX 5090 Laptop (sm_120a, 24 GB) — with llama.cpp on the same rig as the floor to beat. Every optimization is gated on bit-exactness (kernel-check vs CPU reference, prefill/decode argmax match, K=1..8 speculative self-consistency).

State at v0.1.0 (all numbers same-day, both engines, full power)

Plain decode, tg128 (bw24 / llama.cpp):

• d512: 9B 132.7/124.6 (1.07x), 27B 47.7/43.5 (1.10x), 35B MoE 173.4/170.5 (1.02x) — all three models above llama.cpp
• d6257: 9B 124.5/119.6 (1.04x), 27B 44.9/42.0 (1.07x), 35B 158.5/159.9 (0.99x)

Speculative (MTP head, raw-prompt protocol, p1/p2/p3):

• 9B 243/195/162 vs 186/158/155 · 27B 108/91/79.5 vs 86.4/89.9/73.2 · 35B 203/201/185 vs 215/208/202

Highlights in this release:

• FA_V2 tile-batched online-softmax attention decode (default) — fixed the engine-wide depth slope; the 9B deep-context cell flipped from 0.91x to 1.04x
• Fast path is the default — no environment flags needed; flags remain only for runtime parameters, machine config, and rollback seams
• FR-Spec vocabulary trims + frspec_rank builder (per-tokenizer artifact, published on HF), zero-draft rounds, per-content-class draft depth
• FP8-activation prefill GEMM for F8-native checkpoints (BW24_PP_FP8, VRAM-budgeted)
• Safetensors loading (NVIDIA modelopt NVFP4, MiniMax-M3 REAP50 121GB via NVMe expert streaming)
• Reproduction artifacts: https://huggingface.co/Avifenesh/bw24-bench

Known gaps (tracked in research/tune-data/): prefill 0.55-0.74x of llama.cpp (precision-ceiling decomposition done; exactness-safe levers named), 35B speculative 0.92-0.97x (verify m-scaling curve measured), 35B deep-context 0.99x.