From-scratch LLM inference engine in Rust + CUDA, built for one machine: an RTX 5090 Laptop (Blackwell sm_120a, 24 GB, 175 W with dynamic boost). No frameworks, no ggml — every kernel written and tuned against measured hardware limits, with llama.cpp as the benchmark to beat on the same rig.

The Performance section and card above show same-session measurements against llama.cpp's best config. It also loads HF safetensors checkpoints directly (no GGUF conversion) and runs a 121 GB MoE on this 24 GB card.

• Real, running inference on sm_120a (consumer Blackwell), with every shipped optimization backed by its measured win/loss record.
• Gated on bit-exactness — argmax match and speculative self-consistency, verified on every kernel change.
• Runs Qwen3.5/3.6 dense and MoE checkpoints on 24 GB — from GGUF or straight from HF safetensors, no conversion — including models far larger than VRAM+RAM.

• NVIDIA Blackwell consumer GPU (sm_120a). Primary target: RTX 5090 Laptop.
• CUDA toolkit 13.1 (the build default — crates/bw24-engine/build.rs uses /usr/local/cuda-13.1/bin/nvcc; override with BW24_NVCC=/path/to/nvcc).
• Rust (edition 2024), cudarc 0.19 with dynamic loading.
• A model — GGUF or an HF safetensors directory (pass either path). Tested: Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE (common quants), nvidia/Qwen3.6-27B-NVFP4, MiniMax-M3 REAP50, Hy3-REAP50.

# build
cargo build --release

# verify all kernels against the CPU reference
./target/release/kernel-check

# generate text (the fast path is the default — no flags needed)
BW24_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches in one paragraph."

# speculative decoding with the embedded MTP draft head (Qwen3.6)
BW24_SPEC_K=3 ./target/release/run-spec /path/to/qwen36-27b.gguf

# OpenAI-compatible server
./target/release/bw24-server

Every tuned kernel path is the default; environment flags exist only for runtime parameters (prompt, draft depth, trims), machine-specific configuration, and rollback seams (BW24_FAST=0 drops to the f32 oracle path). The catalog lives in docs/FLAGS.md.

run-gen prints a prefill/decode correctness gate (prefill argmax must match decode argmax) before timing anything — if that line says MISMATCH, the numbers after it don't count.

• NVFP4 (W4) decode path — block-scaled FP4 matvec with split-plane repack, warp-level dp4a, and an int8 W4A8 tensor-core GEMM for prefill. Auto-dispatches per matrix shape.
• MTP speculative decoding — draft with the model's embedded multi-token-prediction head, verify K+1 tokens in one batched target forward. The whole draft chain runs inside a captured CUDA graph; exactness is enforced by a K=1..8 self-consistency gate (all K must emit identical tokens).
• MoE on 24 GB — expert-major CSR batching, decode-once dequant kernels, int8 tensor-core expert GEMM, and an SLRU expert-residency cache with VRAM → host → disk spill.
• FlashAttention-style kernels — fused prefill and decode attention with quantized KV (q8_0 K / q5_1 V default; FP8 and q4_0 arms exist behind flags — q4_0 V measured quality-taxed and stays off), register-resident dequant, split-K for long context.
• CUDA-graph decode — the full per-token decode is one graph replay; per-step host round-trip is 4 bytes.
• Hybrid architectures — full-attention + gated-delta-net (SSM) layer mixes, as in Qwen3.6.
• Safetensors loader — HF checkpoints load directly (no GGUF conversion): modelopt NVFP4 repacks byte-exact into the GGUF block layout, FP8-E4M3 and large-BF16 tensors re-encode to Q8_0/NVFP4 at load, V-head permutations apply on packed bytes, MoE experts stream through a disk-tier repack cache for models far bigger than VRAM+RAM.
• Sigmoid-router MoE (MiniMax/DeepSeek-style) — e_score_correction_bias selection, swigluoai activation, gate-optional attention; with the measured law that cross-kernel-family FP-order differences are architectural on discontinuous top-k routing (exactness binds within a config).

Every kernel change must pass, in order:

1. kernel-check — every quant kernel vs a CPU reference.
2. run-gen argmax gate — prefill and decode paths must agree on the next token.
3. run-spec self-consistency — speculative output at K=1..8 must be token-identical to plain decode.

Floating-point summation order is part of the contract: two mathematically equal kernels that reduce in different orders can flip an argmax at tight logit margins. Several "faster" kernels were rejected for exactly this (research/tune-data/).

Measured 2026-07-09 on the target rig (RTX 5090 Laptop, N≥2 medians, both engines measured in the SAME session and thermal regime, no flags (tuned paths are defaults)) against llama.cpp built on the same machine, same exact prompts, both engines re-baselined the same day. Boards move with the tuning campaign — research/tune-data/rig5090.jsonl is the running record; the README is refreshed with every board-moving merge.

Plain decode first (no speculation, tg128 at 512-token context — the honest floor comparison):

Depth is part of the contract: at 6.3k-token context the leads hold (1.04-1.07x) except the 35B at 0.99x. Every attention/split change validates across the depth axis, not just one point.

Speculative decoding (MTP head, both engines at their measured best config) as the bonus layer on top:

The three columns are short-code / medium-code / long-agentic prompts, raw-continuation protocol, llama.cpp at serve-best config. Optimal config is content-class dependent. Caveat on the long-agentic column: a text audit found the greedy continuations there partially degenerate into repetition on both engines symmetrically — ratios stand, absolutes overstate real agentic throughput (rig5090.jsonl tag p3-degeneration-audit; prompt replacement pending). The speculative edge comes from three mechanisms — FR-Spec vocabulary trims, whole-round confidence gating, and per-content-class draft depth — detailed in HANDOVER.md.

Reproducing: every artifact is public — trimmed draft-head GGUFs, exact prompts, and full configs at huggingface.co/Avifenesh/bw24-bench. llama.cpp build/serve flags are in docs/COMPETITOR-SETUP.md; the harness is research/e2e/run-e2e.sh.

Known gaps (tracked and diagnosed in research/tune-data/): prefill trails llama.cpp at 0.55-0.76x (decomposed — the residual is int8-vs-native-FP4 tensor-core ceiling under the exactness contract), the 35B loses two cells narrowly, and vLLM's batched MTP wins on big-VRAM boxes.

Beyond llama.cpp: the safetensors path runs checkpoints llama.cpp cannot — NVIDIA's official Qwen3.6-27B-NVFP4 (2.3x the tuned local vLLM reference) and giant spilled MoEs: MiniMax-M3 REAP50 (121 GB) and Hy3-REAP50 (82 GB) both generate correct text on this 24 GB / 60 GB machine through VRAM→RAM→NVMe expert tiers. Details in HANDOVER.md.

• Built for sm_120a only; tuning choices assume this exact memory/compute ratio.
• Model coverage: Qwen3.5/3.6 dense and MoE, MiniMax-M3, Hy3 — not a general GGUF runner.
• Single GPU, single stream. No tensor parallelism, no continuous batching.
• APIs and flags change without notice; moving research codebase.

• ARCHITECTURE.md — tech stack + sm_120a feasibility ledger (what the silicon can and cannot do, measured).
• HANDOVER.md — the living state-of-work doc (current standings, laws, open lanes); internal but readable.
• docs/decisions/ — design decision records: internal weight format, quant/GEMM policy, safetensors import, hybrid-architecture plan.
• docs/COMPETITOR-SETUP.md — how each competitor engine is built and tuned to its peak on this box (the "beat them at their best" contract).
• research/tune-data/current-board.json — the numbers behind the Performance section and docs/perf-card.svg, both regenerated by tools/update-perf-board.py — edit the JSON, never the generated regions directly.
• research/sm120-empirical-capabilities.md — microbenched silicon peaks for this GPU.
• research/benchmarks.md — the A/B measurement protocol.
• research/tune-data/ — every tuning experiment as JSONL: config → measured result, wins and losses both. ~215 records and counting; treat it as a labeled corpus of what sm_120a actually rewards.

Issues and PRs welcome — see CONTRIBUTING.md.

MIT — see LICENSE.