Repository navigation
Exp 2026 08 22 W4A4 Reference Model
Outcome: native W4A4 (per-channel int4 + Hadamard) now builds, persists, reloads and produces coherent output end-to-end on a real model. It is still 0.59x int8 prefill, so it is not yet a win — but it went from "hangs and kills the board" to a measurable reference in one session.
| model |
qwen3.5-0.8b-bf16.gguf (unquantized BF16 source) |
| pack |
qwen35-08b-w4a4.orkpack, 161 MB, 102 weights, ORKPACK_DT_I4_NATIVE
|
| build |
ORK_QUANT=4 ORK_MIXED_W4A4=1 ORK_ORKPACK_PATH=<pack> (one-time; then it self-selects) |
| quantizer | per-channel int4 + block-Hadamard (rotation is unconditional since 2026-08-22) |
Reloads cleanly: [ORK PERSIST] this run: loaded 102 from disk, packed 0.
| config | prefill | decode |
|---|---|---|
| int8 (W8A8), derived pack | 180.5 / 179.5 t/s | 2.47 |
| native W4A4 + Hadamard | 105.8 / 107.5 t/s | 2.48 / 2.49 |
W4A4 is 0.59x int8 on prefill. Decode is identical because this model's decode is CPU-bound (GDN), so W4A4 buys nothing there.
Canonical int8 cross-check the same day, Qwen3-1.7B-Q8_0 warm: 211.2 / 206.4 t/s prefill, 11.9 decode — i.e. the int8 path is healthy and unregressed by the session's driver changes.
ork_ppl now exists (it was prescribed by AGENTS for months but had never been committed).
qwen3.5-0.8B, 6 x 512-token windows, 3066 scored tokens, ubatch pinned at 512:
| config | PPL |
|---|---|
| int8 (W8A8) | 19.4345 |
| native W4A4 + Hadamard | 49.1659 |
2.53x worse than int8. So Hadamard has NOT made per-channel W4A4 coherent — but it is also nowhere near the ~104 on record for per-channel W4A4, so that figure needed re-deriving and now has been. Combined with prefill (below), native W4A4 currently loses on both axes.
A cautionary note on the earlier eyeball check: 40 generated tokens looked entirely coherent (see
below) at PPL 49. Greedy continuation of a familiar README is simply not a sensitive quality probe —
that is the gap ork_ppl closes, and the reason AGENTS insists on it.
- int8:
…/blob/main/LICENSE) [ > Production-grade KV-cache and weight quantization for llama.cpp, with cross-backend kernel support for Apple Silicon, NVIDIA CUDA, AMD
Both plausibly continue the README — at PPL 49.17. Read that as a warning about smoke tests, not as evidence of quality.
Worth stating because it is an obvious thing to want. They are alternatives, not a stack:
| quantizer | compute | rotation | |
|---|---|---|---|
| NF4 / LUT ("route B") | codebook, inflated to int8 before the MAC | W8A8 | n/a |
| native W4A4 | raw per-channel int4 | the 4-bit MAC | Hadamard, required |
src/npu/i4/pack.c:36 sets w->dtype = DT_I8 for the NF4 pack, and line 89 is explicit —
"int4-stored: fill = inflate nibbles -> int8 + tile" (orki_nf4_inflate_chan_f32). NF4's 4-bit
value is an index, and the NPU's 4-bit MAC multiplies values, not codebook indices, so NF4 can
only ever feed W8A8. There is then no 4-bit MAC for a rotation to protect. Conversely native W4A4
needs raw symmetric int4, which is incoherent unrotated — hence Hadamard. Pack formats differ too:
ORKPACK_DT_I4 (4) vs ORKPACK_DT_I4_NATIVE (5).
GPTQ works. qwen3.5-0.8B, ork_ppl, 6 x 512-token windows, 3066 scored tokens:
| config | PPL | vs RTN |
|---|---|---|
| int8 (W8A8) | 19.43 | — |
| W4A4 + Hadamard, RTN | 49.17 | — |
| W4A4 + Hadamard, GPTQ | 38.19 | -22.3% |
Still 1.97x int8, so native W4A4 is not competitive yet — but the mechanism is proven end-to-end on a real model, which it never was before.
rank(H) <= samples, and ork-driver quantizes a weight on FIRST USE — one batch. The convert pass
runs at M=4, so single-shot calibration gives a rank-4 Hessian against K up to 3584: damping owns
the null space, H^-1 ~ I/lambda there, the propagation row is zero, and GPTQ silently degenerates
to RTN. A PPL comparison in that state would have shown ~nothing and read as "GPTQ doesn't help".
So: phase 1 registers each weight and accumulates H = (A*R)^T (A*R) over every forward (RTN codes
only so the passes have something to compute with; persist skipped); phase 2 re-reads, re-rotates,
runs GPTQ against the accumulated H, re-packs and persists that.
It was briefly ORK_GPTQ_NBATCH. That is a knob whose correct value is computable — and one that
fails silently — so the backend now reports the largest registered K and the caller feeds windows
until rows >= K. On this model that is 3584 rows, where the hand-picked value gave 2048, leaving
the 24 ffn_down weights at 57% rank. The knob was hiding a bug.
A 45-minute pack is not a tool anyone iterates with. Two fixes, both "the access pattern, not the algorithm" — the same defect class found three times in this quantizer:
| phase | was | now | fix |
|---|---|---|---|
| 1 (Hessian) | ~16 min | ~4 min | rank-1-per-row streamed all of H per row (98 MiB x 512 = 50 GiB/weight/batch). Now rank-B blocked, block held transposed for unit-stride, lower triangle only |
| 2 (quantize) | 29.1 min | 13.0 min | parallel across WEIGHTS in chunks of 4 (inner loops are short at small K so cores idled); pack+persist stay serial since the NPU pack is not thread-safe |
PPL is 38.1931 either way — identical to four decimals, so the blocking is numerically equivalent, not merely close.
Earlier, ork_i4_gptq itself got 10.7x (92 s -> 8.6 s at K=2048) from the same class of fix:
a stride-K column walk in the inverse product, and a chol_upper that did two column walks and
turned out to be deletable entirely (the upper factor is the transpose of the lower one).
The prediction was activation-dominated. It is parity: mean A/W = 0.97 over 14 weights (range 0.75-1.19).
| weight | W-only | A-only | both | A/W |
|---|---|---|---|---|
| blk.0.attn_qkv K=1024 N=6144 | 0.1576 | 0.1350 | 0.2066 | 0.86 |
| blk.0.ffn_down K=3584 N=1024 | 0.1351 | 0.1601 | 0.2029 | 1.19 |
| blk.1.attn_gate K=1024 N=2048 | 0.0739 | 0.0717 | 0.1007 | 0.97 |
| blk.2.ssm_out K=2048 N=1024 | 0.1109 | 0.0866 | 0.1475 | 0.78 |
The two terms are independent — they add in quadrature, confirmed tightly:
sqrt(0.1576^2 + 0.1350^2) = 0.2074 against a measured 0.2066.
That is the strategically important part. At parity, eliminating either half ENTIRELY buys only 35% (W) or 24% (A). Neither side alone brings W4A4 near int8, and improving one by X% moves the total by roughly half that. Measured with RTN weights, so it is conservative on A's share — with GPTQ's improved W the ratio rises to ~1.07.
Two further readings:
- The absolute error is ~0.20 relative on the matmul output, where int8 would be ~1-2%. PPL 38 vs 19 is what that arithmetic predicts, not a tuning shortfall. W4A4 here is fundamentally lossy.
- It varies 2.6x across layers (0.097 to 0.257), which surfaces a lever that was not on the list.
-
Activation clipping — still first, but for a different reason than assumed: not because A
dominates, but because it is the untouched half, worth as much as the weight side, and needs no
format change. Per-row
absmax/7over a post-rotation ~Gaussian row lands ~3.6 sigma, so the step is ~0.51 sigma and most of the 4-bit range is spent on tails. An MSE-optimal clip is cheap. - Selective per-layer precision — the 2.6x spread means a handful of layers own the damage; promoting just those to int8 targets the actual distribution instead of improving everything.
- Full-rank calibration — free now that the batch count derives itself; one repack.
-
Per-group scales (group=128) — DEFERRED, not dropped. The diagnostic caps it: it addresses
only the W half, so at parity it cannot exceed ~35% even if perfect. It also needs a pack-format
change (
[N]->[N x K/128]) and costs K/G submits/core, pushing further into the axis W4A4 is already losing. But that is a reason to sequence it later and make it optional, not to abandon it:ork_i4_gptqalready supports it, so the remaining work is format + gating. Ship it as a per-model option (a pack-footer flag, so only models whose quality justifies the submit cost pay for it) rather than a global default — the cost/benefit is model-dependent, which is exactly what a per-model gate is for. - Calibration data — the llama.cpp README is narrow-domain AND is the same text PPL is measured on, so 38.19 carries a mild optimistic bias. Standard practice is ~262k rows; we use 3584.
Not worth it: act-order is likely redundant with the Hadamard rotation (it exists to handle outlier-dominated importance, which rotation flattens); damp is second-order.
-
ork_ppldoes not existtools/re/ork_ppl.cpp; the fork's CMakeLists already listed it). AGENTS' usage line corrected — it is a cmake target next toork_bench, notmake ork_ppl. -
mode_probe's int8 SDP rows scored thec->taskpoisoning asok(0,2500)= "safe" for six weeks; see Exp-2026-08-22 mode_probe Kernel Oops.
Runs, persists, reloads — but 0.59x int8 prefill AND 2.53x int8 PPL. It loses on both axes, so per-channel native W4A4 is still not viable; what changed today is that it is now measurable instead of hanging the board.
The promising direction is the one the hardware actually supports: per-group scaling + Hadamard
(= QuaRot). ork_i4_mm_run_grouped already takes per-(row,group) aScale/bScale with an fp32
drain, and weights.h records that the per-(row,group) form "composes with Hadamard = QuaRot".
Two things block it:
- the quantizer is a stub —
ork_i4_gptqreturns-ENOSYS(task #56, parked pending an AutoGPTQ cross-check), so there is nothing producing good grouped int4 weights; - cost is K/G submits/core, the 16x submit penalty on record.
Note also why a per-group scale works on the 4-bit MAC while an NF4 codebook cannot: a scale is linear, so it factors out of the sum and is applied in the fp32 drain; a codebook is a non-linear lookup that must be resolved before the multiply, and once resolved the value is no longer 4-bit. The SDP LUT sits downstream of the MAC, so it cannot help — and a dummy pass-through matmul to reach it earlier does not either (the output is int8 in activation layout, needing a host repack, and is no longer 4 bits).