Repository navigation
Exp 2026 08 18 GPTQ Int4
Repo: ork-driver (src/ork_gptq.c, examples/test_gptq.c). Status: WIP — implemented, gated off,
NOT cross-checked (task #56). Preserved: branch wip/moe-saga-2026-08-18 (ork-driver snapshot). Knob: ORK_GPTQ (pack-time).
Patch (in this zip):
orkdriver-wip-stashed.patch (src/ork_gptq.c + examples/test_gptq.c) + parent-experiments-stashed.patch (the ORK_GPTQ sig bit)—git applyonto the banked tree to restore this experiment; the git-native copy is the snapshot branch above.
Error-compensated column-sequential int4 rounding (GPTQ; Frantar et al. 2022) produces better W4A4 codes than ork's crude symmetric absmax/7 rounding, at a heavy one-time pack cost — improving PPL for the native int4 doorbell (composes with Hadamard = QuaRot: rotate then GPTQ).
Hand-rolled, dependency-free CPU implementation (Cholesky-based Hessian inverse + error-feedback sweep,
O(N·K²/2) per weight), mirroring AutoGPTQ's GPTQ.quantize to fp tolerance. Emits UNIFORM symmetric
int4 [-8,7] + per-(row,group) scale = the native-W4A4 form (dequant w = code·scale), feeding ork's int4
doorbell. Gated at the orkpack build via ORK_GPTQ; nothing routes through it by default.
Code complete and self-contained; not yet validated. No PPL/golden comparison run.
WIP, off by default. Correct-by-construction but unproven until a golden cross-check confirms it matches AutoGPTQ.
Task #56: compare ork_gptq against AutoGPTQ on a fixed (W, H) golden (bit/fp tolerance); then a
model-level PPL A/B (GPTQ int4 vs absmax int4 vs NF4). Note the current prefill win is NF4-on-CPU (no
Hadamard, no GPTQ) — GPTQ matters only for the native-int4/NPU or QuaRot line, so prioritize accordingly.
Resume from wip/moe-saga-2026-08-18 (ork-driver).
orkpack_entry.qerr is the per-weight quality figure a re-tiering policy consumes (ORK_PACK_RANK
ranks by it), but it was only ever written by ggml_backend_ork_gptq_finalize(), i.e. under ORK_GPTQ
with a calibration Hessian. Every ordinary int4 pack write left it 0. Measured over the v6 packs on
board .236, reading the index only: q27_rtn_v6.orkpack 400 entries, 0 carrying qerr.
So a user following the documented two-pass flow — build a uniform pack, then re-tier from its qerr —
got an empty ranking, because pass 1 recorded nothing.
ork-driver#2 asked for the producer;
ork_i4a8_pack_cpu_blob_qerr supplies it.
qerr = sqrt( Σ_n Σ_k imp[k]·(w − q)² / Σ_n Σ_k imp[k]·w² )
with imp[k] the caller's imatrix importance, or 1 when NULL — defined exactly as this experiment's
path defines it, so the two rank compatibly. That equivalence is the point rather than a detail:
diag(H) and the imatrix diagonal are the same quantity up to a scale factor (H = AᵀA gives
diag(H)[k] = Σ aₖ², and the imatrix stores the mean squared activation of channel k), and the metric
is relative, so the scale cancels.
Writing a different metric into the same field would have been worse than leaving it 0: a plain
‖W−Q‖ relative error ranks differently, and ggml-ork's own note already records that the
diag(H)-weighted metric "disagrees with eyeballed rankings". Two incomparable numbers in one field with
no provenance is a trap, not a feature.
For uniformly distributed test weights the quantiser noise is analytic, so the metric can be checked rather than eyeballed:
| step | absmax/7 = 0.1429 |
| RMS quantiser error | step/√12 = 0.04125 |
RMS(w), w ~ U(−1,1)
|
1/√3 = 0.5774 |
| predicted relative error | 0.0714 |
measured qerr |
0.07117 (within 0.4%) |
NF4 scoring higher on the same data (0.0936) is expected and not a defect: NF4's levels are tuned for normally distributed weights, so on uniform data a uniform grid is the better fit. Worth stating, because the naive reading of that row is "NF4 is worse", which is the opposite of why NF4 is used.
test_offline_load (CPU only, no NPU). The assertion that matters most is that the emitted blob is
byte-identical with qerr_out requested — a metric that perturbs the pack would be worse than no
metric — plus range, determinism, NULL qerr_out, and that an imatrix moves the value:
[qerr uniform ] blob identical, qerr=0.07117 imatrix-weighted qerr=0.06691
[qerr NF4 ] blob identical, qerr=0.09358 imatrix-weighted qerr=0.08579
[qerr non-pow2] blob identical, qerr=0.07130 imatrix-weighted qerr=0.06697
Added as a variant, not an extra parameter: ork_i4a8_pack_cpu_blob is public and has live callers, and
AGENTS.md §458 requires the public API stay stable when extending. It is now a one-line forward with
qerr_out = NULL, so there is one implementation and the two cannot drift — the same shape this
codebase already uses for ork_i4a8_mm_pack / ork_i4a8_mm_pack_im.