QuantForge v0.1.0

QuantForge v0.1.0

Post-training quantization (PTQ) research toolkit — RTN, AWQ and a GPTQ-based flagship,
implemented from the papers in NumPy with no deep-learning framework.

Results (4-bit, 96 cells; 384 cells across bits {2,3,4,8})

method	NMSE ↓	ActErr ↓	SQNR ↑
rtn_tensor (A)	2.25e-01	2.27e-01	7.05 dB
rtn_group (B)	8.91e-02	9.05e-02	10.73 dB
awq (C)	2.27e-01	2.06e-01	7.02 dB
quantforge (D)	1.06e-01	7.13e-02	10.01 dB

Flagship vs strongest baseline: ActErr +33.80% (primary axis), NMSE +5.23%.
Aggregate over all bit widths: ActErr +19.29%, NMSE +5.82% — leads on both axes.

Primary axis is ActErr

GPTQ minimises ‖(W−Ŵ)X‖² (output domain) while NMSE is ‖W−Ŵ‖²/‖W‖² (weight
domain). These provably diverge — the compensation migrates weight error into
directions the activations do not excite, which is the mechanism itself. The
no_hessian ablation (AWQ + LS scale refit, no GPTQ) attains the best weight NMSE of
any variant, the direct evidence.

gate.passed follows the primary axis; nmse_passed stays false and visible per bit
width rather than being dropped. no_hessian ships as a documented flag for
weight-fidelity use cases.

Damping safety factor: 3.0

24-regime grid over outlier gain (1×–500×) × fraction (0.04–0.25), both error axes:

axis	S=3	S=10	S=20	S=100
ActErr edge, mean	+0.754	+0.683	+0.572	+0.008
ActErr edge, min	+0.639	+0.507	+0.164	−2.055

S=3 is positive in all 24 regimes; S=20 degrades to +0.164 in the worst; S=100 is
what collapses. 3.0 kept as the more robust choice.

Correction. Earlier revisions justified this with a table indexed by "spread 5 / 116 /
2766" and "S=20 collapses to −71.6%". Those spread values came from an unbounded
diag.max()/diag.min() ratio — it measures one quiet channel, not the spectrum — and
produced impossible figures (114708 at d_in=256). The correct ratio
λ_max/mean(diag) is bounded by d_in and spans 15.3–82.5 here. The edge values are
unchanged; only the conditioning annotation was wrong. tests/test_quant.py now asserts
the bound.

Engineering

• 188 tests, 0 failures · coverage 85.6% (statement + branch) · ruff check clean
• Bit-identical reruns (BLAS pinned to one thread): records True, full report True
• 384 cells in 9.4 s · CI matrix on Python 3.12 and 3.13

Counter-intuitive findings (reported, not hidden)

1. 8-bit group-wise quantization is a net loss (−167.86% NMSE); real 8-bit schemes
   should stay per-tensor.
2. act-order is a net negative here — removing it improves both axes.
3. MSE-optimal clipping, not GPTQ, is the largest single contributor (+130% NMSE
   when removed).
4. AWQ contributes nothing here — the exponent search converges to α=0.

Six numeric defects found by writing the invariants first

1. np.linalg.cholesky returns the input matrix in the opposite triangle; without
   triu() the Hessian leaks into every compensation coefficient.
2. The propagation ratio is U_ij/U_ii (paper form). G_ij/G_ii agrees only at
   d_in == 2 and gave a 40× worse objective in testing.
3. W1[:, t] is a view — the update mutates the column before its error is read.
4. np.linspace group boundaries give fractional edges when group_size does not
   divide d_in, assigning columns to the wrong group.
5. AWQ/GPTQ return weights in the scaled coordinate system; without dividing by s
   the result is not a drop-in replacement for W.
6. The damping diagnostic used an unbounded ratio and reported spreads above the
   d_in bound. Now λ_max/mean(diag), with a test asserting the bound.

---

Author: 晨星 (Chenxing)