Repository navigation
Exp 2026 07 05 Int8 SiLU Output Quant
Question: the on-NPU int8 SwiGLU chain is coherent + faster at small batch but loses PPL (hybrid ~16, all-int8 ~22 vs baseline ~9/11.5). Where does the gap live, and can it be closed on-NPU?
Isolated the gate's three error sources. Decisive: same int8 gate matmul + same LUT-index path, but fp32
silu OUTPUT → PPL 8.94 ≈ baseline; flip only the silu output to int8 → ~16. So the ~7-PPL gap is entirely
the per-tensor int8 quantization of silu(g) (stored int8 to feed the ewmul), NOT the LUT-input resolution
(the LUT input is the int32 accumulator → ~1024-entry int16 LUT, already fine). The output CVT (0x4010) is
int8-only on the matmul-fused program (int16/fp16 there wedges — see the fp16 exp log).
s_silu = 1.15·max|silu|. A few large-gate outlier neurons blow up the max, so the bulk small silu values
get coarse int8. Classic LLM activation-outlier problem — but on the OUTPUT scale, not the input.
int8 fused silu on most layers (fast) + exact CPU fp32 silu on the few high-gmax outlier layers (their
int32 gate acc → per-channel host dequant → fp32 silu = ablation mode-0 quality). gmax (max|gate|) is
computed free in prep; a layer above threshold uses the exact path. Sweep (Qwen3-1.7B-Q8_0, 16 chunks):
| config | exact layers | PPL |
|---|---|---|
| baseline (per-op) | – | 11.50 |
| int8 chain (fused all) | 0 | 21.93 |
| gmax>30 | 7 / 28 | 12.31 |
| gmax>20 | 13 | 11.91 |
| gmax>12 | 19 | 11.77 |
Knee at ~7/28 layers = 92% of the gap recovered. Portability: gmax scale is per-model → ship as top-K /
median+c·MAD, auto-calibrated per model at load, NOT a fixed threshold.
Tested "clip silu-output outliers so the per-tensor scale stays small" (a plausible on-NPU fix). Sweep (16ch, no-clip=21.93): clip@99.9pct=86.7, @99.5=155, @99.0=274, @98.0=584 — monotonically WORSE. Clipping just 0.1% of silu values quadrupled PPL. The large silu outliers carry critical signal (they dominate the down-projection sum) — the LLM.int8() lesson: preserve outlier RANGE, don't clip. This confirms the sprinkle's premise: give outliers precision (per-channel/exact), don't remove them.
- Clip outliers → REFUTED (above), catastrophic.
-
Asymmetric zero-point (reclaim wasted
[-128,0]codes via0x4080/R) → preserves range, ~1 bit, safe but modest; won't reach the sprinkle. Not measured (banked as marginal). - Per-channel output scale → the right diagnosis, but the fused stage carries only a scalar R (per-channel requant is host-side on this HW, as RKNN does). = exactly what the CPU sprinkle does on the outlier layers.
- Un-fuse (int32 matmul → standalone SDP fp16/int16 silu → separate ewmul) → recovers the full gap but surrenders fusion + a SRAM roundtrip = architecturally rkllm/baseline. The guaranteed fallback.
Per-tensor int8 on the fused silu output has a hard ~21.9 ceiling that no in-stage trick escapes (one global scale can't serve both the critical outliers and the bulk). The CPU-fp32 sprinkle (per-channel/exact on the ~7 outlier layers, 12.31 PPL) is the practical on-device recovery; un-fuse is the full-recovery fallback.