gemma4 GGUF: Q8_0 is declared in BLOCK_SHAPE/GGML_NAME/__all__ but missing from _DEQUANT, so unsloth UD-*_XL quants fail with NotImplementedError

[gerbear1990]

Summary

python/freetoken/models/gguf/dequant.py declares Q8_0 everywhere except the one table that
matters. GGML_Q8_0 = 8 has a BLOCK_SHAPE entry (32, 34), a GGML_NAME entry "Q8_0",
and is listed in __all__ — but there is no dequant_q8_0, so Q8_0 is absent from _DEQUANT
and dequantize() falls through to:

fn = _DEQUANT.get(ggml_type)
if fn is None:
    raise NotImplementedError(
        f"dequant for ggml type {GGML_NAME.get(ggml_type, ggml_type)} not implemented"
    )

The module docstring states the intended scope as "Q4_0, Q6_K, plus trivial F32/F16/BF16",
so this looks like a deliberate gap rather than an oversight — but Q8_0 being half-declared
makes it read as supported, and it is the format that blocks the most widely distributed
Gemma-4 GGUFs.

Impact

unsloth's "UD-…-XL" dynamic quants — the default download for most people — place Q8_0 on the
attention projections, the dense FFN, and token_embd. In
unsloth/gemma-4-26B-A4B-it-GGUF → gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf, 237 of 658
tensors are Q8_0:

tensor role	type	count
token_embd.weight	Q8_0	1
blk.N.attn_q/k/v/output.weight	Q8_0	115
blk.N.ffn_gate/up/down.weight	Q8_0	90
blk.N.ffn_down_exps.weight	Q8_0	30
blk.N.ffn_gate_up_exps.weight	Q8_0	1
blk.N.ffn_gate_up_exps.weight	Q6_K	29
norms, scales, router, rope_freqs	F32	392

So of the three Gemma-4 GGUFs a user is likely to already have on disk, the two UD-*_XL
ones are unloadable on quant type and the dense one is unloadable on config (filed separately).

Suggested fix

Q8_0 is the simplest ggml block format — 32 int8 weights and one fp16 scale per 34-byte block
— so this should mirror dequant_q4_0 closely:

def dequant_q8_0(raw: torch.Tensor, out_dtype: torch.dtype) -> torch.Tensor:
    d = raw[:, :2].contiguous().view(torch.float16).to(torch.float32)   # [n,1]
    qs = raw[:, 2:].view(torch.int8).to(torch.float32)                  # [n,32]
    return (d * qs).to(out_dtype)

plus the _DEQUANT[GGML_Q8_0] = dequant_q8_0 entry. BLOCK_SHAPE and row_bytes already
handle it. If Q8_0 is intentionally out of scope for the packed on-GPU expert path, then the
clearer fix is the opposite one — drop it from BLOCK_SHAPE/GGML_NAME/__all__ and fail
early at load with a message naming the tensor and type, rather than a bare
NotImplementedError deep in the loader.

Secondary: expert_quant is hardcoded for gemma4 GGUF

models/gemma4/gguf.py sets expert_quant="q4_0" unconditionally, but this checkpoint's
expert banks are Q6_K on 29 layers and Q8_0 on one. Even with a Q8_0 dequant available, the
expert bank loader would be told the wrong block layout (Q4_0 is 18 bytes/32, Q6_K is
210 bytes/256). Happy to split this out if you'd prefer it tracked separately.

Relationship to existing issues

• gemma4 GGUF: embedding quant type hardcoded to Q6_K, so a Q4_0-embedding GGUF fails to load with a bare AssertionError #188 — embedding quant type hardcoded to GGML_Q6_K. Same class of bug, and this
  checkpoint would hit it too (token_embd.weight is Q8_0, not Q6_K). gemma4 GGUF: embedding quant type hardcoded to Q6_K, so a Q4_0-embedding GGUF fails to load with a bare AssertionError #188 fixes reading
  the type from the file; this issue is about the type then having no implementation.
• perf(moe): avoid max-stride padding for heterogeneous GGUF expert banks #194 — heterogeneous GGUF expert banks; related, since the mixed Q6_K/Q8_0 expert banks
  above are exactly that shape.

Method / caveat

Found by parsing the GGUF KV + tensor-info tables directly and reading dequant.py; I have
not driven this checkpoint to a runtime traceback, because the same file also trips #188
at the embedding before the expert banks are reached. Reported as a static finding — the
tensor census above is reproducible from the file's header alone.

Environment

• FreeToken 0.1.2 (PyPI wheel + freetoken_kernel_cache-0.1.2+cu130)
• RTX 5090 (sm_120, 32 GB), driver 616.56, CUDA 13

[Cyber-Marty]

Confirmed and fixed locally. dequant_q8_0 now follows the ggml block_q8_0 layout (fp16 scale + 32 int8 quants, w = d*q, no offset) and is registered in _DEQUANT, so dequantize() routes GGML_Q8_0 instead of raising.

Tests (pure torch, CPU): round-trip through a reference quantizer mirroring quantize_row_q8_0 (d = max|w|/127, q = round(w/d)) bounds the error at half a quantization step; a hand-computed exact-values case; and a dequantize() dispatch check. PR incoming.