Skip to content

Centralize the KVarN support matrix + extend it to the server (TurboQuant head-dim gate) #437

Description

@pekkah

Context

Follow-up from #432 / #436 code review. The KVarN-vs-Lloyd-Max support matrix — which combinations of (SnapKV, head dim, backend, full-vs-partial offload, MoE, FP32 window) support KVarN — is currently hand-copied across three frontends:

  • RunCommand.Execute (inline kvarnBlocked ternary chain + a second explicit-kvarn validation block it must "mirror")
  • PerplexityCommand.ResolveAutoQuantizer (the only copy with unit tests)
  • InferenceEngineLoader.ResolveTq (local function + per-call-site reason strings)

The ground truth actually lives in Engine guards (CudaForwardPass KVarN head-dim ≤ 256, ForwardPass.EnableTurboQuant SnapKV throw, KVarNCompressor pow-2 [8,1024], TurboQuantCodebooks Lloyd-Max 128/256). The #432 warning text is copied 4×.

Why it matters

The partial-offload crash fixed in #436 (80c1445) was a direct symptom: the RunCommand copy downgraded auto-KVarN → Lloyd-Max on a head dim Lloyd-Max has no codebook for, because that copy didn't know the Lloyd-Max head-dim constraint. When the announced #180 follow-ups land (KVarN + SnapKV composition, KVarN for CUDA MoE / head dim > 256), each of the 3+ copies must change in lockstep; a missed site silently resolves auto → Lloyd-Max and reintroduces the exact #432 quality collapse on one surface only, with a now-wrong warning.

Proposed

  1. A single Engine-level helper next to TqQuantizer (e.g. static class TqSupport { ResolveAuto(headDim, isMoE, snapKv, backend, fullOffload, window, out reason); TryParseMode(...); }) that all three frontends call. Unit-test the matrix once.
  2. Extend it to the server: InferenceEngineLoader.cs:85 currently hard-rejects every head dim ∉ {128,256} whenever TurboQuant is on, before quantizer resolution — so KVarN's broader head-dim support (32/64/512/1024, which the CLI runs fine on the CPU path) is unreachable on the server, and explicit TqMode=kvarn on such a model gets a misleading "128 or 256" error. Relax the gate to the codec-aware envelope.
  3. Fold in the minor items from the fix(tq): default --tq-mode to auto — KVarN where supported, Lloyd-Max fallback with quality warning (#432) #436 review: the SnapKV-fallback warning that's only valid on CPU (misleading before the CUDA throw), and the server silently ignoring TqMode=kvarn when TurboQuant=false (the CLI rejects it).

Not blocking #436 — that PR is correct with the targeted guard; this is the deeper altitude fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EuBdnPNJ2XdaFp9PjYoJyz

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions