You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up from #432 / #436 code review. The KVarN-vs-Lloyd-Max support matrix — which combinations of (SnapKV, head dim, backend, full-vs-partial offload, MoE, FP32 window) support KVarN — is currently hand-copied across three frontends:
RunCommand.Execute (inline kvarnBlocked ternary chain + a second explicit-kvarn validation block it must "mirror")
PerplexityCommand.ResolveAutoQuantizer (the only copy with unit tests)
InferenceEngineLoader.ResolveTq (local function + per-call-site reason strings)
The ground truth actually lives in Engine guards (CudaForwardPass KVarN head-dim ≤ 256, ForwardPass.EnableTurboQuant SnapKV throw, KVarNCompressor pow-2 [8,1024], TurboQuantCodebooks Lloyd-Max 128/256). The #432 warning text is copied 4×.
Why it matters
The partial-offload crash fixed in #436 (80c1445) was a direct symptom: the RunCommand copy downgraded auto-KVarN → Lloyd-Max on a head dim Lloyd-Max has no codebook for, because that copy didn't know the Lloyd-Max head-dim constraint. When the announced #180 follow-ups land (KVarN + SnapKV composition, KVarN for CUDA MoE / head dim > 256), each of the 3+ copies must change in lockstep; a missed site silently resolves auto → Lloyd-Max and reintroduces the exact #432 quality collapse on one surface only, with a now-wrong warning.
Proposed
A single Engine-level helper next to TqQuantizer (e.g. static class TqSupport { ResolveAuto(headDim, isMoE, snapKv, backend, fullOffload, window, out reason); TryParseMode(...); }) that all three frontends call. Unit-test the matrix once.
Extend it to the server: InferenceEngineLoader.cs:85 currently hard-rejects every head dim ∉ {128,256} whenever TurboQuant is on, before quantizer resolution — so KVarN's broader head-dim support (32/64/512/1024, which the CLI runs fine on the CPU path) is unreachable on the server, and explicit TqMode=kvarn on such a model gets a misleading "128 or 256" error. Relax the gate to the codec-aware envelope.
Context
Follow-up from #432 / #436 code review. The KVarN-vs-Lloyd-Max support matrix — which combinations of (SnapKV, head dim, backend, full-vs-partial offload, MoE, FP32 window) support KVarN — is currently hand-copied across three frontends:
RunCommand.Execute(inlinekvarnBlockedternary chain + a second explicit-kvarn validation block it must "mirror")PerplexityCommand.ResolveAutoQuantizer(the only copy with unit tests)InferenceEngineLoader.ResolveTq(local function + per-call-site reason strings)The ground truth actually lives in Engine guards (
CudaForwardPassKVarN head-dim ≤ 256,ForwardPass.EnableTurboQuantSnapKV throw,KVarNCompressorpow-2 [8,1024],TurboQuantCodebooksLloyd-Max 128/256). The #432 warning text is copied 4×.Why it matters
The partial-offload crash fixed in #436 (80c1445) was a direct symptom: the
RunCommandcopy downgraded auto-KVarN → Lloyd-Max on a head dim Lloyd-Max has no codebook for, because that copy didn't know the Lloyd-Max head-dim constraint. When the announced #180 follow-ups land (KVarN + SnapKV composition, KVarN for CUDA MoE / head dim > 256), each of the 3+ copies must change in lockstep; a missed site silently resolves auto → Lloyd-Max and reintroduces the exact #432 quality collapse on one surface only, with a now-wrong warning.Proposed
TqQuantizer(e.g.static class TqSupport { ResolveAuto(headDim, isMoE, snapKv, backend, fullOffload, window, out reason); TryParseMode(...); }) that all three frontends call. Unit-test the matrix once.InferenceEngineLoader.cs:85currently hard-rejects every head dim ∉ {128,256} whenever TurboQuant is on, before quantizer resolution — so KVarN's broader head-dim support (32/64/512/1024, which the CLI runs fine on the CPU path) is unreachable on the server, and explicitTqMode=kvarnon such a model gets a misleading "128 or 256" error. Relax the gate to the codec-aware envelope.TqMode=kvarnwhenTurboQuant=false(the CLI rejects it).Not blocking #436 — that PR is correct with the targeted guard; this is the deeper altitude fix.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EuBdnPNJ2XdaFp9PjYoJyz