fix(tq): default --tq-mode to auto — KVarN where supported, Lloyd-Max fallback with quality warning (#432)

CLAUDE.md

@@ -113,7 +113,7 @@ The solution (`SharpInference.slnx`) is a four-layer stack, bottom-up:
 Supporting libraries:
 - **SharpInference.Diffusion** — Native image-generation pipelines. `ZImagePipeline` (Z-Image-Turbo: `ZImageDiT` single-stream S3-DiT + Qwen3-4B encoder + FLUX VAE) and `ImagePipeline` (`FluxDiT` multi-stream MMDiT + CLIP-L/T5 encoders). Includes `VaeDecoder`, `RRDBNet` (Real-ESRGAN 4× upscaler), `EulerFlowScheduler`, 2D RoPE, FP8 conversion, and Safetensors/GGUF weight loaders. Text encoders live in `TextEncoders/`.
 - **SharpInference.Vision** — Gemma 4 encoder-free vision projector (`gemma4uv`). `VisionModel` loads the mmproj GGUF; `GemmaUvVisionEmbedder` does im2col patches → projection → soft tokens; `ImagePreprocessor`/`ImageIO` handle image loading.
-- **SharpInference.TurboQuant** — KV cache compression using Lloyd-Max codebooks (3-4 bit). Codebook data lives in `codebooks/`.
+- **SharpInference.TurboQuant** — KV cache compression. Two codecs: KVarN (Hadamard + dual-axis Sinkhorn variance normalization + asymmetric RTN, 4-bit K / 2-bit V, 128-token tiles — the `--tq` default where supported) and Lloyd-Max codebooks (3-4 bit; severely degrades quality on QK-norm models such as Qwen3, issue #432 — kept as a fallback for Vulkan / partial-offload / MoE-on-GPU / SnapKV). Codebook data lives in `codebooks/`.
 - **SharpInference.Pipeline** — 3-tier memory hierarchy (VRAM → pinned RAM → NVMe), SLRU expert cache, async prefetcher.
 
 ## Key Interfaces & Patterns
@@ -122,7 +122,7 @@ Supporting libraries:
 - `IImageOpsBackend` (in Core) — extends `IComputeBackend` with convolutional image ops (Conv2d, LeakyRelu, CatChannels, PixelShuffle, Upsample2x). Implemented by `CudaBackend` and `VulkanBackend` for the RRDBNet upscaler and VAE.
 - `IForwardPass` (in Core) — per-token forward pass. Implementations in Engine: `ForwardPass` (CPU dense), `GpuForwardPass` (Vulkan), `CudaForwardPass` (CUDA dense), `HybridForwardPass`/`CudaHybridForwardPass` (dense + MoE expert offload), `HybridGdnForwardPass`/`CudaHybridGdnForwardPass` (qwen35moe hybrid Gated-DeltaNet + MoE). Has `Forward`, `Prefill`, `TruncateTo`, `ResetCache`, `VocabSize`, `MaxSeqLen`.
 - `IBatchedForwardPass` (in Engine) — multi-token batched prefill/decode used by continuous batching.
-- `PagedKvCache` (in Engine) — lazily allocated paged KV cache used by `ForwardPass`. Pages (16 positions) allocated on first write; `TruncateTo` is a soft operation (enables prefix reuse); `Reset` returns pages to a warm pool. Other cache types: `KvCache` (simple), `CudaSequenceKvCache` (per-sequence GPU), `TurboQuantKvCache` (3-bit compressed). `IMultiSlotKvCache` abstracts per-sequence/multi-slot caches. `SnapKvSelector` does prefill-time SnapKV eviction; `GdnStateCache` snapshots Gated-DeltaNet state for MTP rollback.
+- `PagedKvCache` (in Engine) — lazily allocated paged KV cache used by `ForwardPass`. Pages (16 positions) allocated on first write; `TruncateTo` is a soft operation (enables prefix reuse); `Reset` returns pages to a warm pool. Other cache types: `KvCache` (simple), `CudaSequenceKvCache` (per-sequence GPU), `TurboQuantKvCache` (KVarN 4/2-bit or Lloyd-Max 3-4 bit compressed). `IMultiSlotKvCache` abstracts per-sequence/multi-slot caches. `SnapKvSelector` does prefill-time SnapKV eviction; `GdnStateCache` snapshots Gated-DeltaNet state for MTP rollback.
 - `IInferenceEngine` (in Engine) — top-level generation interface used by the server: `GenerateAsync(prompt, sp, ct) → IAsyncEnumerable<string>`. Implemented by `InferenceEngine` (single-user, prefix caching) and `ContinuousBatchingEngine` (multi-user batching, activated via `SHARPI_MAX_BATCH`).
 - `ForwardPass.BatchForwardMulti(tokens[], positions[], caches[])` — batched multi-sequence decode; amortizes weight reads N× across concurrent users. Each sequence has its own `PagedKvCache`. Not supported for MoE or TurboQuant.
 - `ForwardPass.PrefillWithCache(tokens, cache, startPos)` — prefills a per-sequence cache (used by `ContinuousBatchingEngine` during request admission). Admission is chunked (`SHARPI_PREFILL_CHUNK`, default 256 tokens) and interleaved with decode steps; multiple in-flight prompts prefill as one packed pass via `ForwardPass.PrefillPackedMulti` and admission is gated by a KV token budget (`SHARPI_KV_BUDGET_MB`) — issue #183.

src/SharpInference.Cli/PerplexityCommand.cs

@@ -53,9 +53,9 @@ public sealed class Settings : CommandSettings
         public bool TurboQuant { get; init; }
 
         [CommandOption("--tq-mode")]
-        [Description("TurboQuant quantizer for --tq: lloydmax (default; 3-bit Lloyd-Max codebooks) or kvarn (issue #180: 4-bit K / 2-bit V, 128-token tiles; CPU only).")]
-        [DefaultValue("lloydmax")]
-        public string TqModeStr { get; init; } = "lloydmax";
+        [Description("TurboQuant quantizer for --tq: auto (default: kvarn where supported, else lloydmax with a quality warning), kvarn (issue #180: 4-bit K / 2-bit V, 128-token tiles), or lloydmax (3-bit codebooks; severely degrades quality on QK-norm models such as Qwen3 — issue #432).")]
+        [DefaultValue("auto")]
+        public string TqModeStr { get; init; } = "auto";
 
         [CommandOption("--tq-window")]
         [Description("FP32 recent-token window before compression kicks in (default: 256; min 128 for kvarn — one full tile). Also sets the first position-bucket edge of the report, so pass the same value to the fp32 baseline for bucket-comparable numbers.")]
@@ -75,26 +75,32 @@ public sealed class Settings : CommandSettings
     /// CudaForwardPass (-g -1, issue #180 Task 5a) — any other -g is rejected
     /// outright. The window floor is one compressed tile (128 for KVarN, 32 for
     /// Lloyd-Max FastScan) so the cache constructor can't throw later with a less
-    /// actionable message.
+    /// actionable message. When <paramref name="autoMode"/> comes back true the
+    /// returned quantizer is tentative — <see cref="ResolveAutoQuantizer"/> picks the
+    /// final codec once the model hyperparams are known (issue #432).
     /// </summary>
     internal static bool TryValidateFlags(bool tq, string tqModeStr, int tqWindow, int nGpuLayers,
-        out TqQuantizer quantizer, out string? error)
+        out TqQuantizer quantizer, out bool autoMode, out string? error)
     {
         quantizer = TqQuantizer.LloydMax;
+        autoMode = false;
         switch (tqModeStr.Trim().ToLowerInvariant())
         {
-            case "" or "lloydmax" or "lloyd-max":
+            case "" or "auto":
+                autoMode = true;
+                break;
+            case "lloydmax" or "lloyd-max":
                 quantizer = TqQuantizer.LloydMax;
                 break;
             case "kvarn":
                 quantizer = TqQuantizer.KVarN;
                 break;
             default:
-                error = $"Unknown --tq-mode value '{tqModeStr}'. Expected one of: lloydmax, kvarn.";
+                error = $"Unknown --tq-mode value '{tqModeStr}'. Expected one of: auto, lloydmax, kvarn.";
                 return false;
         }
 
-        if (quantizer == TqQuantizer.KVarN && !tq)
+        if (!autoMode && quantizer == TqQuantizer.KVarN && !tq)
         {
             error = "--tq-mode kvarn requires --tq.";
             return false;
@@ -104,7 +110,9 @@ internal static bool TryValidateFlags(bool tq, string tqModeStr, int tqWindow, i
             error = "the perplexity harness runs either the CPU forward pass (-g 0) or full CUDA offload (-g -1); partial offload is not supported.";
             return false;
         }
-        int minWindow = quantizer == TqQuantizer.KVarN ? 128 : 32;
+        // Auto mode only resolves to KVarN when the window fits a whole tile, so its
+        // hard floor is the Lloyd-Max one.
+        int minWindow = !autoMode && quantizer == TqQuantizer.KVarN ? 128 : 32;
         if (tq && tqWindow < minWindow)
         {
             error = $"--tq-window must be >= {minWindow} for --tq-mode {(quantizer == TqQuantizer.KVarN ? "kvarn (one full 128-token tile)" : "lloydmax (one FastScan tile)")}; got {tqWindow}.";
@@ -120,6 +128,28 @@ internal static bool TryValidateFlags(bool tq, string tqModeStr, int tqWindow, i
         return true;
     }
 
+    /// <summary>
+    /// Resolves the auto --tq-mode (issue #432): KVarN wherever it is supported,
+    /// otherwise Lloyd-Max with <paramref name="fallbackReason"/> set so the caller
+    /// can print the quality warning (Lloyd-Max 3-bit collapses on QK-norm models —
+    /// Qwen3-0.6B wikitext-2 PPL: 15.47 fp32 / 15.67 KVarN / 945.6 Lloyd-Max 3-bit).
+    /// Takes primitives instead of hyperparams so the matrix is unit-testable.
+    /// </summary>
+    internal static TqQuantizer ResolveAutoQuantizer(int tqWindow, int nGpuLayers, int headDim,
+        bool isMoE, bool snapKvEnabled, out string? fallbackReason)
+    {
+        bool headDimOk = headDim >= 8 && headDim <= 1024 && (headDim & (headDim - 1)) == 0;
+        fallbackReason =
+            snapKvEnabled ? "SnapKV eviction (SHARPI_SNAPKV_BUDGET) does not compose with KVarN yet"
+            : tqWindow < 128 ? $"KVarN needs --tq-window >= 128 (one full tile); got {tqWindow}"
+            : !headDimOk ? $"KVarN needs a power-of-2 head dim in [8, 1024]; this model has {headDim}"
+            : nGpuLayers == 0 ? null
+            : isMoE ? "KVarN on CUDA supports dense models only"
+            : headDim > 256 ? $"KVarN on CUDA requires head dim ≤ 256; this model has {headDim}"
+            : null;
+        return fallbackReason is null ? TqQuantizer.KVarN : TqQuantizer.LloydMax;
+    }
+
     /// <summary>
     /// NLL of <paramref name="target"/> under the full-vocab log-softmax of
     /// <paramref name="logits"/>, in nats. Two passes (max, then sum-exp) with
@@ -145,12 +175,12 @@ internal static double NegativeLogLikelihood(ReadOnlySpan<float> logits, int tar
     protected override int Execute(CommandContext context, Settings settings, CancellationToken cancellation)
     {
         if (!TryValidateFlags(settings.TurboQuant, settings.TqModeStr, settings.TqWindow,
-                settings.NGpuLayers, out TqQuantizer quantizer, out string? flagError))
+                settings.NGpuLayers, out TqQuantizer quantizer, out bool tqModeIsAuto, out string? flagError))
         {
             AnsiConsole.MarkupLine($"[red]Error:[/] {Markup.Escape(flagError!)}");
             return 1;
         }
-        if (settings.TurboQuant && quantizer == TqQuantizer.KVarN && SnapKvConfig.FromEnvironment().Enabled)
+        if (!tqModeIsAuto && settings.TurboQuant && quantizer == TqQuantizer.KVarN && SnapKvConfig.FromEnvironment().Enabled)
         {
             AnsiConsole.MarkupLine("[red]Error:[/] --tq-mode kvarn does not compose with SnapKV eviction yet (issue #180 follow-up); unset [yellow]SHARPI_SNAPKV_BUDGET[/].");
             return 1;
@@ -195,6 +225,19 @@ protected override int Execute(CommandContext context, Settings settings, Cancel
         var hp = ModelHyperparams.FromGgufMetadata(model.Metadata, model);
         var tokenizer = GgufTokenizer.FromGgufModel(model);
 
+        // Auto --tq-mode (issue #432): prefer KVarN, fall back to Lloyd-Max with a
+        // loud quality warning where KVarN is unsupported.
+        if (tqModeIsAuto && settings.TurboQuant)
+        {
+            quantizer = ResolveAutoQuantizer(settings.TqWindow, settings.NGpuLayers, hp.HeadDim,
+                hp.IsMoE, SnapKvConfig.FromEnvironment().Enabled, out string? fallbackReason);
+            if (fallbackReason is not null)
+                AnsiConsole.MarkupLine(
+                    $"[yellow]Warning:[/] --tq is falling back to the Lloyd-Max 3-bit quantizer ({Markup.Escape(fallbackReason)}). " +
+                    "Lloyd-Max severely degrades quality on QK-norm models such as Qwen3 (issue #432); " +
+                    "pass [yellow]--tq-mode lloydmax[/] explicitly to silence this warning.");
+        }
+
         // Same head-dim compatibility rules as the run command (issue #180): Lloyd-Max
         // ships hardcoded codebooks for 128/256; KVarN accepts any power-of-2 in
         // [8, 1024] on CPU, [8, 256] on CUDA (the shared-memory WHT cap).

src/SharpInference.Cli/README.md

@@ -48,8 +48,8 @@ Flag names are intentionally compatible with `llama.cpp` / `llama-cli`.
 | `--min-p` | `0.05` | Min-p sampling |
 | `-g, --n-gpu-layers` | `0` | Layers on GPU (`0` = CPU only, `-1` = all) |
 | `-c, --ctx-size` | model default | Context / max sequence length |
-| `--tq` | off | TurboQuant KV cache compression (3-bit, ~5× VRAM reduction) |
-| `--tq-mode` | `lloydmax` | TurboQuant quantizer: `lloydmax` (3-bit codebooks) or `kvarn` (4-bit K / 2-bit V Sinkhorn RTN, 128-token tiles; CPU only, no SnapKV) |
+| `--tq` | off | TurboQuant KV cache compression (~4-8× KV memory reduction; quantizer picked by `--tq-mode`) |
+| `--tq-mode` | `auto` | TurboQuant quantizer: `auto` (kvarn where supported, else lloydmax with a quality warning), `kvarn` (4-bit K / 2-bit V Sinkhorn RTN, 128-token tiles; CPU or full-CUDA-offload dense, no SnapKV), or `lloydmax` (3-bit codebooks — severely degrades quality on QK-norm models such as Qwen3, issue #432) |
 
 Run `sharpi-cli --help` for the full reference.