Programmatic load APIs cannot set KV cache quantization and parallel=1 together

[BoldFaceType]

Which version of LM Studio?
LM Studio 0.4.16.0 / FileVersion 0.4.16+1
LMS CLI commit: efce996
Runtime selected: llama.cpp-win-x86_64-nvidia-cuda12-avx2@2.20.1
SDK tested: @lmstudio/sdk 1.5.0

Which operating system?
Microsoft Windows 11 Home, version 10.0.26200, build 26200

What is the bug?
The programmatic load surfaces appear unable to set K/V cache quantization and Max Concurrent Predictions together for the same model load.

For google/gemma-4-12b-qat, I can set --parallel 1 through the lms CLI, but I cannot set K/V cache quantization there. I can set K/V cache quantization through the TypeScript SDK with llamaKCacheQuantizationType: "q8_0" and llamaVCacheQuantizationType: "q8_0", but the loaded model ends up with PARALLEL 4 in lms ps instead of the desired single concurrent prediction. The REST load endpoint rejects the equivalent K/V cache quantization and max-parallel fields as unrecognized.

Flash Attention is enabled in the SDK config. The issue does not appear to be that Flash Attention is required for V-cache quantization; the issue is that there does not seem to be one supported headless/programmatic load path that can apply both existing settings together.

Impact: this makes it difficult to script a reproducible local server configuration for long-context Gemma 4 12B QAT. Choosing CLI --parallel 1 loses K/V cache quantization, while choosing SDK K/V cache quantization falls back to the default max concurrent predictions of 4, increasing VRAM/context pressure.

Screenshots
Not applicable.

Logs
CLI load can set context and parallel successfully:

lms load google/gemma-4-12b-qat --context-length 32768 --parallel 1 --gpu max --identifier google/gemma-4-12b-qat -y
lms ps

Observed after CLI load:

IDENTIFIER                MODEL                     STATUS    SIZE       CONTEXT    PARALLEL    DEVICE    TTL
google/gemma-4-12b-qat    google/gemma-4-12b-qat    IDLE      7.15 GB    32768      1           Local

SDK load can set K/V cache quantization successfully, but lms ps reports PARALLEL 4:

import { LMStudioClient } from "@lmstudio/sdk";

const client = new LMStudioClient();
const id = "google/gemma-4-12b-qat";

await client.llm.unload(id).catch(() => {});
await client.llm.load(id, {
  identifier: id,
  verbose: false,
  config: {
    contextLength: 32768,
    gpu: { ratio: "max" },
    flashAttention: true,
    offloadKVCacheToGpu: true,
    useFp16ForKVCache: false,
    llamaKCacheQuantizationType: "q8_0",
    llamaVCacheQuantizationType: "q8_0"
  }
});

Observed after SDK load:

IDENTIFIER                MODEL                     STATUS    SIZE       CONTEXT    PARALLEL    DEVICE    TTL
google/gemma-4-12b-qat    google/gemma-4-12b-qat    IDLE      7.15 GB    32768      4           Local

REST load rejects the equivalent fields:

{
  "model": "google/gemma-4-12b-qat",
  "context_length": 32768,
  "parallel": 1,
  "max_parallel_predictions": 1,
  "flash_attention": true,
  "offload_kv_cache_to_gpu": true,
  "echo_load_config": true,
  "llama_k_cache_quantization_type": "q8_0",
  "llama_v_cache_quantization_type": "q8_0"
}

Observed REST response:

{
  "error": {
    "message": "Unrecognized key(s) in object: 'llama_k_cache_quantization_type', 'llama_v_cache_quantization_type', 'max_parallel_predictions'",
    "type": "invalid_request",
    "code": "unrecognized_keys"
  }
}

To Reproduce

1. Install/load google/gemma-4-12b-qat in LM Studio on Windows.
2. Select llama.cpp-win-x86_64-nvidia-cuda12-avx2@2.20.1 runtime.
3. Load the model using the LMS CLI with --context-length 32768 --parallel 1 --gpu max; verify lms ps shows PARALLEL 1.
4. Try to set K/V cache quantization through the CLI; there does not appear to be a CLI flag for K/V cache quantization.
5. Load the same model using the TypeScript SDK with Flash Attention enabled and llamaKCacheQuantizationType: "q8_0", llamaVCacheQuantizationType: "q8_0"; verify lms ps shows PARALLEL 4.
6. Try the REST /api/v1/models/load endpoint with equivalent K/V quantization and max-parallel fields; observe the unrecognized_keys response.

Expected behavior: at least one headless/programmatic load surface should allow setting both K/V cache quantization and Max Concurrent Predictions/parallel in the same model load. For example, the CLI could expose K/V cache quantization flags, the SDK could expose/apply maxParallelPredictions, or REST could accept both sets of fields.

Actual behavior: each surface can apply only part of the desired configuration, so the final loaded model cannot be configured headlessly with both q8_0 K/V cache and PARALLEL 1.

Possibly related but not duplicate issues found before filing: #1719, #1740, #542, #839.

[jarvis535claw]

Summary

I'm using Hermes → LiteLLM → LM Studio (latest build) via the OpenAI-compatible API with JIT model loading enabled.

Configuration

• Hermes configured to use LiteLLM as a custom OpenAI-compatible endpoint.

• LiteLLM 1.88.0 configured with:

  model_name: qwen/qwen3.6-27b
  custom_llm_provider: openai
  model: qwen/qwen3.6-27b

• LM Studio "My Models" contains a single entry for qwen/qwen3.6-27b with custom load settings (context length, K/V cache quantisation, etc.).

What I observe

1. Hermes requests qwen/qwen3.6-27b.

2. LiteLLM preserves the model name correctly.

   • A direct request to LiteLLM:

     {
       "model": "qwen/qwen3.6-27b"
     }

     returns:

     {
       "model": "qwen/qwen3.6-27b"
     }

3. LM Studio /v1/models exposes both:

   • qwen/qwen3.6-27b
   • qwen3.6-27b

4. When the model is JIT-loaded, the Loaded Models panel shows:

   qwen3.6-27b
   

   (without the qwen/ prefix)

5. The model is loaded using the default load settings rather than the custom per-model settings configured in "My Models".

Expected behaviour

If a client requests:

model = "qwen/qwen3.6-27b"

LM Studio should either:

• preserve that identifier when selecting the model profile, or
• correctly map aliases so that the saved per-model load settings are applied.

Actual behaviour

The model appears to be canonicalised to qwen3.6-27b during JIT loading, and the saved per-model load settings are not applied.

Additional observations

• This occurs with the latest LM Studio build.
• LiteLLM's live configuration contains only the prefixed model names.
• LiteLLM's API returns the prefixed model name correctly.
• The issue therefore appears to occur inside LM Studio's JIT loading/model alias handling rather than in LiteLLM.