Feature Request: Add max-kv-size Support to MLX HTTP Server

[stperic]

Problem
mlx_lm.generate() supports max_kv_size parameter, but mlx_lm.server doesn't. This prevents users from controlling KV cache size for memory optimization and performance tuning.
Proposed Solution
Add max-kv-size support in one or more ways:
Option 1: Startup Parameter
bashpython -m mlx_lm.server --model --max-kv-size 4096
Option 2: API Request Parameter
pythonresponse = client.chat.completions.create(
model="local-model",
messages=[...],
extra_body={"max_kv_size": 4096}
)
Option 3: Both (Recommended)

Startup parameter sets default
API requests can override per-request

Use Cases

Memory optimization: Set smaller cache on limited RAM systems
Long context: Increase cache for document processing
Mixed workloads: Different cache sizes for different request types

Benefits

Better memory management
Performance tuning flexibility
Feature parity with mlx_lm.generate()
Aligns with other frameworks (vLLM, TGI, mlx-openai-server)

Related

Issue #1170: Context size configuration
Third-party mlx-openai-server already implements --context-length

[mabaeyens]

This is already implemented and waiting for review in #1353, which adds --max-kv-size to the server alongside the KV quantization flags. Option 1 from your list (startup parameter); no per-request override.

Confirming the gap is still real on current main, in case anyone is unsure whether this aged out: server.py has no --max-kv-size argument, and git log -S'max_kv_size' -- mlx_lm/server.py finds nothing across the file's history. The plumbing underneath exists, BatchGenerator.__init__ already accepts max_kv_size, so what is missing is only the CLI wiring, which is what #1353 does.

Worth watching that PR rather than opening anything new here.