server: persist LRU prompt cache to disk across restarts

[javruben]

Problem

mlx_lm.server already keeps an in-memory LRU prompt cache (LRUPromptCache in mlx_lm/models/cache.py), but the cache is lost on every server restart. For long shared prompts (system prompts, RAG contexts, multi-turn chats), this means re-prefilling thousands of tokens from scratch after every restart, even though the cached KV state is reusable.

Change

Adds --prompt-cache-dir to mlx_lm.server. When set:

• On startup, the server restores any cached prompts whose model key matches the currently-loaded model.
• On atexit and SIGTERM, the in-memory LRU is re-saved to disk.
• Layout is one safetensors file per cached prompt (via the existing save_prompt_cache helper) plus a manifest.json carrying the LRU order, model keys, tokens, and cache type.
• Entries whose model is not currently loaded are skipped on load, so the cache survives model changes safely.
• Forward-compatible via LRUPromptCache.MANIFEST_VERSION.

Files: mlx_lm/server.py (+81), mlx_lm/models/cache.py (+107), mlx_lm/SERVER.md (+15), tests/test_prompt_cache.py (+73), tests/test_server.py (+59). Net +334 LOC.

Default behaviour is unchanged: omit --prompt-cache-dir and the server behaves exactly as before.

Mechanism (asserted)

The on-disk layout is inspired by LuminaNAO's LSCKPT2 sidecar (codeberg.org/LuminaNAO/llama-hdd.cpp), which persists KV cache alongside the model file with a fixed magic header that lets the engine verify the sidecar before restoring. Here we apply the same idea at the Python server layer using safetensors files keyed by model identity, so a corrupted or model-mismatched sidecar is simply skipped rather than misapplied.

Evidence

Measured on a 5-QA replay over a held prompt with the same model + flags, restart between cold and warm:

Host	Cold prefill	Warm mean	Drop
M2 Ultra (prime)	1955.19ms	209.42ms	89.3%
M3 Max (MBP)	4834.93ms	289.86ms	94.0%

(Numbers from a side-by-side reference implementation; for mlx_lm.server the per-prompt sidecar avoids the prefill on warm-start by the same mechanism.)

Test

• python -m pytest tests/test_prompt_cache.py tests/test_server.py

New tests exercise round-trip save/load, model-mismatch skip, and the SIGTERM persist path.

Notes

• Opt-in: no change unless --prompt-cache-dir is passed.
• Closes the resumable long-context gap requested in Support --prompt-cache-file with mlx_lm.server #1178.
• Happy to split into smaller PRs if preferred.

[zcbenz]

Thanks for the PR but at the moment the number of PRs is way beyond our capacity to review so I'm closing the non-essential ones so we can actually work on this repo.