[Question]: is KV cache quantization and DSpark supported?

[qdrddr]

Hi community!
Does mesh-LLM supports KV cache and quantization so I can benefit from quantized KV cache and prompt prefixes that is available in projects such as LMCache? Is DSpark/DFlash supported so I can benefit from faster inference?

[yudin-s]

Short version: prompt-prefix caching exists, but I would not treat Mesh-LLM as having LMCache-style KV-cache/offload integration or DSpark/DFLASH support yet.

There are three separate features here:

1. Prompt-prefix reuse: yes. The Skippy OpenAI serving docs describe an automatic prompt-prefix cache. It records prefix state and reports things like usage.prompt_tokens_details.cached_tokens, skippy.kv.status, skippy.kv.cached_prompt_tokens, and skippy.kv.hit_kind. In current source, this is wired through kv_cache.mode = "lookup-record" and payloads such as resident-kv / kv-recurrent.

2. LMCache-style integration: I do not see that in the current repo. A code search for LMCache comes up empty. So if you specifically mean LMCache-compatible prefix sharing, external KV movement, or LMCache APIs, I would assume no for now.

3. DSpark / DFlash: also not supported as a working runtime path from what I can see. The speculative decoding wiring plan mentions DSpArk/DFLASH/EAGLE only as future model-native speculative strategies to detect after the runtime can resolve and launch them. The same plan says unsupported requested modes should be reported as unsupported rather than silently skipped. That reads as roadmap/design work, not a feature you can enable today.

So the practical answer is: use Mesh-LLM/Skippy prompt-prefix cache for repeated prompts or long shared prefixes, but do not expect current Mesh-LLM to provide LMCache-compatible KV-cache quantization/offload or DSpark/DFlash acceleration. If you need those exact capabilities, it is probably a feature request rather than a configuration question.

If this answers the support matrix question, please mark it as the answer so others can find the current status quickly.

[yudin-s]

Thanks for the correction on DSpark/DFlash. I should have scoped that part more narrowly: I was checking whether Mesh-LLM has an in-repo/runtime integration path for those features, not whether the external mlx-dspark project itself works. Your link is a useful correction there.

For KV-cache quantization specifically, the current repo clearly has GGUF/model quantization work through skippy-quantize and tensor recipes, but I still do not see a runtime KV-cache quantization/offload knob in the OpenAI server / kv_integration path comparable to LMCache-style external KV handling. So I would separate it this way:

• model/GGUF quantization: yes, actively present in the Skippy tooling
• prompt-prefix/KV reuse: yes, via the Skippy cache path
• runtime KV-cache quantization/offload as a serving feature: I do not see a supported switch for that yet
• DSpark/DFlash: external acceleration exists; whether Mesh-LLM should expose it directly is probably a maintainer roadmap question

So the only part I would still call a feature request is Mesh-LLM-native runtime integration for KV-cache quantization/offload or explicit DSpark/DFlash configuration, not the existence of the external DSpark implementation itself.

[michaelneale]

@yudin-s DFlash being diffusion for predicing tokens? if so yes that seems highly relevant cc @i386. It is specific to deepseek architectures?

[yudin-s]

Yes, that is the right direction: DFlash is the block-diffusion drafter side of speculative decoding. It proposes a block of future tokens in parallel, then the target model verifies the tokens, so the output should stay target-model-correct rather than becoming a separate approximate decoder.

I would not describe it as "DeepSeek architectures only", though. The mlx-dspark README frames DSpark/DFlash as coming from the DeepSpec/DeepSeek line, but the practical targets it lists are dense Qwen3 and Gemma-4 family models with matched DSpark/DFlash drafter checkpoints. DeepSeek-V4/Flash/Pro is the headline upstream use case, while the local MLX package is demonstrating the drafter method on compatible dense targets.

So for Mesh-LLM the useful integration question is probably: can Skippy resolve a compatible target + drafter pair, run a verification loop, and expose a mode choice/fallback. It is not just a generic "enable DFlash for any model" flag; it needs a matched drafter/checkpoint and the right target hidden-state/embedding interface. If no matched drafter exists, the current prompt-prefix/KV reuse path remains a separate feature rather than a replacement for DSpark/DFlash.

[michaelneale]

interesting @yudin-s - you mention mlx - thoughts on https://github.com/jbg/safemlx and a shift to mlx across platforms over say llama cpp?

[yudin-s]

My read is: safemlx is a useful signal, but I would not make it the default replacement for llama.cpp yet.

What safemlx changes is that MLX is no longer just "Apple Silicon Python experiments". That repo is Rust crates around MLX/MLX-LM, and its README explicitly calls out Apple Silicon plus Linux/NVIDIA CUDA builds, model/runtime additions such as Gemma 4 and drafter support, and cross-compilation for Apple mobile/spatial targets. So for a Mesh/Skippy direction, it is interesting if the goal is a Rust-hosted runtime path that can use MLX semantics and maybe keep Apple/CUDA/mobile code closer together.

The caveat is ecosystem breadth. llama.cpp is still the safer baseline for general local inference because its contract is very broad: GGUF model format, many quantization paths, server/CLI tooling, and a long list of backends including Metal, CUDA, HIP, Vulkan, SYCL, BLAS, WebGPU, etc. That matters if Mesh wants predictable fallback across mixed user hardware.

So I would treat MLX/safemlx as a promising accelerator/runtime lane, especially for Apple Silicon and maybe CUDA experiments, not as a universal runtime switch yet. A practical architecture would keep llama.cpp/GGUF as the broad compatibility backend, then add MLX/safemlx behind an explicit capability probe for model families where it wins: native MLX weights, matched drafter/checkpoints, Apple unified memory cases, or workloads where the Rust API surface is cleaner. If a model or platform does not match that lane, fall back to llama.cpp rather than forcing a conversion path.