How compatible is this with DFlash/PFlash/etc?

[TomLucidor]

Thanks for creating a new KV cache mechanism! Last season it was TurboQuant/RotorQuant, and things are already heating up in the quantization space. But other than RAM savings, I so care about inference speeds, and whatever hacks can 3x/10x performance. This includes whatever LuceBox is cooking (which Rapid-MLX includes) https://github.com/raullenchai/Rapid-MLX https://github.com/Luce-Org/lucebox-hub

[Omc12]

Sorry for the delayed reply, missed the notification on this!

The main difference between dkv and quantization methods like TurboQuant/RotorQuant comes down to how attention is computed. Quantization shrinks individual KV vectors (paying a dequant/rotation compute cost), but decode still scans every token in the context. dkv uses structural compression—anchors + low-rank SVD deltas + exact residuals—and scores queries directly in the low-rank subspace without decompressing full keys. Paired with top-$K$ block routing, memory traffic stays bounded rather than scaling linearly with context. On long prompts, this also speeds up prefill: at 64k on Apple Silicon, dkv prefill clocks in at 477s vs 821s on dense (1.7x faster) while avoiding OOM.

As for the 3x–10x speedups in LuceBox (DFlash) and Rapid-MLX: those multipliers come from speculative decoding and kernel/graph fusion, which turn memory-bandwidth-bound single-token steps into parallel multi-token verification. dkv is completely orthogonal to that—it shrinks the KV footprint and prefill cost, while speculative decoding multiplies tokens per forward pass. You can stack them directly using dkv's --draft-model flag (and --fastdc for CUDA graphs) to get both the RAM savings and the speculative throughput boost on long sequences.

[TomLucidor]

What are some good combos then for DKV + other acceleration and memory-reduction techniques? Got any good recommendations (especially MLX)?

[Omc12]

What I would recommend pairing with dkv, especially on Apple Silicon / MLX:

External acceleration tools:

• Speculative decoding (Rapid-MLX / mlx-lm draft models / PLD): This gives the biggest real-world speed boost on Mac (usually 2x–3.5x tokens/sec). You pair a 4-bit base model (e.g. 7B or 14B) with a tiny 0.5B draft model or prompt lookup decoding (PLD). dkv compresses the KV cache underneath so the target model's long context doesn't eat up your RAM while the draft model speeds up decode.
• Prefix caching (Rapid-MLX / LMCache): If you're doing coding agent or RAG workflows where large system prompts repeat, dkv compresses cached blocks by 3x–4x so you can keep multiple long-context chat sessions resident in unified memory without spilling into macOS swap.
• 4-bit quantized weights (mlx-community): Basic standard, but keeps static model size small (~2–4 GB) so you have maximum unified memory bandwidth free for active context.

Internal dkv settings on MLX:

• --micro-block-size 1024: Best knob on Mac right now. It drops the KV pool size to 0.28x of dense (3.6x smaller) and improves multi-hop retrieval over the old 256 default since the context is split into fewer, richer chunks.
• DKV_COMPRESSED_DECODE=auto: Uses Apple Silicon's fast dense attention under 8k tokens and flips to dkv sparse routing above 8k, avoiding Python dispatch overhead on short prompts.
• DKV_CACHE_LIMIT_GB=1: Caps prefill memory to stop RAM spikes from hitting swap.
• DKV_POOL_ATTENDED_ONLY=1: On hybrid models like Qwen3.5 (where only 6 of 24 layers use full attention), this skips allocating KV pools for inactive layers and saves ~75% of cache RAM.

Putting it together in a run command:

DKV_COMPRESSED_DECODE=auto DKV_CACHE_LIMIT_GB=1 \
python ACTIVE_RUNTIME/serving/cli.py \
  --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --draft-model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --micro-block-size 1024 \
  --preset mid

This lets 4-bit weights handle static RAM, speculative decoding drive 2x–3x generation throughput, and dkv keep 32k–64k context within memory limits.