Repository navigation
VeloxQuant-MLX: Exploring KV-cache compression and custom Metal kernels for faster LLM inference on Apple Silicon #4496
rajveer43
started this conversation in
Show and tell
Replies: 1 comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi MLX community! 👋
I've been working on VeloxQuant-MLX, an open-source project exploring KV-cache compression, quantization, and custom Metal kernels for LLM inference on Apple Silicon using MLX.
The project started as an experiment around a simple question:
It has since grown into a larger benchmarking and experimentation framework with 43+ KV-cache compression methods, custom Metal kernels, profiling tools, and integrations for testing different memory/latency/quality trade-offs.
The project has now crossed 60K+ downloads, with around 30K downloads in the last month, and I'm hoping to bring more MLX developers into the project.
What VeloxQuant is exploring
VeloxQuant currently focuses on several areas:
The goal isn't to claim that one compression method is universally better.
I'm trying to build a framework where we can actually measure the trade-offs between:
memory ↔ latency ↔ throughput ↔ model quality
on real Apple Silicon hardware.
Some interesting results so far
A few experiments have produced results that I found particularly interesting.
Decode batching
Batching independent decode requests produced roughly:
3.83× higher real tokens/sec
while trying to batch work inside individual transformer layers provided almost no improvement because of residual dependencies.
That distinction became quite interesting when looking at where batching actually helps MLX inference.
KIVI / compressed KV decode
For long-context workloads, fused KIVI-style decode kernels have shown significant reductions in KV-cache memory usage, with some experiments producing large speedups compared with less optimized compressed-cache implementations.
For a simple example with:
440 tokens × 8 KV heads × head_dim 64the approximate KV representation becomes:
The interesting question is what happens to end-to-end latency and model quality after achieving those memory reductions.
That's something I'm continuing to benchmark.
Custom Metal kernels
I've also been experimenting with custom Metal kernels including work around:
scalar_affine_attendrabitq_fused_attendfused_sdparvq_attendrabitq_encodequantize_vqflash_prefill_attendrabitq_prefill_attendOne of the most useful lessons has actually come from the kernels that didn't win.
For example, in some prefill experiments, MLX's existing SDPA implementation was already operating surprisingly close to the achievable compute roofline on the tested Apple Silicon hardware.
Our custom Flash-style prefill kernel was substantially slower.
Rather than hiding those results, I'm trying to document them because understanding why an optimization fails is often just as useful as finding one that works.
Why I'm sharing this here
I've built most of VeloxQuant independently so far, but the project is reaching the point where broader hardware testing and deeper MLX expertise would be extremely valuable.
In particular, I'd love help from people interested in:
There are many experiments that need to be reproduced across different Apple Silicon generations — M1, M2, M3, M4 and newer hardware.
I'd especially love independent benchmarking and criticism of the implementations.
If something is wrong, slower than it should be, or could use MLX more effectively, please tell me. That's one of the main reasons I'm sharing it with this community.
Contributing
If you'd like to experiment with VeloxQuant, you can fork the repository, run the benchmarks on your Mac, try different models/compression methods, and share the results.
PRs, benchmark results, Metal kernel improvements, bug reports, new compression approaches, and even failed experiments are very welcome.
I'm particularly interested in finding contributors who want to work deeply on MLX + Metal inference optimization rather than just adding another compression algorithm.
GitHub:
https://github.com/rajveer43/VeloxQuant-MLX
Project website:
https://veloxquant.dev/
I'd really appreciate feedback from the MLX community.
What parts of the MLX inference stack do you think still have the most room for optimization on Apple Silicon?
And if you've worked on Metal/MLX kernels before, I'd love to hear what you think we should benchmark or optimize next.
Thanks!
Rajveer
All reactions