Repository navigation
Help #1605
Unanswered
sauravsingla
asked this question in
Q&A
Help
#1605
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
I’m looking for independent low-level performance feedback on MemVanta, an open-source experiment around memory-constrained CPU LLM inference.
The optimization problem is deliberately simple:
maximize tokens/sec
subject to preserving the existing peak-memory constraint.
Areas where feedback would be especially useful include quantized kernels, SIMD/vectorization, cache locality, memory layout, threading and matrix multiplication.
Reproducible challenge:
sauravsingla/MemVanta#11
Failed optimization experiments are welcome too. The objective is to establish the actual memory/throughput frontier with reproducible evidence.
Would appreciate any technical suggestions or independent attempts.
All reactions