Skip to content

VelociQ8 1.0

Choose a tag to compare

View all tags
@dhaern dhaern released this 21 Oct 06:54
· 9 commits to main since this release
v1.0
1ab8331

First public release of VelociQ8, based on llama.cpp v0.7.0. Version 1.1 was published at the same time and adds mixture-of-experts support; use 1.1 unless you need 1.0.

Changes

  • Q8A64 weight format with CPU kernels for Arm (dotprod) and x86-64 (AVX2 + FMA), plus a portable C path for other Arm CPUs.
  • BF16, F16 and F32 models convert to Q8A64 while loading. Pre-converted Q8A64 files load directly.
  • Faster attention for embedding models on Arm.
  • Memory locking (--load-mode mlock) works with Q8A64 weights on Linux and Windows.

On EmbeddingGemma 300M, VelociQ8 saves 47 % of the time per document against llama.cpp v0.7.0 with Q8_0 on Oracle Cloud A1 (4 threads), and 25 to 26 % on a Ryzen 9 5950X. Benchmarks are in docs/benchmarks.md and accuracy in docs/quality.md.

Downloads

File For
velociq8-1.0-linux-arm64.tar.gz Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.0-linux-arm64-multi.tar.gz Any 64-bit Arm CPU. It picks the kernels for the CPU at startup
velociq8-1.0-linux-x64.tar.gz x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.0-windows-x64.zip Windows on the same x86-64 CPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license and the other third-party notices are in licenses/ inside each archive.

Assets 7
Loading