Skip to content

VelociQ8 1.1

Choose a tag to compare

View all tags
@dhaern dhaern released this 21 Oct 06:54
· 11 commits to main since this release
v1.1
1ab8331

VelociQ8 1.1 is based on llama.cpp v0.7.0, like 1.0.

Changes

  • Mixture-of-experts models now run their expert weights in Q8A64. On granite-3.1-1b-a400m and OLMoE-1B-7B, prompt processing is 1.8 to 2.1 times faster than llama.cpp v0.7.0 with Q8_0 on the Arm test machine and 1.5 times faster on the Ryzen 9 5950X, with less divergence from the original weights. Text generation is level with llama.cpp, except granite on Arm, where it is 8 to 9 % faster.
  • On Arm CPUs with i8mm, the linux-arm64-multi archive runs prompt processing on the i8mm matrix instructions. The output is bit-identical to the dotprod kernels. It was tested under emulation and has not been timed on real hardware.
  • Q8A64 files of mixture-of-experts models use a new file revision. VelociQ8 1.0 refuses them with an error. Files of dense models did not change, and 1.1 loads every file that 1.0 loads.
  • The archives now include llama-quantize. It writes Q8A64 files from BF16, F16 or F32 GGUF models: llama-quantize model-BF16.gguf model-Q8A64.gguf Q8A64 4. Ready-made files are on Hugging Face.
  • Docker image for Arm64 and x86-64 servers: ghcr.io/dhaern/velociq8:1.1. It runs llama-server; setup, including HTTPS, is in docs/usage.md.
  • Newly checked models: granite-3.1-1b-a400m, OLMoE-1B-7B, LFM2.5-1.2B, Qwen3.5-0.8B and gemma-4-E2B.
  • Dense models give the same output as in 1.0, checked bit for bit on the control models, and run at the same speed on Arm and x86-64.

Benchmarks are in docs/benchmarks.md and accuracy in docs/quality.md.

Downloads

File For
velociq8-1.1-linux-arm64.tar.gz Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.1-linux-arm64-multi.tar.gz Any 64-bit Arm CPU. It picks the kernels for the CPU at startup, including i8mm
velociq8-1.1-linux-x64.tar.gz x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.1-windows-x64.zip Windows on the same x86-64 CPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license and the other third-party notices are in licenses/ inside each archive.

Assets 7
Loading