VelociQ8 1.3

VelociQ8 1.3 is based on llama.cpp b11803, the release that Ollama 0.40.9 uses.

Changes

• GPU support in the portable Ollama archives for x86-64 (Linux and Windows). They include the official Vulkan backend of Ollama 0.40.9, which works on NVIDIA, AMD and Intel GPUs with their normal drivers. Layers that go to the GPU are converted from Q8A64 to Q8_0 while loading, keeping the Q8A64 rotation, and run on the llama.cpp Q8_0 kernels. The rest stays on the CPU with the VelociQ8 kernels. The same Q8A64 file works on both.
• New -cuda Ollama archives for NVIDIA GPUs, with the official CUDA 13 backend of Ollama. They need a Turing GPU or newer (GTX 16, RTX 20 and later) and a current driver. They are about 0.7-0.8 GB, so they are only on GitHub.
• The GPU path has been run on Windows 11 with an AMD Radeon RX 6900 XT through Vulkan: with EmbeddingGemma 2 it gave the same retrieval quality as the Q8_0 file of the Ollama library and was about 10 % slower. The -cuda archives have not been run on a GPU yet.
• LoRA adapters now load on rotated weights, and a LoRA on the token embeddings of a model with a tied output (Llama 3.2, for example) now works instead of stopping the server.
• Without a GPU, Q8A64 output is the same as in 1.2, checked bit for bit on the control models. The other archives and the Docker image still run on the CPU only.
• Docker image: ghcr.io/dhaern/velociq8.

Downloads

File	For
velociq8-1.3-linux-arm64.tar.gz	Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.3-linux-arm64-multi.tar.gz	Any 64-bit Arm CPU. It picks the kernels for the CPU at startup, including i8mm
velociq8-1.3-linux-x64.tar.gz	x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.3-windows-x64.zip	Windows on the same x86-64 CPUs
velociq8-1.3-ollama-0.40.9-linux-arm64.tar.gz	Portable Ollama 0.40.9 for the CPU, Linux on Arm
velociq8-1.3-ollama-0.40.9-linux-x64.tar.gz	Portable Ollama 0.40.9, Linux on x86-64: CPU, or the GPU through Vulkan
velociq8-1.3-ollama-0.40.9-windows-x64.zip	Portable Ollama 0.40.9, Windows on x86-64: CPU, or the GPU through Vulkan
velociq8-1.3-ollama-0.40.9-linux-x64-cuda.tar.gz	The same Linux archive plus CUDA for NVIDIA GPUs
velociq8-1.3-ollama-0.40.9-windows-x64-cuda.zip	The same Windows archive plus CUDA for NVIDIA GPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license, the Ollama license, the NVIDIA CUDA license in the -cuda archives and the other third-party notices are in licenses/ inside each archive.

VelociQ8 1.2

VelociQ8 1.2 is based on llama.cpp b11738, the release that Ollama 0.40.8 uses.

Changes

• Ollama support. The velociq8-1.2-ollama-0.40.8-* archives are a portable Ollama 0.40.8 for the CPU: the official ollama executable with the VelociQ8 server. Extract and run ollama serve; nothing is installed and an Ollama already on the machine is not touched. It loads Q8A64 files and converts BF16, F16 and F32 models while loading. Steps are in docs/usage.md. Tested on Linux on Arm. The Windows archive passes the static checks but has not been run on Windows yet.
• Q8A64 files of embedding models now carry their pooling type and context length in the file header, which Ollama needs to detect them. Older files still load in every version. The files on Hugging Face have been updated.
• The archives include llama, the single command that the Llama app for Windows runs, and llama-tts for text-to-speech models.
• Docker image: ghcr.io/dhaern/velociq8.
• Q8A64 output is the same as in 1.1, checked bit for bit on the control models.

Downloads

File	For
velociq8-1.2-linux-arm64.tar.gz	Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.2-linux-arm64-multi.tar.gz	Any 64-bit Arm CPU. It picks the kernels for the CPU at startup, including i8mm
velociq8-1.2-linux-x64.tar.gz	x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.2-windows-x64.zip	Windows on the same x86-64 CPUs
velociq8-1.2-ollama-0.40.8-linux-arm64.tar.gz	Portable Ollama 0.40.8 for the CPU, Linux on Arm
velociq8-1.2-ollama-0.40.8-linux-x64.tar.gz	Portable Ollama 0.40.8 for the CPU, Linux on x86-64 with AVX2 and FMA
velociq8-1.2-ollama-0.40.8-windows-x64.zip	Portable Ollama 0.40.8 for the CPU, Windows on x86-64 with AVX2 and FMA

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license and the other third-party notices are in licenses/ inside each archive.

VelociQ8 1.1

VelociQ8 1.1 is based on llama.cpp v0.7.0, like 1.0.

Changes

• Mixture-of-experts models now run their expert weights in Q8A64. On granite-3.1-1b-a400m and OLMoE-1B-7B, prompt processing is 1.8 to 2.1 times faster than llama.cpp v0.7.0 with Q8_0 on the Arm test machine and 1.5 times faster on the Ryzen 9 5950X, with less divergence from the original weights. Text generation is level with llama.cpp, except granite on Arm, where it is 8 to 9 % faster.
• On Arm CPUs with i8mm, the linux-arm64-multi archive runs prompt processing on the i8mm matrix instructions. The output is bit-identical to the dotprod kernels. It was tested under emulation and has not been timed on real hardware.
• Q8A64 files of mixture-of-experts models use a new file revision. VelociQ8 1.0 refuses them with an error. Files of dense models did not change, and 1.1 loads every file that 1.0 loads.
• The archives now include llama-quantize. It writes Q8A64 files from BF16, F16 or F32 GGUF models: llama-quantize model-BF16.gguf model-Q8A64.gguf Q8A64 4. Ready-made files are on Hugging Face.
• Docker image for Arm64 and x86-64 servers: ghcr.io/dhaern/velociq8:1.1. It runs llama-server; setup, including HTTPS, is in docs/usage.md.
• Newly checked models: granite-3.1-1b-a400m, OLMoE-1B-7B, LFM2.5-1.2B, Qwen3.5-0.8B and gemma-4-E2B.
• Dense models give the same output as in 1.0, checked bit for bit on the control models, and run at the same speed on Arm and x86-64.

Benchmarks are in docs/benchmarks.md and accuracy in docs/quality.md.

Downloads

File	For
velociq8-1.1-linux-arm64.tar.gz	Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.1-linux-arm64-multi.tar.gz	Any 64-bit Arm CPU. It picks the kernels for the CPU at startup, including i8mm
velociq8-1.1-linux-x64.tar.gz	x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.1-windows-x64.zip	Windows on the same x86-64 CPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license and the other third-party notices are in licenses/ inside each archive.

VelociQ8 1.0

First public release of VelociQ8, based on llama.cpp v0.7.0. Version 1.1 was published at the same time and adds mixture-of-experts support; use 1.1 unless you need 1.0.

Changes

• Q8A64 weight format with CPU kernels for Arm (dotprod) and x86-64 (AVX2 + FMA), plus a portable C path for other Arm CPUs.
• BF16, F16 and F32 models convert to Q8A64 while loading. Pre-converted Q8A64 files load directly.
• Faster attention for embedding models on Arm.
• Memory locking (--load-mode mlock) works with Q8A64 weights on Linux and Windows.

On EmbeddingGemma 300M, VelociQ8 saves 47 % of the time per document against llama.cpp v0.7.0 with Q8_0 on Oracle Cloud A1 (4 threads), and 25 to 26 % on a Ryzen 9 5950X. Benchmarks are in docs/benchmarks.md and accuracy in docs/quality.md.

Downloads

File	For
velociq8-1.0-linux-arm64.tar.gz	Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.0-linux-arm64-multi.tar.gz	Any 64-bit Arm CPU. It picks the kernels for the CPU at startup
velociq8-1.0-linux-x64.tar.gz	x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.0-windows-x64.zip	Windows on the same x86-64 CPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license and the other third-party notices are in licenses/ inside each archive.