Skip to content

VelociQ8 1.3

Choose a tag to compare

View all tags
@dhaern dhaern released this 24 Oct 09:18
· 4 commits to main since this release
v1.3
fbef50d

VelociQ8 1.3 is based on llama.cpp b11803, the release that Ollama 0.40.9 uses.

Changes

  • GPU support in the portable Ollama archives for x86-64 (Linux and Windows). They include the official Vulkan backend of Ollama 0.40.9, which works on NVIDIA, AMD and Intel GPUs with their normal drivers. Layers that go to the GPU run on the llama.cpp GPU kernels. The rest stays on the CPU with the VelociQ8 kernels. The same Q8A64 file works on both.
  • New -cuda Ollama archives for NVIDIA GPUs, with the official CUDA 13 backend of Ollama. They need a Turing GPU or newer (GTX 16, RTX 20 and later) and a current driver. They are about 0.7-0.8 GB, so they are only on GitHub.
  • The GPU path has been run on Windows 11 with an AMD Radeon RX 6900 XT through Vulkan: with EmbeddingGemma 2 it gave the same retrieval quality as the Q8_0 file of the Ollama library and was about 10 % slower. The -cuda archives have not been run on a GPU yet.
  • LoRA adapters now load on Q8A64 weights on the GPU, and a LoRA on the token embeddings of a model with a tied output (Llama 3.2, for example) now works instead of stopping the server.
  • Without a GPU, Q8A64 output is the same as in 1.2, checked bit for bit on the control models. The other archives and the Docker image still run on the CPU only.
  • Docker image: ghcr.io/dhaern/velociq8.

Downloads

File For
velociq8-1.3-linux-arm64.tar.gz Arm servers with dotprod: Ampere Altra, AWS Graviton2 and later, Neoverse N1 and later
velociq8-1.3-linux-arm64-multi.tar.gz Any 64-bit Arm CPU. It picks the kernels for the CPU at startup, including i8mm
velociq8-1.3-linux-x64.tar.gz x86-64 with AVX2 and FMA: Intel Haswell, AMD Zen and later
velociq8-1.3-windows-x64.zip Windows on the same x86-64 CPUs
velociq8-1.3-ollama-0.40.9-linux-arm64.tar.gz Portable Ollama 0.40.9 for the CPU, Linux on Arm
velociq8-1.3-ollama-0.40.9-linux-x64.tar.gz Portable Ollama 0.40.9, Linux on x86-64: CPU, or the GPU through Vulkan
velociq8-1.3-ollama-0.40.9-windows-x64.zip Portable Ollama 0.40.9, Windows on x86-64: CPU, or the GPU through Vulkan
velociq8-1.3-ollama-0.40.9-linux-x64-cuda.tar.gz The same Linux archive plus CUDA for NVIDIA GPUs
velociq8-1.3-ollama-0.40.9-windows-x64-cuda.zip The same Windows archive plus CUDA for NVIDIA GPUs

The Linux archives need glibc 2.28 or newer. Check each download against SHA256SUMS:

sha256sum -c --ignore-missing SHA256SUMS

The binaries are free to use under the VelociQ8 license. The llama.cpp MIT license, the Ollama license, the NVIDIA CUDA license in the -cuda archives and the other third-party notices are in licenses/ inside each archive.

Assets 12
Loading