Skip to content

Releases

QVB8 Inference Engine v1.2.0

Choose a tag to compare

@addiesam addiesam released this 15 Nov 19:41

QVB8 Inference Engine v1.2.0

Certified KV cache that moves between machines, a 27B model on one 16 GB V100, one-click install.

QVB8 stores model weights in a compressed format where the error of every restored weight is guaranteed to be below a chosen bound (tau). It is not a 4-bit (Q4) quantization: the bound is proven, not hoped for. The engine runs the compressed weights directly on the GPU and serves them through an OpenAI-compatible API, with speculative decoding.

What is new in v1.2.0

1. A certified KV cache that moves between machines

The saved KV state of a conversation is certified data (a proven error bound), defined by its numbers and not by a GPU. A prompt prefilled on a DGX Spark (Blackwell) was restored on a V100 (Volta) instead of being computed again, with the same answer: 10.3 s instead of 40.2 s for a 12,288-token prefix. Both sides check a transfer signature and a hash before the state is used.

2. A 27B model on one V100 16 GB

Qwen3.8-27B (QVB8R 2e-3 pack, 13.6 GB) runs on one 16 GB V100 with a DRAM offload. With VS Code + Cline it finished coding tasks of 600+ conversation rounds. It is slower than a big GPU, but it finishes the job.

3. Easy start

./install.sh creates the environment; then ./qvb8 get qwen-27b and ./qvb8 start qwen-27b (or qwen-9b). The prebuilt CUDA kernels are included, so no CUDA toolkit is needed on the GB10 and V100.

4. Several cards, several machines

  • --pipeline auto uses every visible card and places the layers by weight size and free memory.
  • Servers of one machine share saved prefixes (a pool in /dev/shm); a plain PC with a lot of RAM can hold copies (prefix store) and old KV blocks (KV host).
  • A router gives one URL for many servers and sends each request to the server that already holds the longest matching prefix.

5. Works with Cline and OpenCode

docs/manual.md has the Cline settings and a step-by-step explanation of round 1, round 2 and round n (what is computed, what is restored), with sequence diagrams.

6. More models

All LLM families of the engine: Qwen (dense and MoE) and Nemotron-H join Ripple, each with its own launcher. New drafter kind: DSpine.

7. Safer server

Admission control, a scheduler that reports when it stops, bounded request sizes, 400 for malformed requests, and a token check between machines (QV_POOL_TOKEN).

Speeds (single runs, for reference)

DGX Spark

  • Qwen3.8-27B (qvb8 1e-3): decode 18.7 tok/s on code, prefill about 980 tok/s
  • Qwen3.6-35B-A3B: decode 53.6 tok/s on code, prefill about 1,920 tok/s

V100 16 GB

  • One card, Qwen3.8-27B (qvb8r 2e-3): decode 4.9 tok/s in the benchmark, prefill about 135 tok/s
  • Two cards, Qwen3.8-27B (qvb8r 2e-3): decode 7.9 tok/s in the benchmark, prefill about 310 tok/s
  • In real Cline use on a V100: about 10 tok/s

Models on Hugging Face

Qwen3.8-27B · Qwen3.5-9B · Qwen3.6-35B-A3B · Ripple-65B-Preview

Install

git clone https://github.com/addiesam/qvb8.git && cd qvb8
./install.sh
./qvb8 get qwen-27b
./qvb8 start qwen-27b
./qvb8 chat

Honest limits

The data center layouts in the README (H100 / H200 / 288 GB cards, 16-card clusters, a trillion-parameter model) are simulations, not results. A split over more than two cards has not been run on real cards.

Try it and tell us

The use cases in the README are samples. Because the error bound is proven, new uses are open to test. Open an issue and tell us what you find. If you can lend GPU time or hardware, see "Support this project" in the README.

Full details: README.md and docs/manual.md.