Skip to content

VKNN 1.2.0

Choose a tag to compare

@katolikov katolikov released this 10 Jul 14:44
b40211a

On-device GPU LLM decode gets faster and the demo app gains a model picker.

Runtime optimization (LLM decode, Vulkan)

  • Engine-side greedy argmax (8-byte readback per token instead of the full logits row).
  • RoPE chain fusion (one dispatch per q/k site, ~288 fewer dispatches per token).
  • Fused decode attention (one flash-lite kernel per layer over the GQA KV cache).
  • Decode GPU span ~21.6 → ~19.7 ms/token; per-token host loop ~9 → ~0.5 ms.
  • All fusions are load-time, hint-gated, and never change a compiled .vxm — existing models speed up on load.

Robustness

  • Out-of-range Gather/RoPE indices and undersized weight uploads hard-error instead of crashing.
  • Multi-bucket weight-lifetime crash fixed (shared mmap blob across prefill/decode plans).
  • vknn_compile requantizes every bucket of a multi-bucket .vxm; vknn_run_io gains --bucket N.

Demo app

  • Per-tab fp16 / int4 model picker (persistent), whole-window prompt prefill, engine-side argmax.
  • 3D-splat capture gains a parallax motion gate and a capture-quality warning.

Gates

10-CNN + yonosplat8 byte-identical vs 1.1.0; chat/VLM token streams unchanged; 0 CPU fallbacks; 311 host tests; both target devices.

vknn-demo-1.2.0.apk is the Android demo (Chat / VLM / 3D Splat / Library).