Skip to content

VKNN 1.2.3

Choose a tag to compare

View all tags
@katolikov katolikov released this 11 Jul 18:29
v1.2.3
a09fdbb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Patch release on top of 1.2.1/1.2.2. Adds Llama 3.2 1B Instruct as an on-device Chat model.

Added

  • Llama 3.2 1B Instruct (int4) as a selectable Chat model in the Android demo, served from katolikov/Llama-3.2-1B-vknn — 100% Vulkan (0 CPU fallback), GQA (16 layers, 8 KV heads), int4 weights. The app now carries a model-aware tokenizer/chat-template dialect (Qwen ChatML and Llama-3), so each model uses its own byte-level BPE pre-tokenizer, control tokens, and prompt template. Built with Llama.

Fixes

  • GQA decoders load in the app. The chat JNI required a position_ids input and rejected any model without one ("not a qwen2 with-past decoder"). A GQA export (Llama-3.2) derives position from the mask internally and exposes no position_ids; it now loads — mirroring the engine's own chat example (position_ids optional, with a token-by-token prefill fallback). Qwen (which has position_ids) is unchanged.

vknn-demo-1.2.3.apk is the updated Android demo.

Assets 3
Loading