Skip to content

VKNN 1.2.2

Choose a tag to compare

View all tags
@katolikov katolikov released this 11 Jul 15:08
v1.2.2
634ba9d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Patch release on top of 1.2.1 — host-side latency levers + autotune-cache reuse. Byte-identical to 1.2.1 on every correctness gate.

Performance

  • GPU input convert. A whole-GPU graph's rank-4 fp32 image input now converts (fp16 + NC4HW4 layout) on the GPU instead of the host, removing the fixed input-pack from the run() wall. Largest win on fast networks and where the host was the bottleneck; output is byte-identical.
  • Prefill logits row readback (Session::setOutputRow). The whole-window prompt prefill reads back only the last real token's logits row instead of the full [S, vocab] matrix (~78 MB at S=256, vocab=151936), cutting one-time time-to-first-token. The decode token stream is unchanged.

Changed

  • Autotune-cache reuse (ADR-0016). Each cached autotune entry records the level it was measured at, so raising --tuning (fast → heavy) re-sweeps the heavier candidate set instead of reusing the lighter one, and --tuning none now reuses a cached fast/heavy tune when one is present — it means "run no new sweep", not "ignore a stored one". --tuning none --no-cache still emits the deterministic default kernels, so the determinism/byte gates are unaffected.

Fixes

  • setOutputRow output shape/data consistency on an out-of-range row fallback.
  • README: the bit-identical / device-independent guarantee now reads --tuning none --no-cache.

Verified byte-identical on the correctness gates — 10 CNNs across three input resolutions, Qwen and SmolVLM2 decode streams, and --tuning none --no-cache reproducing the deterministic default kernels — plus host CI.

vknn-demo-1.2.2.apk is the updated Android demo.

Assets 3
Loading