VKNN 1.3.1

1.3.1 — full-GPU VLM

The vision-language model now runs fully on the GPU. vknn_compile fuses the token-embedding lookup into the decoder (which takes input_ids directly) and replaces the host image-feature splice with an on-GPU ScatterND, so a VLM turn — text or image — runs end-to-end on the GPU with no per-token host round-trip.

Engine (#38)

• Embed→decoder merge (generic): when one graph's output feeds another's input of the same name+shape, the producer's subgraph is copied into the consumer at compile time. On a VLM the embedding Gather moves inside the decoder. Model-agnostic — LLM prefill/decode buckets, the 3DGS encoder, and CNNs compile byte-identically.
• On-GPU image splice: an image-capable decoder copy where an on-GPU ScatterND overwrites the image-token rows with the vision features (new image_hidden_states / image_positions inputs). Text turns dispatch to a plain decoder and pay nothing.

Correctness & untrusted-input hardening (#37)

A whole-engine release audit (correctness / untrusted-input / edge-case classes, on top of the #33 memory-safety pass): 22 fixes, each confirmed by an adversarial verification pass — out-of-bounds reads on malformed .onnx/.vxm inputs, a JSON-parser hang, quantized-tensor OOB, declared-dtype output readback, and a Vulkan Gemm gate. Behaviour-preserving for valid inputs; guards fire only on malformed/corrupt/error paths.

Models

• SmolVLM2-2.2B-vknn — renamed HF repo katolikov/SmolVLM2-2.2B-vknn; both fp16 (~4.5 GB) and int4 (~1.35 GB) files recompiled with the fused, full-GPU pipeline.

Verification (two phone-class GPUs)

• Text/image decode logits byte-identical to the previous host-splice path; the end-to-end greedy token stream is byte-identical through the full KV-cached multi-turn decode.
• No regression: CNN (ResNet-50, MobileNetV2, EfficientNet-B0, DenseNet-121, MNASNet, ShuffleNet V2) and YoNoSplat 8-view PSNR/SNR bit-exact to the prior baselines, 0 CPU fallback; every non-VLM .vxm recompiles byte-identically.
• VLM per-token decode ~6 ms faster (no CPU embed hop); model reload is bit-exact; 335 host tests pass.

APK: vknn-demo-1.3.1.apk (arm64, debug-signed).

VKNN 1.3.0

1.3.0 — engine memory-safety & robustness hardening

No new models or UI: this release rebuilds the on-device engine with a full memory-safety and robustness pass and re-verifies every model class on-device. Every change is behaviour-preserving for valid inputs — the new guards fire only on malformed / corrupt / error paths, or fix an alignment/UB issue that leaves the computed value unchanged — so existing model outputs are unchanged.

Hardening (#33)

A whole-engine audit (Vulkan + CPU backends, core runtime, ONNX import, DMA-BUF examples) for memory leaks and Coverity-class defects, fixing 16 confirmed issues at the root:

• Out-of-bounds / overflow: an unaligned fp16 weight decode (undefined behaviour, a fault on strict-alignment targets); an int64 overflow in the ONNX external-data range check; a memcpy sized by a negative element count; unchecked lengths in the ONNX wire reader; an out-of-range ScatterND write/read; unbounded allocation from untrusted .vxm count fields; and out-of-range quantized side-tensor / bias / slope copies.
• Vulkan handle lifetime: fail fast on a device without VK_KHR_push_descriptor instead of a null call; free command buffers on the exception path and on segment teardown; rewrite the tuning cache via a temp file + atomic rename with a checked write.
• DMA-BUF examples: normalise MAP_FAILED and clean up fds/mmaps on every error path.

Verification

Re-verified on-device on two phone-class GPUs:

• CNNs (ResNet-50, MobileNetV2, EfficientNet-B0, DenseNet-121, MNASNet, ShuffleNet V2): cosine ≥ 0.9999 vs golden, PSNR 61–83 dB, 0 CPU fallback.
• YoNoSplat 8-view encoder: all six 3D-Gaussian outputs cosine ≥ 0.9999, PSNR 65–81 dB, 0 fallback.
• Qwen2.5-Coder 0.5B (int8): clean on-GPU decode, 0 fallback.
• Host unit tests pass; the memory-leak detector and the Address/UndefinedBehaviour sanitizers are clean (the previously-reported misaligned-load UB is gone).

Test output is also quiet by default now (#34): the negative-path diagnostics no longer bury the test log — run vknn_tests --vknn-log to see them when diagnosing a failure.

APK: vknn-demo-1.3.0.apk (arm64, debug-signed).

VKNN 1.2.5

1.2.5 — Llama 3.1 8B + a remote model catalog

Llama 3.1 8B Instruct (int4) — the largest on-device model yet

Adds Llama-3.1-8B-Instruct at int4 to the Chat picker. It decodes entirely on the GPU (Vulkan) with 0 CPU fallback, verified on-device: peak GPU working set ~6.4 GB, so a dense 8B fits on an ~11–12 GB phone-class GPU. Weights: katolikov/Llama-3.1-8B-vknn (~5.3 GB), reusing the Llama-3 chat dialect + tokenizer. Compiling a dense 8B int4 became possible thanks to a compiler fix that stores int8/uint8 initializers natively instead of widening them to fp32 (behaviour-preserving — every existing model's .vxm is byte-identical).

Remote model catalog — ship a model without an app update

The model list is now the built-in catalog merged with a remote catalog.json (GitHub raw). Adding a model becomes a one-line manifest edit instead of an APK rebuild: the app fetches the manifest on launch, merges it over the built-in list (remote wins by id), caches it, and falls back to the built-in list offline. Entries reference a built-in chat dialect by id; unknown-dialect entries are skipped.

APK: vknn-demo-1.2.5.apk (arm64, debug-signed).

VKNN 1.2.3

Patch release on top of 1.2.1/1.2.2. Adds Llama 3.2 1B Instruct as an on-device Chat model.

Added

• Llama 3.2 1B Instruct (int4) as a selectable Chat model in the Android demo, served from katolikov/Llama-3.2-1B-vknn — 100% Vulkan (0 CPU fallback), GQA (16 layers, 8 KV heads), int4 weights. The app now carries a model-aware tokenizer/chat-template dialect (Qwen ChatML and Llama-3), so each model uses its own byte-level BPE pre-tokenizer, control tokens, and prompt template. Built with Llama.

Fixes

• GQA decoders load in the app. The chat JNI required a position_ids input and rejected any model without one ("not a qwen2 with-past decoder"). A GQA export (Llama-3.2) derives position from the mask internally and exposes no position_ids; it now loads — mirroring the engine's own chat example (position_ids optional, with a token-by-token prefill fallback). Qwen (which has position_ids) is unchanged.

vknn-demo-1.2.3.apk is the updated Android demo.

VKNN 1.2.2

Patch release on top of 1.2.1 — host-side latency levers + autotune-cache reuse. Byte-identical to 1.2.1 on every correctness gate.

Performance

• GPU input convert. A whole-GPU graph's rank-4 fp32 image input now converts (fp16 + NC4HW4 layout) on the GPU instead of the host, removing the fixed input-pack from the run() wall. Largest win on fast networks and where the host was the bottleneck; output is byte-identical.
• Prefill logits row readback (Session::setOutputRow). The whole-window prompt prefill reads back only the last real token's logits row instead of the full [S, vocab] matrix (~78 MB at S=256, vocab=151936), cutting one-time time-to-first-token. The decode token stream is unchanged.

Changed

• Autotune-cache reuse (ADR-0016). Each cached autotune entry records the level it was measured at, so raising --tuning (fast → heavy) re-sweeps the heavier candidate set instead of reusing the lighter one, and --tuning none now reuses a cached fast/heavy tune when one is present — it means "run no new sweep", not "ignore a stored one". --tuning none --no-cache still emits the deterministic default kernels, so the determinism/byte gates are unaffected.

Fixes

• setOutputRow output shape/data consistency on an out-of-range row fallback.
• README: the bit-identical / device-independent guarantee now reads --tuning none --no-cache.

Verified byte-identical on the correctness gates — 10 CNNs across three input resolutions, Qwen and SmolVLM2 decode streams, and --tuning none --no-cache reproducing the deterministic default kernels — plus host CI.

vknn-demo-1.2.2.apk is the updated Android demo.

VKNN 1.2.1

Patch release on top of 1.2.0.

Fixes

• Model picker mode filter: the per-tab variant picker no longer lists a model from the wrong tab — the VLM tab offered Qwen text decoders (which failed to load) and vice versa. Ad-hoc .vxm files are now offered only in the mode their name identifies, and a stale cross-mode selection reverts to the default.

Added

• int4 SmolVLM2 as a selectable VLM variant (katolikov/smolvlm2-vknn/smolvlm2-2.2b-i4.vxm, ~1.35 GB) — the camera coach at ~1/3 the size with the same GPU pipeline and S=128 whole-window prefill; verified on device with a grounded image description and 0 CPU fallbacks.

vknn-demo-1.2.1.apk is the updated Android demo.

VKNN 1.2.0

On-device GPU LLM decode gets faster and the demo app gains a model picker.

Runtime optimization (LLM decode, Vulkan)

• Engine-side greedy argmax (8-byte readback per token instead of the full logits row).
• RoPE chain fusion (one dispatch per q/k site, ~288 fewer dispatches per token).
• Fused decode attention (one flash-lite kernel per layer over the GQA KV cache).
• Decode GPU span ~21.6 → ~19.7 ms/token; per-token host loop ~9 → ~0.5 ms.
• All fusions are load-time, hint-gated, and never change a compiled .vxm — existing models speed up on load.

Robustness

• Out-of-range Gather/RoPE indices and undersized weight uploads hard-error instead of crashing.
• Multi-bucket weight-lifetime crash fixed (shared mmap blob across prefill/decode plans).
• vknn_compile requantizes every bucket of a multi-bucket .vxm; vknn_run_io gains --bucket N.

Demo app

• Per-tab fp16 / int4 model picker (persistent), whole-window prompt prefill, engine-side argmax.
• 3D-splat capture gains a parallax motion gate and a capture-quality warning.

Gates

10-CNN + yonosplat8 byte-identical vs 1.1.0; chat/VLM token streams unchanged; 0 CPU fallbacks; 311 host tests; both target devices.

vknn-demo-1.2.0.apk is the Android demo (Chat / VLM / 3D Splat / Library).

VKNN 1.1.0

VKNN 1.1.0

3D Gaussian-splat rasterizer overhaul

• The tile-entry sort is now a stable 4-pass LSD radix sort (replaces the padded bitonic network): ~30x less sort traffic at typical scene sizes.
• Tile entries are emitted at per-gaussian scanned offsets: renders are byte-reproducible run-to-run and across devices.
• Sort buffers persist across renders; the steady state is a single submission (no mid-render fence).
• Opacity-aware binning radius and a per-quadrant coverage mask cull work that provably cannot contribute — output stays byte-identical while entry counts drop ~2.5-4.5x.
• The compositor batches tile entries through shared memory and can pack ARGB on the GPU (no host conversion loop).
• Render resolution is decoupled from the encoder input size; the demo app renders the orbit view at 512x512.
• Net effect at 224x224: ~11x faster per render. The demo app orbits continuously while dragging (previously a render was issued only after lifting the finger).

Engine-resident I/O links (GPU-resident LLM KV cache)

• New Session::linkOutputToInput API keeps designated outputs on the GPU and feeds them into the next run's inputs with ranged in-place copies — no host round-trip, values bit-exact.
• The chat and VLM decode loops adopt it: the per-token KV transfer (tens of MB each way) disappears; measured decode step ~1.6x faster, in-app chat throughput up ~30%.
• Host boundary pack/unpack conversions are parallelized (bit-identical output).

Correctness hardening

• Unbound non-batch dynamic input dims are a hard compile error naming the dimension and the --dim flag to set (previously they silently froze, producing near-zero-cosine outputs on dynamic models).
• GridSample: the GPU fp16 path now implements real cubic sampling (it previously fell back to bilinear silently), reads grid coordinates at fp32, and saturates its stores.
• fp16 saturation to +/-65504 now also covers host boundary packs, weight uploads, and six overflow-capable kernels.
• Rebinding an input with a shape that packs differently from the compiled plan is rejected (previously it silently misread memory).
• Dynamic-shape chains resolve in one import pass; the shape-inference warning storm on deep encoders is gone.
• Each planned bucket logs an un-throttled CPU-fallback summary (op counts and first node names), and vknn_run_io rejects input files whose size does not match the declared payload.

Demo app

• At most one heavy model stays resident: loading a model in any mode frees the previously loaded one first (fixes the double-load out-of-memory), and every mode has an explicit Unload control.
• App version 1.1 (versionCode 2).

Notes

• The embedded shader set changed; per-model caches self-heal (one rebuild on first load).
• The APK attached is debug-signed (no release keystore is configured for the demo app).

VKNN 1.0.0

VKNN v1.0.0

First tagged release of VKNN — a generic, data-driven ONNX inference engine for Android
GPUs (Vulkan compute) with a bit-exact CPU oracle/fallback. New models run without manual
code or shader changes, at zero hot-path cost for fixed-shape models.

Highlights

• Runtime dynamic shapes via declared plan buckets. --shape, --bucket, and
  Session::prepareShapes() replace the old silent batch=1 freeze. A fixed-shape model is
  exactly one bucket on a zero-key fast path — byte-identical .vxm, no extra allocation or
  re-record. Multi-bucket VXM4 files share one content-deduped weight pool; single-bucket
  files stay byte-identical VXM3.

• Quantized checkpoints run. QDQ, QLinear, and dynamic-quant models are dequantized to
  float at import (--no-dequantize opts out). The load-bearing correctness point: a Q→DQ
  round-trip equals clamp(x, (qmin−zp)·scale, (qmax−zp)·scale), so the collapse drops the
  8-bit rounding but preserves the saturation clamp (a ReLU fused into a quant range is
  kept). This is dequantized execution, not int-exact quantized inference.

• Full GPU parity. Every executable operator has a Vulkan kernel; the engine-emitted
  --support-report shows zero CPU fallbacks across the benchmark zoo. Closed on GPU:
  Quantize/DequantizeLinear, Cast-from-int64, TopK, general grouped Conv (per-group
  lowering), runtime-operand Clip/BatchNorm/LayerNorm/ConvTranspose/Pad, integer
  ConstantOfShape/Range/Cast, and rank>8 flat tensors. Only control-flow and const-folded
  import ops stay off the GPU, by design.

• Autotuned tiled GEMM. matmul_tiled tile sizes (TM/TN/TK) are specialization
  constants raced per shape under --tuning fast/heavy and cached by a shape signature —
  bit-neutral, so --tuning none keeps today's bytes. −11% GPU time on the Whisper encoder.

Correctness fixes

15 root-caused fixes surfaced during the generic-engine work, including:

• a PlanBucket use-after-free that was silently crashing all device GPU execution (the
  bucket held its Graph by value; now a unique_ptr with a liveness invariant);
• an fp16 attention-mask overflow producing 0·−inf = NaN (out-of-fp16-range constants are
  clamped to the finite extreme at conversion, so 0·−65504 = 0);
• forward-Conv/pool ignoring auto_pad; 1-D convolution geometry; int64 boundary decode;
  several rank-0-scalar null-derefs.
• fp16 activation-store overflow. A conv/winograd kernel accumulates in fp32 but narrowed
  the result to fp16 on store; a finite accumulation past 65504 (a conv stack with no
  normalization) became ±inf → NaN → a silent all-zero output. fp16 activation stores now
  saturate to the finite extreme (±65504), the same NaN-avoiding clamp already applied to
  out-of-range constants. Byte-identical on the CNN suite (which never overflows).
• Fused-pointwise broadcast bounds. A broadcast operand whose axis extent was neither 1
  nor the output extent indexed past its buffer (a CPU-oracle segfault / GPU out-of-bounds);
  both backends now validate the broadcast and reject a malformed one with a clear error.

On-device convert hardening (v1.0.0)

• The .vxm writer now writes to a temp file and atomically renames on success, so an
  interrupted compile (e.g. Android's low-memory killer on a large model) never leaves a
  truncated .vxm at the final path.
• benchmark/run.py convert --on device rebuilds the current Android vknn_compile before
  pushing (no more stale-binary "incompatible VXM version"), pushes the ONNX's declared
  external-data file(s) (not just a literal weights.bin, so on-device weights are no longer
  all-zero), writes the .vxm into the run's configured device dir, and fails loudly on a
  nonzero on-device compile status.
• The loader distinguishes a stale/incompatible VXM<n> container (names the version and
  the reconvert fix) from a truncated/corrupt file.

Validation

Verified on both Xclipse devices (960 + 940):

• Host gate green: ./build.sh, ./build.sh --android, ./build.sh --docs, scripts/ci_host.sh
  — 186 tests plus support-report / epilogue-sync / shader-contract / determinism checks.
• CNN accuracy, both devices: 8/8 models GPU-vs-ONNX-Runtime cosine ≥ 0.99997, argmax matches,
  0 NaN, and byte-identical across the two devices (resnet50, mobilenetv2/v3, efficientnet_b0,
  shufflenet, densenet121, mnasnet, yolov8n). The fp16 store-saturation and broadcast-validation
  fixes are byte-identical to the pre-fix baseline on this suite.
• yonosplat 8-view dl3dv encoder, both devices: 6/6 outputs cosine ≥ 0.99999 (65–81 dB PSNR),
  0 NaN, 0 CPU fallbacks, bit-identical across the two devices — the fixes leave the transformer
  path unchanged.
• Device-convert hardening verified on both devices: an on-device convert loads and matches its
  golden, and an interrupted compile leaves no truncated .vxm.
• Generic-engine battery: quantized GPU==CPU, 2-bucket dynamic shapes, grouped conv zero
  fallbacks; dl3dv 8-view encoder 8.64–8.69 s, faster than the pre-work baseline (8.84 s) while
  gaining the autotuning.

See docs/adr/0012-generic-data-driven-engine.md for the full design record and
docs/limitations.md for known gaps.