Skip to content

VKNN 1.0.0

Choose a tag to compare

View all tags
@katolikov katolikov released this 07 Jul 15:32
v1.0.0
a2e8520

VKNN v1.0.0

First tagged release of VKNN — a generic, data-driven ONNX inference engine for Android
GPUs (Vulkan compute) with a bit-exact CPU oracle/fallback. New models run without manual
code or shader changes, at zero hot-path cost for fixed-shape models.

Highlights

  • Runtime dynamic shapes via declared plan buckets. --shape, --bucket, and
    Session::prepareShapes() replace the old silent batch=1 freeze. A fixed-shape model is
    exactly one bucket on a zero-key fast path — byte-identical .vxm, no extra allocation or
    re-record. Multi-bucket VXM4 files share one content-deduped weight pool; single-bucket
    files stay byte-identical VXM3.

  • Quantized checkpoints run. QDQ, QLinear, and dynamic-quant models are dequantized to
    float at import (--no-dequantize opts out). The load-bearing correctness point: a Q→DQ
    round-trip equals clamp(x, (qmin−zp)·scale, (qmax−zp)·scale), so the collapse drops the
    8-bit rounding but preserves the saturation clamp (a ReLU fused into a quant range is
    kept). This is dequantized execution, not int-exact quantized inference.

  • Full GPU parity. Every executable operator has a Vulkan kernel; the engine-emitted
    --support-report shows zero CPU fallbacks across the benchmark zoo. Closed on GPU:
    Quantize/DequantizeLinear, Cast-from-int64, TopK, general grouped Conv (per-group
    lowering), runtime-operand Clip/BatchNorm/LayerNorm/ConvTranspose/Pad, integer
    ConstantOfShape/Range/Cast, and rank>8 flat tensors. Only control-flow and const-folded
    import ops stay off the GPU, by design.

  • Autotuned tiled GEMM. matmul_tiled tile sizes (TM/TN/TK) are specialization
    constants raced per shape under --tuning fast/heavy and cached by a shape signature —
    bit-neutral, so --tuning none keeps today's bytes. −11% GPU time on the Whisper encoder.

Correctness fixes

15 root-caused fixes surfaced during the generic-engine work, including:

  • a PlanBucket use-after-free that was silently crashing all device GPU execution (the
    bucket held its Graph by value; now a unique_ptr with a liveness invariant);
  • an fp16 attention-mask overflow producing 0·−inf = NaN (out-of-fp16-range constants are
    clamped to the finite extreme at conversion, so 0·−65504 = 0);
  • forward-Conv/pool ignoring auto_pad; 1-D convolution geometry; int64 boundary decode;
    several rank-0-scalar null-derefs.
  • fp16 activation-store overflow. A conv/winograd kernel accumulates in fp32 but narrowed
    the result to fp16 on store; a finite accumulation past 65504 (a conv stack with no
    normalization) became ±inf → NaN → a silent all-zero output. fp16 activation stores now
    saturate to the finite extreme (±65504), the same NaN-avoiding clamp already applied to
    out-of-range constants. Byte-identical on the CNN suite (which never overflows).
  • Fused-pointwise broadcast bounds. A broadcast operand whose axis extent was neither 1
    nor the output extent indexed past its buffer (a CPU-oracle segfault / GPU out-of-bounds);
    both backends now validate the broadcast and reject a malformed one with a clear error.

On-device convert hardening (v1.0.0)

  • The .vxm writer now writes to a temp file and atomically renames on success, so an
    interrupted compile (e.g. Android's low-memory killer on a large model) never leaves a
    truncated .vxm at the final path.
  • benchmark/run.py convert --on device rebuilds the current Android vknn_compile before
    pushing (no more stale-binary "incompatible VXM version"), pushes the ONNX's declared
    external-data file(s) (not just a literal weights.bin, so on-device weights are no longer
    all-zero), writes the .vxm into the run's configured device dir, and fails loudly on a
    nonzero on-device compile status.
  • The loader distinguishes a stale/incompatible VXM<n> container (names the version and
    the reconvert fix) from a truncated/corrupt file.

Validation

Verified on both Xclipse devices (960 + 940):

  • Host gate green: ./build.sh, ./build.sh --android, ./build.sh --docs, scripts/ci_host.sh
    — 186 tests plus support-report / epilogue-sync / shader-contract / determinism checks.
  • CNN accuracy, both devices: 8/8 models GPU-vs-ONNX-Runtime cosine ≥ 0.99997, argmax matches,
    0 NaN, and byte-identical across the two devices (resnet50, mobilenetv2/v3, efficientnet_b0,
    shufflenet, densenet121, mnasnet, yolov8n). The fp16 store-saturation and broadcast-validation
    fixes are byte-identical to the pre-fix baseline on this suite.
  • yonosplat 8-view dl3dv encoder, both devices: 6/6 outputs cosine ≥ 0.99999 (65–81 dB PSNR),
    0 NaN, 0 CPU fallbacks, bit-identical across the two devices — the fixes leave the transformer
    path unchanged.
  • Device-convert hardening verified on both devices: an on-device convert loads and matches its
    golden, and an interrupted compile leaves no truncated .vxm.
  • Generic-engine battery: quantized GPU==CPU, 2-bucket dynamic shapes, grouped conv zero
    fallbacks; dl3dv 8-view encoder 8.64–8.69 s, faster than the pre-work baseline (8.84 s) while
    gaining the autotuning.

See docs/adr/0012-generic-data-driven-engine.md for the full design record and
docs/limitations.md for known gaps.

Assets 2
Loading