Home

VKNN

VKNN (Vulkan Neural Network, namespace vknn) is a small, dependency-free C++17 inference engine that runs ONNX models on Android GPUs. It imports a model with a hand-rolled protobuf parser, lowers it to a backend-agnostic NCHW IR, runs graph passes (shape inference, BatchNorm folding, pointwise-chain fusion, quantized-node dequantization, constant folding), partitions the graph into maximal same-backend segments, and executes each segment on one of two backends: a Vulkan compute backend tuned for a mobile GPU (the primary path — NC4HW4 packed layout, one pre-recorded command buffer per static segment, fp16 storage with fp32 accumulation) and a scalar + NEON CPU backend that serves as both the numeric reference (the "oracle") and the automatic fallback. There are no third-party runtime dependencies — only Vulkan and the C++ standard library — and every result is checked against an onnxruntime golden.

Highlights

• Dual backend, one dispatch loop. The Vulkan GPU path runs every executable operator; the CPU path is the bit-exact reference and transparent fallback. Backend assignment is per-node, and residency is reconciled only at segment boundaries.
• Full-GPU coverage. A whole benchmark model runs on the GPU with 0 CPU fallbacks. Only data-dependent control flow (Loop / If / NonMaxSuppression) and const-folded import-time shape ops stay off the GPU.
• Runs an on-device LLM. The Qwen2.5-Coder-0.5B autoregressive decoder runs entirely on the GPU with zero CPU compute fallbacks, driven by a host-side KV-cache decode loop. See Running an LLM on VKNN.
• Compile once, run many times. vknn_compile turns an ONNX model into an optimized .vxm that skips ONNX parsing and graph passes at load, with declared plan buckets for dynamic shapes.
• Precision tiers. fp16 storage + fp32 accumulation (low), selective-fp32 geometry tail (normal), or full fp32 (high) — every store rounded to nearest even.
• Zero-copy I/O. Caller-owned DMA-BUF fds bind straight to the GPU boundary buffer on the UMA device, with no host staging copy.
• Autotuned, cached kernels. Load-time GEMM/conv autotuning and a self-validating, multi-variant per-model cache make warm loads skip shader compilation, weight prepacking, and tuning.
• Extensible in one file. A new operator or backend is a single self-registering source file — no edits to core dispatch.

Navigation

• Building VKNN — host and Android builds, vknn_compile, docs site, prerequisites.
• Architecture — the dual-backend design, the CPU oracle, the ONNX importer and passes, the NC4HW4 / flat layouts, plan buckets, the 0-CPU-fallback support gate, and the ADR index.
• Operator Coverage — the op families that run on the GPU and how support is checked.
• Adding an Operator — the practical file-by-file checklist for a new op.
• Running an LLM on VKNN — the flagship guide: export, convert, the GPU op work, the decode loop, the chat app, validation, and precision.
• Supported Models — the model families VKNN runs today.