Skip to content

v2.2.0

Latest
Latest

Choose a tag to compare

View all tags
@dbcp1 dbcp1 released this 13 Aug 17:30
v2.2.0
145c752

LiteRT Release Notes: Version 2.2.0

Executive Summary & Scope

🌟 Release 2.2.0 Summary

1. LiteRT Runtime

Overview: Core C/C++ inference engine, delegate interfaces, platform APIs (Swift, Python, C), memory management, and hardware acceleration plugins.

🤖 Model/ Operator Coverage

XNNPACK supports for fp16 DEPTHWISE_CONV_2D, CONV_2D, FULLY_CONNECTED, and BATCH_MATMUL
⚡ Performance and Memory

  • Added a new precision mode, kLiteRtDelegatePrecisionFp16WithFp32Accum.Currently it is GPU only. When enabled the GPU backend will use FP16 storage and arithmetic with FP32 accumulation for CONV_2D, DEPTHWISE_CONV_2D, FULLY_CONNECTED, TRANSPOSE_CONV and BATCH_MAT_MUL operators , which can provide a midground between pure 32-bit and pure 16-bit computation.
  • Prepares input on GPU for the models which don’t run embedding lookup on CPU, including Gemma3 or before, and Open weight models like Qwen3
  • Added ROPE composite op for GPU to improve the performance for LLMs.
    The decode performance increased ~5%
  • For best GPU performance, weights are re-arranged in batches on the GPU to be the optimal format for execution. In this release, the batch size was changed to scale with the total size of the model to reduce memory usage and improve initialization time. The impact will vary depending on model and device type but Gemma 4 E2B on a Linux NVIDIA machine saved ~200MB.

🛠️ Hardware Acceleration

  • [Experimental] New CPU accelerator YNNPACK can be enabled through the compilation flag: --define litert_enable_ynnpack=true and runtime flag: --enable_ynnpack=true.
  • Improved MTK NPU backend's channelwise quantization support for classical vision models, such as resnet18 and mobilenet.
  • Added support for Intel NPU (LunarLake and PantherLake), enabling both JIT and AOT compilation. Verified on a variety of classic and selected GenAI models (Gemma4 2B instructions) on Windows and Linux.
  • LiteRT QAIRT accelerator
    • Major Features and Improvements
      • Supported specifying the Qualcomm SoC using either the SoC name or numeric SoC model.
      • Aligned SoC coverage with QAIRT SDK 2.47.
      • Refined OP validation and debugging logs.
      • Added documentation for QNN Native Path, QC options, and CMake toolchain builds.
      • Improved compiler plugin to avoid redundant QNN library reloads.
      • Enhanced OP coverage: A8W2 Fully Connected, BatchToSpaceNd, and SpaceToBatchNd.
    • Bug Fixes and Other Changes
      - Skipped custom op delegates when the custom op package is not registered.

🔌 APIs & Platform support

  • ABI version information has been added to data structures to enhance backward compatibility for data types used in interfaces between key LiteRT modules (libLiteRt.so, GPU Accelerators, Dispatch, Compiler). Once this compatibility feature is complete, the system will operate without issues even when using different 2.2.x version binaries together.
  • The new LiteRT Rust binding has been open-sourced. The google-ai-edge-litert crate was published on crates.ai and can be used with the standard Rust Cargo build manager. The binding is built on top of the LiteRT C++ Binary SDK.
  • Introduced new SetSelectedSignatures API
  • Added two new tensor element data type: F8E4M3FN and F8E5M2.
  • LiteRt.js API added support for streamed weight loading Dynamic Range Quantized models to WebGPU.
  • Tensor API:
    • Added RegisterCustomOpParser and MixinRegistrar to map custom flatbuffer operators (CUSTOM, STABLEHLO_COMPOSITE) to typed LiteRT operations and dynamically bind backend execution logic.
    • Added FeedbackLoopConfig to automate recurrent KV cache swapping and zero-copy buffer bindings between execution steps without requiring manual per-iteration re-binding.
    • Added Tensor API python support, extended with transformer operation surfaces, explicit Q/DQ export, quantized TFLite lowering, and Safetensors loading with compressed weight ingestion (qb4w).
    • Added JIT compilation and tracing support (compile() / WrittenModel) for authoring and executing custom tensor graphs directly in LiteRT JS.

🐛 Security, Vulnerability and Stability

  • Added integer overflow checks to convolution, reshape and pad-like CPU kernels,
  • Stricter input shape restriction in Conv3DTranspose: older versions of LiteRT also allowed this op’s number of output channels to be a multiple of the number of filter channels, but that this case was never implemented correctly and is no longer permitted.
  • Fixed a transitivity violation in TopK comparator when NaN is present. NaNs are now treated as smaller than every non-NaN value, For example, for input [1, 2, 3, NaN] with k=3, the output should be [1,2,3].
  • Enforced that the 1D size operand (operand 1) for ResizeBilinear and ResizeNearestNeighbor TFLite ops is dynamic or has dimension length 2
  • Added initial fuzzing test based on https://github.com/google/fuzztest
  • Accelerator Test Suite (ATS) improvements over all accelerator backends:
    • Expanded Operator & Subgraph coverage: Added test graph generators for Pad, Concatenation, Softmax, Transpose, BatchMatmul, FullyConnected, Reshape, Reduction, BinaryBroadcast, and OneHot, alongside a compound TransformerLayer generator built with the litert::tensor C++ API.
    • Advanced Conformance and Precision Verification: Introduced the ConformanceSpec multi-criteria validation framework (MSE, exact match, quantized buckets, accumulation tolerance), expanded FullyConnected quantization with dynamic/shuffled weights, and enabled FP16 and mixed-precision delegate testing.
    • Harness Performance and Infrastructure: Optimized test registration to be ~30x faster (reducing filter setup time from ~170s to ~6s), added full delegation hints for GPU/CPU to prevent premature validation errors, and lazily initialized test buffers to significantly reduce log noise during test execution.
  • A fix was submitted that was preventing the GPU weight cache from being utilized.
Assets 5
Loading