Skip to content

Release v0.6.0

Choose a tag to compare

@github-actions github-actions released this 01 Nov 16:42
88bfaff

SiMa.ai LLiMa 0.6.0

LLiMa documentation

LLiMa 0.6.0 moves generative AI workloads from the Platform 2.1.3 baseline used by 0.5.0 to Platform 3.0.0. It adds new model execution paths, speculative decoding methods, and compiler and benchmarking tools for Modalix.

New since 0.5.0

Use Gemma 4 MTP and DFlash speculative decoding. These new methods join the existing EAGLE3 path and accelerate token generation for LLMs/VLMs up to 4x. A draft model proposes tokens and the target checks them before acceptance. Gemma 4 MTP uses compatible assistant checkpoints; DFlash adds block proposals, including support for Qwen3.5 hybrid-attention state. Compile matching target/draft packages for the selected method. Speed depends on the model pair and workload; no universal speedup is claimed. See Gemma 4 MTP (#251) and DFlash (#303).

Full Qwen 3.5 support. Supported Qwen 3.5 vision-language models from 0.8B to 9B enable local text and image understanding, giving customers more choice for assistants, reasoning and visual applications.

Compile and run Mixture-of-Experts models. New compiler and runtime paths generate the router, expert, and combine stages, then execute the experts selected for each token. Out-of-the-box support includes gpt-oss-20b and OLMoE-1B-7B. This expands LLiMa beyond dense-model execution; it does not establish support for every MoE architecture or checkpoint. See compiler support (#207) and runtime support (#210).

Add SmolVLM2 image and sampled-frame workloads. New model handling covers the 256M, 500M, and 2.2B checkpoint families, with image preprocessing and the vision-to-language connector. Supply images or previously sampled frames: video-file decoding and high-resolution crop splitting are outside this addition. Validation varies by checkpoint. See #323.

Compile through one native graph path. Model components now use the shared ModelGraph API instead of separate ONNX and native implementations. This includes native Whisper graph generation and a named-input reference-execution API.
This simplifies the path to add new model architectures to LLiMa.
GGUF K-quant unpacking is vectorized, filter sharing is enabled by default, and packed INT4 layout and sharing fixes improve compilation behavior. See native compilation (#272), reference execution (#293), and the compilation guide.

Load model assets through a provider interface. The new FileProvider interface lets integrations supply model configuration, tokenizer data, embeddings, and ELF files while keeping ordinary disk loading as the default. It enables Core's PCIe integration to keep assets on the host and fetch them for execution on Modalix. PCIe transport and host application APIs are supplied by Core. See #284.

Benchmark with dataset prompts. MoLE adds performance runs using BFCL v3, IFEval, AlpacaEval, GSM8K, and DialogSum prompts. Compare ordinary and speculative decoding with the same settings, and export timings, token counts, draft contribution, and failures. These dataset runs measure performance, not accuracy, and do not execute tools. Installation also gains a dedicated dependency profile and supported Python 3.13 use. See #318 and the MoLE guide.

Runtime improvements

  • Uses Platform 3.0 MLA-RT DMA-BUF execution directly, removing the build dependency on Neat Internals artifacts.
  • Releases MLA-RT buffer imports when their buffers are freed.
  • Fixes streaming output at the generation limit and grouped sliding-window prefill history.
  • Adds ARM NEON scanning for greedy token selection, without claiming a measured end-to-end speedup.
  • Fixes INT4 filter sharing across token counts and pads supported pre-quantized projections for 32-way layouts.

Qwen3.5 vision-language execution, NVMe embedding offload, and per-layer Whisper encoder loading were already highlighted in 0.5.0. They remain part of LLiMa and are not new features of this release.

Upgrading from 0.5.0

  • Move to Platform 3.0.0. The 0.5.0 release targeted Platform 2.1.3. Use the matching runtime, compiler, and platform providers for 0.6.0.
  • Rebuild C++ consumers against matching LLiMa headers and libraries; installed class layouts changed.
  • Update explicit compiler-stage options: replace --onnx with --source_to_fp and --quantize with --fp_to_quant, or use the default compilation pipeline. Generated ONNX artifacts are no longer produced.
  • Migrate custom compiler components to ModelGraph and generate_graph().
  • Compile new speculative pairs with the updated compiler. Gemma 4 MTP and DFlash require method-specific artifacts and metadata. Existing EAGLE3 configurations without a method field continue to default to EAGLE3.

Installation and integration

  • Modalix runtime: sima-lmm-core, sima-lmm-dev, and sima-lmm-cli provide the C++ runtime, headers/CMake integration, and CLI.
  • Runtime installation: use sima-cli neat install llima@<version-or-ref> for an available published artifact. A download-only llima/debs profile is also provided.
  • Compiler tooling: the sima-lmm wheel supplies tooling for a compatible Model Compiler environment. Separate compiler and MoLE artifact profiles support their installation workflows.
  • Platform 3.0.0: the release uses the official 3.0.0 SDK sysroot and B2499 SDK package baseline. Runtime builds link directly to the platform MLA-RT and SiMa DMA-heap libraries.
  • C++ application integration: use matching runtime headers, libraries, and compiled models. The Python layer supports CLI orchestration; it is not a separate public runtime API.

See system requirements, deployment, and the package guide.

Release status and known limitations

This is a 0.6.0 prerelease. Rebuild C++ consumers and regenerate or migrate compiled model packages when required by the updated runtime configuration. Keep the runtime, compiler, model configuration, and platform providers compatible.

The main-branch validation run passed package builds and publication, DevKit runtime tests, Model Compiler tests, and artifact promotion. These results cover the selected models and tests; they do not establish accuracy, performance, or concurrent-workload behavior for every supported model. Use the documented model-specific limits and compiled context size.

Known follow-ups:

  • Qwen3.5 Compilation A compiler layout regression reduces decoding throughput (TPS) for Qwen3.5 models compiled with this release. Use the provided precompiled models on HuggingFace for best performance once available.
  • Image-conditioned inference: a review raised a possible cache-coherency issue where CPU writes to text scales could overwrite neighboring image scales written by MLA. The reported failure has not been reproduced; investigation remains open in #345.
  • Text before a streamed tool call: #252 remains an open record for preserving an assistant preamble followed by a structured tool call.

Changes since 0.5.0