Release v0.3.0

LLiMa v0.3.0

LLiMa v0.3.0 updates the low-level GenAI runtime and model-compilation layer for Neat 0.3.0 and Platform 2.1.2.

This release adds speculative-decoding model generation, expands vision-model compilation, makes strided KV cache the standard path, improves Phi-4 generation quality, supports concurrent requests across served models, and makes installation and model-regression workflows more reliable.

Compatibility: Pair LLiMa v0.3.0 with the Neat 0.3.0 package set and Platform 2.1.2. Most application developers should use the higher-level Core and PyNeat APIs; use LLiMa directly for GenAI runtime, model compilation, or lower-level model integration.

Highlights

Speculative Decoding And EAGLE3 Support

• Adds generation support for EAGLE3 draft models used by speculative decoding.
• Adds the model, configuration, export, and runtime paths required to build draft-model artifacts alongside the main language model.
• Expands regression coverage for draft-model generation and end-to-end compilation.

This gives model developers a supported path for preparing LLiMa models that use speculative decoding to accelerate token generation.

Expanded Vision-Model Compilation

• Adds SiMa builder paths for Qwen and Gemma vision models.
• Moves model generation to the current afe.ir.sima_builder.SimaBuilder interface.
• Adds VLM reference configurations and compilation coverage for vision and multimodal model paths.

This expands the model-compilation foundation for Qwen- and Gemma-based VLM applications in the Neat 0.3.0 line.

Strided KV Cache By Default

• Makes strided KV cache the default compile and runtime configuration.
• Aligns generated model artifacts and runtime configuration on the same KV-cache layout.
• Removes the need to enable the strided path separately for new builds.

Build environments must use the matching AFE release, and models should be regenerated with the Neat 0.3.0 toolchain.

Correct Phi-4 And LongRoPE Generation

• Fixes degraded Phi-4 output that could collapse into repetitive sentences.
• Corrects RoPE frequency generation for proportional and LongRoPE configurations.
• Adds attention_factor handling to Python, C++, GGUF import, and generated cosine/sine tables.
• Calculates the Phi LongRoPE fallback when the model configuration does not provide it.

The result is correct positional scaling and more stable generation for Phi-4 and other LongRoPE-based models.

Safer Concurrent GenAI Requests

• Gives each request thread its own prepared MLA queue instead of sharing one process-wide queue.
• Prevents simultaneous requests for different served models from overwriting shared queue state.
• Preserves per-request batching before work is submitted to mlashmcomplex.

This improves reliability when applications serve multiple models or issue concurrent GenAI requests.

Cleaner SDK And CMake Integration

• Removes build-machine and SDK-sysroot paths from the installed SimaLMM::sima_lmm_runtime CMake target.
• Exports package-relative include directories so downstream projects can resolve their own DevKit or SDK dependencies.
• Keeps the public API unchanged while making the installed development package more portable.

More Reliable Installation And Upgrades

• Moves the LLiMa package line to version 0.3.0.
• Accepts the Neat-compatible simaai-memory-lib package range for Platform 2.1.2 instead of requiring one exact Debian revision.
• Allows Neat memory-library revisions such as 2.1.1+neat1 to satisfy LLiMa dependencies.
• Keeps LLiMa aligned with the Internals and Core packages in the Neat 0.3.0 artifact set.

Reproducible Model-Compilation Inputs

• Adds a Vulcan-backed cache for selected Hugging Face safetensors used in LLiMa compilation regression testing.
• Records the resolved Hugging Face revision, file list, size, checksum, artifact location, and public artifact URL for every cached model.
• Reduces repeated public Hub downloads and makes CI inputs stable and traceable.
• Seeds coverage for Qwen3, LFM2, LFM2-VL, and Qwen3-VL model paths.

Clearer Interactive CLI Behavior

• Removes the obsolete list command entry from interactive help and autocomplete.
• Documents that interactive CLI chat history is enabled by default.
• Fixes a small system-prompt issue in CLI mode.

Installation

Install LLiMa through the standard Neat package flow:

sima-cli neat install llima

For normal application development, installing the matching Core v0.3.0 bundle resolves the compatible LLiMa and Internals dependencies automatically.

Compatibility And Migration Notes

• LLiMa package version: 0.3.0.
• Compatible platform release: 2.1.2.
• Use LLiMa from the same Neat 0.3.0 artifact set as Core and Internals.
• Build and compilation environments require the AFE version that provides afe.ir.sima_builder.SimaBuilder.
• Strided KV cache is the default and required compilation path for the new model-generation workflow.
• Regenerate model artifacts with the matching Neat 0.3.0 compiler stack before deployment.
• Public C++ and Python API signatures remain unchanged.

Validation

Release validation included:

• LLiMa build, packaging, policy, and PR checks
• Phi-4 and LongRoPE configuration and generation-path checks
• Concurrent prepared-queue ownership checks
• CMake export and downstream package-layout checks
• Memory-library Debian dependency-range checks
• Model and unit tests for the new model-generation paths
• ONNX regression tests
• VLM reference configurations
• GGUF and end-to-end compilation tests
• EAGLE3 draft-model generation coverage
• Hugging Face cache dry-run permission checks and a real four-model publish with zero failures

Key Included Pull Requests

• #39 — Portable CMake exports and concurrent request queueing
• #36 — Reproducible Hugging Face safetensors cache
• #56 — Phi-4 and LongRoPE generation correction
• #59 — Neat-compatible memory-library dependency range
• #61 — Interactive CLI help and history cleanup
• #62 — LLiMa 0.3.0 package version
• #63 — EAGLE3, vision builders, strided KV cache, and expanded tests
• #70 — Final promotion to the 0.3 release branch

Full Changelog: v0.2.0...v0.3.0

Contributors

Thank you to everyone who contributed to this release:

• @dotimothy
• @florianvoss-commit
• @Onat-Inak
• @shanguangzai

Release v0.2.2

Release Summary

This is a version-only release for the 0.2 release line. It does not introduce product behavior changes compared with v0.2.1.

What Changed

• Bumped the release version to v0.2.2.
• No source, packaging, API, runtime, or user-facing behavior changes are included relative to v0.2.1.

User Impact

Users already on v0.2.1 should not need to change code, configuration, deployment steps, or validation expectations for v0.2.2.

Full Changelog

v0.2.1...v0.2.2

Release v0.2.1

v0.2.1 Patch Release

This patch improves SimaLMM packaging reliability and concurrent request handling for LLiMa deployments.

What changed

SimaLMM now exports cleaner CMake package metadata. The installed SimaLMM::sima_lmm_runtime target no longer exposes build-machine or sysroot-specific include paths. Instead, it points downstream projects at package-relative include directories that are valid after installation.

This release also isolates the prepared MLA request queue per request thread. Concurrent GenAI requests for different served models no longer share one process-global queue before submission to mlashmcomplex.

A small chat prompt handling issue was also fixed so system prompts are stored as plain text consistently across chat templates.

Why this matters

Before this patch, downstream projects could inherit include paths from the machine or sysroot that built the package. That made installed-package builds more fragile, especially when consuming SimaLMM from a different DevKit, SDK, or clean build environment.

The shared request queue also created risk for concurrent model-serving workloads. Requests from different threads or served models could interact through shared client-side queue state. With thread-local queue state, each request path keeps its own prepared MLA queue until it is submitted.

User benefit

• More reliable downstream builds that consume the installed SimaLMM CMake package.
• Cleaner package behavior across DevKit and SDK environments.
• Safer concurrent request handling when serving multiple model workloads.
• More predictable chat prompt behavior in CLI/chat mode.

Compatibility

This is intended to be a patch-level fix. Public SimaLMM APIs are unchanged, and no migration is required for normal users.

Projects that consume SimaLMM::sima_lmm_runtime should continue using the installed CMake target. The target now avoids leaking build-specific include paths and should be easier to consume from clean environments.

Validation

The changes were reviewed through the SimaLMM package export and DevKit C++ request paths, then promoted through the release branches for v0.2.1.

Included changes

• Fix SimaLMM packaging exports and concurrent request queueing by @florianvoss-commit in #39
• Promote the SimaLMM packaging and queue fixes through main, integration/0.2.1, and release-0.2 in #41, #42, and #43

Full Changelog: v0.2.0...v0.2.1

Release v0.2.0

LLiMa v0.2.0 updates SiMa.ai's low-level GenAI runtime and model tooling for the Neat 0.2.0 / Platform 2.1.2 line. This release aligns LLiMa with the current Neat SDK, Modalix runtime packages, and dispatcher/internals ABI used by the 2.1.x platform generation.

LLiMa is packaged together with the Neat Library. For most application development, install and use the Neat Library first; it provides the higher-level APIs and development workflow that sit above LLiMa. Use LLiMa directly when you need lower-level GenAI runtime, model-management, or Model Compiler integration.

Highlights

Platform 2.1.2 Compatibility

This release updates LLiMa package metadata and build integration for the Platform 2.1.2 generation:

• Package version moves to 0.2.0.
• Platform compatibility moves to 2.1.2.
• SDK dependency resolution is aligned with the current Neat packaging flow.
• Vulcan builds now default to the production artifact environment.
• The required simaai-memory-lib dependency is pinned through CMake configuration instead of relying on the older 2.0.0-era dependency behavior.

Standard sima-cli Installation

LLiMa now follows the standard Neat package installation path:

sima-cli neat install llima

To install a specific release, branch, or artifact reference:

sima-cli neat install llima@<version-or-ref>

This replaces older manual artifact download and Debian package installation guidance with the same package flow used across the Neat ecosystem.

Better SDK And Cross-Build Behavior

The build system has been updated for current eLxr SDK and sysroot workflows:

• Adds explicit compiler, sysroot include, and linker path handling in toolchain-sima.cmake.
• Improves Python 3.11 and nanobind discovery during cross-compilation.
• Uses target SDK headers and libraries where required, while still supporting local development builds.
• Allows local Neat internals package directories through NEAT_INTERNALS_PACKAGE_DIR for development and debugging.
• Keeps Vulcan artifact resolution as the default path for normal CI and release builds.
• Tightens post-install validation so incomplete Neat internals installations fail early with clearer diagnostics.

Runtime Integration With Current Internals

LLiMa's Modalix runtime path has been updated for the current prepared-dispatch APIs:

• Uses prepared MLA dispatch APIs for model execution.
• Uses prepared run references for single-run and queued execution paths.
• Adds relocation update handling through the dispatcher rather than ignoring relocation requests.
• Improves model release behavior so dispatcher access is reused and cleanup errors remain visible.

These changes keep LLiMa synchronized with the 2.1.x internals and dispatcher ABI.

Documentation Refresh

The documentation and README were updated to reflect the current customer-facing workflow:

• Installation guidance now points to sima-cli neat install llima.
• Setup links now point to the current sima-cli documentation.
• Examples links now use the current developer portal location.
• Supported model tables now include direct SiMa.ai Hugging Face model links for LLM, VLM, and ASR models.

Compatibility Notes

• Compatible platform release: 2.1.2.
• Intended Neat package line: 0.2.0.
• Install the Neat Library for the higher-level application APIs and runtime workflow; LLiMa remains the lower-level GenAI runtime and model tooling layer.
• LLiMa v0.2.0 should not be mixed with the 2.0.0 / Neat 0.1.x platform line unless explicitly testing compatibility boundaries.

Validation

Release readiness was based on the successful develop and promotion CI runs referenced in the release PR, including:

• 2.1.x platform preparation and build compatibility validation.
• Documentation and setup link refresh validation.
• Final develop-to-main promotion for the v0.2.0 release.

Included PRs

• #33 - Merge 2.1.1 preparation work into develop
• #30 - Refresh documentation links and supported model table
• #34 - Promote LLiMa develop to main for the v0.2.0 release

Full Changelog: v0.1.0...v0.2.0

Release v0.1.0

Summary

LLiMa v0.1.0 is the first formal release of SiMa.ai's LLiMa runtime and Model Compiler tooling for generative AI workloads on Modalix. It packages the runtime, command-line tools, Python bindings, C++ development files, model compilation helpers, and documentation needed to compile, deploy, run, and evaluate LLM, VLM, and ASR models.

This release is intended for developers using the SiMa.ai GenAI stack on Modalix. It focuses on packaging the initial supported runtime, aligning model compilation and deployment workflows, and documenting the user-facing llima CLI and Model Compiler flows.

LLiMa is packaged and released together with the Neat Library. Install the matching Neat Library package set before using LLiMa; the Neat Library provides the higher-level APIs that most applications should use, while LLiMa supplies the GenAI runtime, model-management, and Model Compiler support used by those APIs.

Release Artifacts

Platform compatibility: this release is compatible with the 2.0.0 platform version.

The Modalix runtime is delivered as Debian packages:

• sima-lmm-core: C++ runtime library, including libsima_lmm_runtime.so
• sima-lmm-dev: public C++ headers and SimaLMM CMake package metadata
• sima-lmm-cli: Python runtime package, nanobind extension, and the llima CLI entry point

The Model Compiler tooling is delivered through the sima-lmm Python package and optional dependency sets:

• sima-lmm[sdk]: model compilation dependencies
• sima-lmm[sdk_ext]: optional MoLE, benchmark, and evaluation dependencies

What's New

LLiMa Runtime And CLI

This release introduces the LLiMa runtime package set and the llima command-line workflow for Modalix.

• Run compiled GenAI model directories with llima run.
• Use CLI mode for interactive model validation.
• Use web mode for OpenAI-compatible chat-completion style workflows.
• Search, download, list, and remove precompiled models with the model manager commands.
• Configure model storage with LLIMA_MODELS_PATH or use the default Modalix model directory.
• Run LLM, VLM, and ASR model paths from one runtime package set.

Model Compilation And Deployment

The release adds Model Compiler-facing tools and documentation for preparing models for Modalix.

Supported input formats include:

• Hugging Face safetensors model directories
• GGUF LLM files
• pre-quantized GPTQ/AWQ-style safetensor models where supported

The compilation and deployment flow now covers:

• LLM and VLM compilation
• GGUF conversion
• Hugging Face transformer integration
• model deployment to Modalix devices
• LoRA deployment helpers
• host-side benchmark and evaluation helpers
• MoLE benchmarking support

Model And Architecture Support

This release brings in model/runtime support for current GenAI workloads, including language, vision-language, and speech paths.

Notable areas include:

• Llama-family LLM support
• Gemma-family LLM and VLM support
• Qwen LLM and VLM support
• Mistral-style tool-call output handling
• Whisper ASR runtime components
• LFM-family model support
• PaliGemma and LLaVA-style VLM paths
• per-layer language model handling
• mixed-precision VLM compilation guidance

See docs/index.md in the repository for the full supported-model table and limitations.

Tool Calling And Chat API Behavior

LLiMa now supports native tool-call round trips for OpenAI/Ollama-compatible chat workflows.

This includes:

• preserving incoming tool-call conversation history
• forwarding native tools payloads through chat-template handling
• parsing model-emitted tool calls into structured API responses
• supporting multiple tool-call syntaxes across common model families
• returning structured tool_calls in non-streaming and streaming responses

Runtime Fixes And Compatibility

Several runtime and packaging fixes are included in this release:

• Corrected dispatcher API usage for the current runtime dependency stack.
• Fixed runtime and packaging paths for the Modalix package layout.
• Fixed Gemma-family sliding-attention buffer sizing during runtime execution.
• Updated package naming and dependency integration for the current Neat Framework runtime packages.
• Improved SDK/sysroot handling so repeated builds do not reinstall overlays when the required payloads are already present.
• Added x86 SDK cross-build fixes for Ubuntu 24.04 based SDK environments.

Documentation

The LLiMa documentation has been restructured and converted to Markdown for external consumption.

Included documentation covers:

• LLiMa overview and supported models
• system requirements and setup
• GenAI model compilation
• model deployment
• the llima CLI runtime workflow
• MoLE benchmarking and evaluation

The documentation also updates user-facing terminology to align with Model Compiler and Neat Framework naming.

Build, Packaging, And Release Flow

This release aligns LLiMa with the current package and artifact publishing flow.

• Adds manifest-driven package versioning for Debian, CMake, and Python package metadata.
• Generates install metadata for package-based installation.
• Builds and publishes sima-lmm-core, sima-lmm-dev, and sima-lmm-cli artifacts.
• Adds branch and tag build support for release artifacts.
• Removes the older tarball-only publishing path.
• Adds a manual release workflow for release-managed builds.
• Ensures tagged LLiMa releases use matching tagged runtime dependency artifacts rather than falling back to a development branch artifact.

Compatibility Notes

• This release is compatible with platform version 2.0.0.
• LLiMa is packaged with the Neat Library and should be installed alongside the matching Neat Library package set.
• Applications should prefer the higher-level Neat Library APIs where available; LLiMa provides the underlying GenAI runtime and tooling.
• This release targets Modalix and the SiMa.ai Neat Framework runtime stack.
• Runtime models must be compiled with the matching Model Compiler flow before they are run with llima.
• GGUF support is for LLMs; VLMs should use the Hugging Face safetensors compilation path.
• Model support is limited to the architectures and constraints documented in docs/index.md.
• Tag builds are expected to use matching tagged runtime dependency artifacts.

Validation Notes

Validation called out in the merged changes includes:

• LLiMa package build validation
• Modalix runtime validation for several VLMs
• tool-call round-trip checks through the OpenAI-compatible chat path
• SDK/sysroot build-path checks
• package metadata generation checks
• documentation review and terminology cleanup
• workflow validation for release and artifact publishing

Before using this release in a production workflow, validate the target model path on the intended Modalix software image and confirm that the matching Neat Framework runtime packages are installed.

Notable Changes

• Added the initial LLiMa package and runtime integration: #5, #11
• Added model manifest resolution and model support updates: #6, #7
• Added full tool-call round-trip support: #8
• Updated package naming and runtime dependency integration: #12, #15
• Changed license metadata to Apache 2.0: #13
• Added CI/release artifact publishing flow: #14, #16, #21, #28, #31, #32
• Aligned package versioning: #18
• Corrected dispatcher API usage: #17
• Restructured and cleaned up documentation: #23, #26
• Ensured tag builds use matching dependency artifacts: #27
• Fixed README workflow badge: #29

Full changelog: https://github.com/sima-neat/llima/commits/v0.1.0