Release v0.4.0

LLiMa v0.4.0

LLiMa v0.4.0 updates the GenAI runtime and Model Compiler layer for Neat 0.4.0 and Platform 2.1.3.

This release adds 8K-token contexts, embedding and KV-cache quantization, EAGLE3 speculative decoding, multi-ELF vision encoders, structured reasoning and tool-call responses, expanded Whisper support, and installable runtime, Compiler, and MoLE artifact profiles.

Compatibility: Use LLiMa v0.4.0 with Platform 2.1.3 and the matching Internals and Core v0.4.0 packages. Most application developers should use the public Core and PyNeat GenAI APIs. Use LLiMa directly for model compilation, deployment, runtime integration, or evaluation.

Highlights

Quantized embeddings and KV caches

Embedding-table and KV-cache quantization are enabled by default for supported compilation paths.

• Embedding tables use INT8 data with BF16 per-row scales.
• KV-cache quantization reduces runtime memory use as context length grows.
• VLMs and Gemma 4 per-layer models use the same quantized embedding contract.
• EAGLE3 target and draft models can use quantized embeddings and KV caches.
• Applications can disable either optimization with --no-quantize_embeddings or --no-quantize_kv_cache.

Previously compiled models that use the older global embedding scale must be recompiled. The v0.4.0 runtime does not support that intermediate artifact format.

8K-token contexts

Most supported models can now compile with an 8K-token context:

llima-compile --max_num_tokens 8192 <model> -o <output>

The compiler uses adaptive cache masks and splits large attention reductions to keep the generated cache models within MLA limits. Sliding-attention models retain their configured sliding window while full-attention models use aligned long-context cache buckets.

The default context remains 4096 tokens. Compiled context lengths must be positive multiples of 1024.

EAGLE3 speculative decoding

LLiMa v0.4.0 supports EAGLE3 speculative decoding for supported full-attention models.

• A draft model proposes tokens that the target model verifies together.
• Target and draft artifacts compile and deploy as one model package.
• The runtime discovers the pair from their shared parent directory.
• Accepted-token cache updates use bounded cache-maintenance ranges.
• Quantized and BF16 speculative-decoding paths are available.
• Existing non-speculative model workflows are unchanged.

Sliding-window attention models remain unsupported for EAGLE3 and fail during compilation with a clear error.

New speculative artifacts require the v0.4.0 runtime. Compile and deploy the target and draft pair together.

Larger vision encoders

High-resolution vision encoders can be split across an ordered set of ELF programs when they exceed the single-ELF MLA row limit.

The runtime executes the programs sequentially with separate intermediate buffers. Existing single-ELF VLM artifacts keep their original configuration and execution path.

This release also fixes:

• Scale handling for prequantized LFM, Gemma, and Qwen vision encoders.
• LFM convolution tensor layout after embedding quantization.
• Gemma 4 text-only initialization when no image token is configured.

Reasoning responses

Supported reasoning models return reasoning separately from the assistant answer.

• OpenAI-compatible responses use reasoning_content.
• Ollama-compatible responses use thinking.
• CLI and HTTP callers can select reasoning per request.
• Reasoning text is not replayed into chat history automatically.
• Tool-call parsing remains isolated from reasoning content.
• Unsupported reasoning requests fail instead of being ignored.

The runtime recognizes reasoning boundaries for supported Qwen3, Qwen3.5, Gemma 4, and LFM2.5 Thinking models.

Tool calling across model families

Streaming and non-streaming tool calls now share one parser and response contract across supported Gemma, Llama, Qwen, Qwen-VL, Mistral, and LFM models.

The runtime handles:

• Wrapped and unwrapped tool-call formats.
• Multiple calls in one response where the model template supports them.
• Quoted, nested, and split streaming arguments.
• Gemma JSON tool-call envelopes.
• Tool allowlists and malformed-call fallback.
• Ordinary streamed content without waiting for the complete response.

Malformed calls and calls to tools outside the request allowlist remain ordinary assistant content rather than being converted into executable tool calls.

Whisper language, translation, and confidence

Whisper now supports:

• Automatic language detection.
• Explicit transcription or translation tasks.
• Detected-language metadata.
• Average token log probability.
• No-speech probability.
• OpenAI-compatible audio translation responses.

Whisper models must be regenerated and redeployed with the v0.4.0 generation flow.

The C++ WhisperModel::run_model and run_model_from_pcm methods now return TranscriptionResult instead of std::string. Direct C++ callers must read the transcript from .text.

Model retrieval and validation

llima pull now downloads large model repositories with bounded concurrency, retries transient failures, and resumes from already verified artifacts.

Every downloaded file is checked against the size and checksum published by the model repository. Incomplete models remain hidden from llima list and llima run, and damaged files are downloaded again instead of being accepted.

Runtime, Compiler, and MoLE artifacts

The root LLiMa artifact is now an installer for Modalix:

sima-cli neat install llima@v0.4.0

It installs the three LLiMa Debian packages and the exact Internals package set used by the build.

Build consumers that only need the LLiMa Debian files should use the download-only profile:

sima-cli neat install llima/debs@v0.4.0

LLiMa builds one sima-lmm Python wheel and publishes it through separate artifact profiles:

# Download the Compiler wheel into an existing Model Compiler environment
sima-cli neat install --type wheel \
  --install-dir <wheel-dir> \
  llima/compiler@v0.4.0

# Install MoLE into its managed virtual environment
sima-cli neat install llima/mole@v0.4.0

MoLE can perform Modalix log-likelihood evaluation without loading the full Hugging Face model on the host or transferring complete logits over ZMQ. The board returns the compact log-probability and greedy-token result needed by the evaluator.

Compatibility and migration

• Platform 2.1.3 is required.
• Install LLiMa, Internals, and Core from one matching Neat v0.4.0 release set.
• Use the matching Model Compiler SDK when compiling models.
• Recompile models that use context lengths not aligned to 1024 tokens.
• Recompile models that use the older global embedding-quantization scale.
• Recompile and redeploy speculative target/draft pairs with the v0.4.0 compiler and runtime.
• Deploy speculative models through their parent directory rather than selecting the target or draft component directly.
• Use the v0.4.0 runtime for newly split multi-ELF vision encoders.
• Update direct Whisper C++ callers for the TranscriptionResult return type.
• Build consumers that previously downloaded the root artifact must switch to llima/debs.
• Select reasoning per request. The deprecated startup-level thinking option has been removed.

Validation

Validation for the included changes covered:

• LLiMa Debian and wheel builds from the tagged source.
• Vulcan artifact publication and promotion.
• Model Compiler unit, graph-integration, ONNX, and direct Model SDK lanes with audited test counts.
• Packaged C++ and Python runtime tests on Modalix.
• CLI, OpenAI-compatible HTTP, and ZMQ requests.
• Persistent AppComplex operation across the complete runtime suite.
• MLA memory returning to baseline after repeated model load and shutdown cycles.
• 8K Gemma 4 text-only compilation and generation.
• EAGLE3 execution on a 16 GB Modalix system.
• Whisper transcription, language detection, translation, and confidence metadata.
• Reasoning separation with Qwen3 and Gemma 4.
• Streaming and non-streaming tool calls across several supported model families.
• Model download, checksum repair, retry, and installed-package tests.
• MoLE log-likelihood evaluation on Modalix.

Key included pull requests

• #110 - 8K contexts and default cache quantization
• #87, #160, and #164 - EAGLE3 speculative decoding
• #163 - Per-row embedding quantization
• #158 - Multi-ELF vision encoders
• #144 - Reasoning response separation
• #88 and #130 - Tool-call parsing and streaming
• #84 - Whisper language, translation, and confidence metadata
• #83 and #143 - Model retrieval and validation
• #40 - Modalix MoLE scoring
• #96, #103, and #109 - Runtime, Compiler, and MoLE artifact profiles
• #121, #131, and #187 - Packaged Modalix regression testing
• #181, #184, and #194 - Internals artifact and SDK receipt alignment

Full changelog: v0.3.0...v0.4.0

Contributors

• @florianvoss-commit
• @Onat-Inak
• @shanguangzai
• @ashish-negi-sima-ai