v0.17.0

Version 0.17.0 makes repeated prompts faster, adds new vision and audio models, and brings more ways to tune the Engine for different GPUs.

Highlights

• Faster repeated prompts with prefix caching - The dynamic Engine can now reuse matching prompt prefixes across requests. Prefix caching is enabled automatically for supported paged KV-cache layouts.
• Runtime profiles for different GPUs - A single model package can select Engine settings and model variants based on available GPU memory. New capability APIs also make it easier for applications to discover Engine limits.
• New LFM2 models - Added support for LFM2-VL and LFM2.5-VL vision models, LFM2-MoE, and LFM2-Audio and LFM2.5-Audio with speech input and output.
• Moonshine streaming speech recognition - Added Moonshine small and tiny streaming ASR models, designed for fast transcription with low CPU use.
• Faster DFlash decoding - DFlash now supports CUDA graph capture, prepacked weights, optional projection fusion, sampled speculative decoding, and direct Qwen 3.8 package export.
• Lower GPU memory pressure - Embedding lookups can be offloaded to CPU while keeping CUDA graph capture enabled for decoding.
• Less work during generation - Paged decoders can compute logits only for the token rows needed for generation and speculative verification.
• Safer conversation editing - Engine requests can rewind to the start of an earlier turn and continue from there without creating a new request.
• A better Model Builder configuration - The new structured configuration organizes target, drafter, speculative decoding, runtime, and search settings in one validated format.
• More Engine flexibility - Added configurable paged block sizes, improved speculative draft controls, and APIs for maximum request length.
• Reliability improvements - Includes fixes for WebGPU graph capture, decoder pipeline KV-cache shapes, paged prefill settings, legacy execution providers, and prefill logits.