Skip to content

v1.31.0

Choose a tag to compare

View all tags
@ardan-bkennedy ardan-bkennedy released this 14 Aug 01:17
· 43 commits to main since this release
v1.31.0
61768a1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release Notes - v1.31.0

Release Date: August 14, 2026

Overview

Version 1.31.0 brings improved error resilience in batch generation, new load mode options, VRAM optimization with per-sequence output limits, GGUF filetype/quantization visibility, and an update to llama.cpp b10423.

Detailed Changes

Improvements

LoadModeAuto Default and LoadModeMMapMLock

  • Author: William Kennedy
  • Changed default LoadMode from mmap to auto, which uses mmap when every selected device supports it and falls back to ordinary loading otherwise
  • Added LoadModeMMapMLock option that combines mmap with resident page locking
  • Updated DerefLoadMode to default to LoadModeAuto instead of LoadModeMMap
  • Updated config documentation and YAML defaults to reflect the new options

NOutputsMaxPerSeq VRAM Optimization

  • Author: William Kennedy
  • Added NOutputsMaxPerSeq context parameter for per-sequence logit row reservation
  • Prevents a single sequence from claiming the total logit rows, freeing significant VRAM at large ubatch sizes with MTP
  • Applied in both generation and IMC prefill paths
  • Context pool fallback now respects the per-sequence limit

GGUF FileType and Quantization Visibility

  • Author: William Kennedy
  • Added FileType and Quantization fields to ModelInfo struct in both SDK and tools
  • Model info display now shows file type and friendly quantization name (e.g., "Q4_K - Medium")
  • Updated ModelInfoFromMetadata to parse general.file_type from GGUF headers
  • Added TestGGMLTypeSizes test covering all supported GGML types including MXFP4, NVFP4, Q1_0, and Q2_0

Batch Generation Error Resilience

  • Author: William Kennedy
  • Added error handling to all batch.Add() calls across batch generation engine
  • MTP draft generation now returns errors properly through the drafter interface
  • Speculative decoding batch operations handle errors with proper slot cleanup
  • Prefill text and media operations rollback batch state on errors and finish slots cleanly
  • Batch sequence engine for embed/rerank models handles add errors

Yzma / llama.cpp Update

  • Author: William Kennedy
  • Updated default llama.cpp version from b10417 to b10423
  • Updated GGML type block sizes for Q2_K, Q8_K, IQ2_XS, IQ3_XXS, and IQ4_XS to match upstream
  • Added GGML type definitions for MXFP4 (39), NVFP4 (40), Q1_0 (41), and Q2_0 (42)

Documentation

Model Configuration Docs

  • Author: William Kennedy
  • Updated Chapter 3 (Model Configuration) with new load mode options
  • Updated Chapter 17 (Troubleshooting) with relevant changes
  • Updated BUI documentation components to reflect new config options

Dependencies

Dependency llama.cpp Update

  • Updated default llama.cpp version from b10417 to b10423
  • Uses new download.Install API instead of download.GetWithContext

Statistics

Category Count
Total Commits 1
New Features 0
Improvements 5
Bug Fixes 1
Documentation 1

Contributors:

  • William Kennedy

Upgrade Notes

Breaking Changes

None

Migration

No migration required. The default LoadMode changed from mmap to auto, but this maintains the same behavior when all devices support mmap. Explicit load-mode: mmap in config files will continue to work as before.

Recommended Actions

  • Review model configuration files if you have explicit load-mode settings — the default is now auto instead of mmap
  • Consider using load-mode: mmap+mlock if you want the combined mmap + mlock behavior that was previously not available as a single option
  • Update any code relying on LoadMode zero value being MMap — it is now Auto
Assets 15
Loading