Skip to content

v1.31.2

Choose a tag to compare

View all tags
@ardan-bkennedy ardan-bkennedy released this 16 Aug 18:29
· 35 commits to main since this release
v1.31.2
8449fce
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release Notes - v1.31.2

Release Date: August 16, 2026

Overview

Version 1.31.2 brings a major rework of Kronk's batch scheduling engine with new diagnostics, a single prefill-batch-size configuration setting replacing nbatch/nubatch, a Slots UI for real-time scheduler visibility, a new media load testing tool, and a BREAKING change to model batch configuration.

Detailed Changes

New Features

Batch Engine Diagnostics API

  • Author: William Kennedy
  • New BatchEngineSnapshot type exposes real-time scheduler state per loaded model
  • Each snapshot includes per-slot phase, generation mode, token counts, and IMC preparation progress
  • Generation contributions track each slot's row count and mode (ordinary, speculative, MTP, M-RoPE direct)
  • Prefill and IMC selector cursors show which slots are eligible and currently selected
  • Available via Model.BatchEngineSnapshot() in the SDK and GET /v1/kronk/models/{model}/slots on the server

Slots Browser UI

  • Author: William Kennedy
  • New Slots tab in the BUI displays live scheduler state for all loaded generation models
  • Shows per-slot phase (idle, starting, prefill, media-prefill, IMC preparation, generation), token counts, request age, and generation mode
  • Model-level summary includes iteration count, batch sizing, MTP/speculative mode, and selector cursors
  • Color-coded phase badges and generation mode labels for quick visual scanning
  • Auto-refreshes every 2 seconds to keep scheduler state current

Media Load Testing Tool

  • Author: William Kennedy
  • New .tools/media-load/media-load.py for benchmarking Kronk with multimodal media inputs
  • Supports parallel clients, configurable image/audio payloads, and streaming responses
  • Generates throughput and latency metrics for vision and audio model workloads

SDK Batch Engine Snapshot

  • Author: William Kennedy
  • kronk.(*Kronk).BatchEngineSnapshot() exposes the latest scheduler state from the generation batch engine
  • Returns (BatchEngineSnapshot, bool) — the bool is false for embedding/rerank models
  • Complements IMCSessions() for complete runtime observability

Improvements

Single Prefill Batch Size Configuration

  • Author: William Kennedy
  • Replaced separate nbatch and nubatch settings with one prefill-batch-size (default: 2048)
  • prefill-batch-size limits how many prompt tokens the current prefill owner contributes per decode iteration
  • Kronk derives internal NBatch and NUBatch capacities from prefill-batch-size, slot count, and generation mode
  • Non-MTP reserves one generation row per slot; MTP reserves 1 + ndraft rows per slot

Round-Robin Prefill Replaced by Prefill Owner

  • Author: William Kennedy
  • Prefill now uses a persistent owner model instead of round-robin across slots
  • One active slot owns prefill until its prompt completes, then the cursor advances to the next eligible slot
  • Generation and speculative rows from ready output slots are staged first, then the remaining tray capacity goes to the prefill owner
  • This gets long-prompt prefill done quickly without blocking streaming output from other slots

IMC Preparation Scheduling

  • Author: William Kennedy
  • Text IMC preparation is deferred to its own scheduling phase after slot admission
  • Newly admitted requests can claim free slots immediately without waiting for IMC decoding
  • If no rows were staged during admission, IMC preparation runs before prefill

Configuration Cleanup

  • Author: William Kennedy
  • Minor cleanup to configuration handling and edge cases
  • Removed unused nbatch/nubatch references from config files and examples

Documentation

Manual Updates

  • Author: William Kennedy
  • Updated Chapter 4 (Batch Processing) with new prefill owner model, batch sizing diagrams, and selector cursor explanations
  • Updated Chapter 3 (Model Configuration) with prefill-batch-size documentation
  • Updated Chapter 5 (Message Caching) with new IMC session selection diagram
  • Updated Chapter 6 (Speculative Decoding & MTP) with MTP batch sizing notes
  • Updated Chapter 11 (Multi-Modal Models) and Chapter 13 (Browser UI)
  • Added new SVG diagrams for batch sizing, prefill batching, IMC cache growth, and IMC session selection
  • Added BREAKING_CHANGES.md documenting the v1.31.2 batch configuration migration

SDK Documentation

  • Author: William Kennedy
  • Updated DocsSDKModel.tsx with new batch engine diagnostics API examples
  • Updated DocsSDKPool.tsx with pool-level snapshot references
  • Updated DocsSDKKronk.tsx with BatchEngineSnapshot() usage

Dependencies

Build Toolchain

  • Author: William Kennedy
  • Updated .make/tools.mk with toolchain improvements

Statistics

Category Count
Total Commits 2
New Features 4
Improvements 4
Bug Fixes 0
Documentation 3

Contributors:

  • William Kennedy

Upgrade Notes

Breaking Changes

Model Batch Configuration Changes

The separate user-configurable nbatch and nubatch settings were replaced by one prompt-processing setting:

# Before
nbatch: 8192
nubatch: 2048

# After
prefill-batch-size: 2048

prefill-batch-size defaults to 2048 and limits how many prompt tokens the current prefill owner contributes to one decode iteration. Kronk now derives llama.cpp's internal logical NBatch and physical NUBatch capacities from that value, the slot count, and the generation mode. Non-MTP reserves one generation row per slot. MTP reserves 1 + ndraft rows per slot and uses one physical batch for the complete prefill-plus-generation tray.

Direct Go SDK consumers must replace these removed APIs:

Removed Replacement
model.Config.PtrNBatch model.Config.PtrPrefillBatchSize
model.Config.PtrNUBatch model.Config.PtrPrefillBatchSize
model.Config.NBatch() model.Config.PrefillBatchSize() for configuration; EffectiveNBatch() for diagnostics
model.Config.NUBatch() model.Config.PrefillBatchSize() for configuration; EffectiveNUBatch() for diagnostics
model.WithNBatch(...) model.WithPrefillBatchSize(...)
model.WithNUBatch(...) model.WithPrefillBatchSize(...)

The Playground request field changed from nbatch and nubatch to prefill_batch_size. The BUI playground and configuration sweeps now expose only Prefill Batch Size. Effective NBatch / NUBatch values remain visible as read-only runtime diagnostics on the Slots screen.

Migration

When migrating an old configuration with different values, use the old nubatch value as the initial prefill-batch-size, then benchmark the workload. A larger value can finish long-prompt prefill in fewer decode calls, but each call takes longer before already-generating slots can run again and requires a larger compute buffer.

Recommended Actions

  • Replace nbatch and nubatch YAML settings with prefill-batch-size in all configuration files
  • Update Go SDK code: replace PtrNBatch/PtrNUBatch with PtrPrefillBatchSize, replace WithNBatch/WithNUBatch with WithPrefillBatchSize
  • Replace model.Config.NBatch() calls with PrefillBatchSize() for config or EffectiveNBatch() for diagnostics
  • Replace model.Config.NUBatch() calls with PrefillBatchSize() for config or EffectiveNUBatch() for diagnostics
  • Update playground request payloads: replace nbatch/nubatch fields with prefill_batch_size
  • Explore the new Slots tab in the BUI for real-time scheduler visibility
  • Review BREAKING_CHANGES.md for complete migration details
Assets 15
Loading