Expert Streaming Engine v0.2.0
This feature release makes ESE hardware-adaptive while preserving its defining
bounded NVMe → RAM → VRAM execution model.
Highlights
- Calibrates real CPU expert kernels and bounded expert uploads, then admits
heterogeneous CPU/GPU MoE execution only when model-backed verification
proves it beneficial and runtime telemetry remains within the measured
envelope. - Adds bounded double-buffered expert prefill streaming and adaptive
per-device expert caches for mixed Turing/Ampere systems. - Rebalances KV, expert-cache, MTP, and multimodal residency at idle safe
points through reversible transactions; failed preparation or publication
restores the exact prior physical and logical state. - Supports configurable concurrent dense-model sessions and concurrency-aware
Studio sweeps. MoE and hybrid modes remain single-session until equivalent
multi-sequence parity evidence exists. - Adds native Kimi Linear 48B-A3B GGUF execution with hybrid KDA/MLA layers,
256-expert top-8 routing, and bounded sidecar-backed expert caching. - Makes configuration sweeps execute a real inference workload during
capacity validation and persist the exact multi-GPU split that was tested.
Validated hardware
The retained local evidence covers an RTX 2080 SUPER (Turing), RTX 3060 Ti and
RTX 3080 (Ampere), individually and in a heterogeneous three-GPU topology.
Ada-or-newer physical validation remains an explicit solo-maintainer waiver;
the release does not claim untested architecture-specific performance.
The Kimi Linear MXFP4_MOE gate allocated 65,536 tokens and produced a
deterministic 9.29 tok/s bounded-cache result on two Ampere GPUs. A separate
Qwen3.8 27B Q4_K_M 65,536-context profile reached 520.91 prompt tok/s and
approximately 25 generation tok/s across the three-GPU topology after its
verified tensor split was corrected to 32,32,36.
Existing model folders, app profiles, chats, sweep checkpoints, and telemetry
consent are preserved when updating from v0.1.x.