v1.1.0 — Unified Engine: Recurrent Rollback MTP, QSA Key Cache & 250k Record

haloq38flash v1.1.0 — Unified Engine: Recurrent Rollback MTP, QSA Key Cache & 250k Context Breakthrough

Qwen3.8-Flash-Next (125B-A6B) on AMD Strix Halo (Ryzen AI MAX+ 395 / Radeon 8060S)

This milestone release introduces the Unified Engine (v075-unified), solving the long-standing MTP draft serialization bottleneck with in-place recurrent state rollback, enabling Unsloth shared MTP heads, integrating Nathan Wilson's QSA pooled key cache, and proving continuous 250,000-token prompt ingestion in under 15 minutes within a strict 96 GB memory budget.

---

Highlights & Key Achievements

1. MTP Recurrent State Rollback (PR #28123 + PR #26)

• Root Cause Solved: Previously, LLM_ARCH_QWEN4EXP did not allocate recurrent rollback slots (need_n_rs_seq() == 0). On every rejected draft token, the engine had to serialize the entire 48-layer recurrent state (112 MB) to host memory, cutting speculative decoding throughput in half down to 15.5 t/s.
• In-Place Rollback Slots: Enabled recurrent rollback for Qwen4exp and updated build_conv_state_at() to maintain rolling convolutional history across all $n_{\text{slots}} = n_{\text{rs_seq}} + 1$ rollback slots in VRAM.
• +91% Decode Acceleration:
  • Prose Prompt generation: 29.6 t/s (vs 15.5 t/s baseline, a +91% speedup!).
  • 32k context with MTP: 500.4 t/s prefill | 27.1 t/s generation.

2. Unsloth Shared MTP Head Support

• Fully qualified loader for mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (2.6 GB).
• Automatically borrows token_embd and output weights directly from the loaded base model in VRAM, eliminating 1.54 GB of redundant parameter duplication.

3. Nathan Wilson's QSA Pooled Key Cache & Microbatch 2048 Tuning

• Eliminates $O(N)$ key recomputation during long-context ingestion.
• Combined with -ub 2048, dispatch matches the GGML_VK_MMID_M128 shader tile geometry (64 tokens per activated MoE expert), boosting 32k prompt processing to 544.4 t/s.

4. Record 250,000-Token Continuous Benchmark

• 250k Prompt Ingestion: Completed in 15.0–15.8 minutes at 262.9–277.3 t/s (down from >23 minutes baseline).
• Zero Watchdog Resets: GGML_VK_MAX_MB_PER_SUBMIT=2048 cleanly bounds Vulkan queue submissions to 2 GiB chunks, eliminating kernel queue lockup timeouts.
• Peak RAM Compliance: Peak resident memory remained strictly at 84.05 GiB (leaving 40.88 GiB of free system headroom), easily within the 96 GB target.
• Zen 5 CCX0 Affinity: Pinned execution via taskset -c 0-7 avoids cross-CCX interconnect latency and cache thrashing.

---

Benchmark Comparison

Metric	Baseline (ad914eb)	Tuned Candidate (5f851647f)	Unified Engine (v075-unified)	Improvement
32k Prefill (dd)	331.4 t/s	434.8 t/s	544.4 t/s	+64.3%
32k MTP Decode (tg)	25.5 t/s	26.7 t/s	27.1–29.6 t/s	+16.1%
250k Prefill (dd)	179.0 t/s	211.2 t/s	277.3 t/s	+54.9%
250k Ingestion Time	23.8 min	20.8 min	15.0 min	Saves 8.8 minutes
Peak RAM at 256k	94.1 GB	94.1 GB	84.1 GB	10.0 GB saved
MTP Drafter RAM	3.9 GB	3.9 GB	2.6 GB (shared)	1.3 GB saved

---

Included Release Assets

• haloq38flash-unified-engine-x86_64.tar.gz: Precompiled release binaries (llama-cli, llama-server, and Vulkan shared libraries).
• strix-halo-radv-driver.tar.gz: Bundled Mesa 26.1 RADV driver runtime for AMD Strix Halo gfx1151.
• 0001-mtp-recurrent-rollback-qwen4exp.patch: Git patch to apply recurrent rollback on upstream strix-llama.cpp.
• build-engine-unified.sh: One-command automated build script.
• launch-server-256k.sh: Production 256k server launcher with prompt caching and CCX0 affinity.
• launch-cli-256k.sh: Interactive 256k terminal CLI.
• launch-server-32k.sh: Low-latency 32k/40k daily driver server launcher.
• bench-unified-250k-ub2048.sh: Benchmark runner for full 250k context.
• Dockerfile & docker-compose.yml: Ready-to-deploy container recipes.

v1.0.0 — Qwen3.8-Flash-Next on AMD Strix Halo (256k Context & Prompt Caching)

haloq38flash v1.0.0 — 256k Context & Prompt Caching Release

Qwen3.8-Flash-Next (125B-A6B) on AMD Strix Halo (Ryzen AI MAX+ 395 / Radeon 8060S)

This milestone release validates extreme 256k context depth (-c 257024), delivers +18% prefill / +16% decode speedups, resolves the Vulkan command submission watchdog timeout, and operationalizes multi-turn server prompt caching within a strict 96 GB RAM budget.

---

Highlights & Key Achievements

• Qualified Candidate Engine (halo-box-strix-llama.cpp @ 5f851647f):

  • Full structural compatibility with MTP sidecar and SSD-PLE (-lm mmap -lzm on).
  • Zero perplexity degradation verified on wiki.test.raw: 2.5552 candidate vs 2.5519 reference ($\Delta = 0.0033$).
  • Upstream flag migration: --tensor-read-lazy on is now -lzm on (--lazy-mode on).

• Record 256k Context Benchmarks (-c 257024):

  • Daily Driver Baseline (ad914eb): 179.0 t/s prefill (dd) | 15.2 t/s decode (tg) (23.8 min prefill)
  • Tuned Engine (5f851647f): 211.2 t/s prefill (+18.0%) | 17.6 t/s decode (+15.8%) (20.8 min prefill)
  • Saves 3 full minutes of ingestion time at 256k with rock-solid stability and zero driver resets.

• AMDGPU Watchdog Timeout Root Cause & Resolution:

  • Fixed previous vk::Queue::submit: ErrorDeviceLost crashes caused by large dispatch command buffers exceeding the AMDGPU Linux kernel 10-second compute queue watchdog (amdgpu.lockup_timeout).
  • Setting GGML_VK_MAX_MB_PER_SUBMIT=2048 limits submission buffer chunks to 2 GiB, guaranteeing sub-2-second command buffers and 100% stability at 256k.
  • Setting RADV_PERFTEST=unified_heap improves VRAM/GTT memory management across Strix Halo's 128 GB unified LPDDR5X.

• Instant Multi-Turn Responses via Server Prompt Caching:

  • Validated llama-server with --cache-ram 8192 --ctx-checkpoints 32 --cache-prompt.
  • 32k context measured: Cold prefill 77.4 s $\to$ warm turn prefill 2.91 s (26.6× speedup).
  • 256k context projected: Cold prefill ~20.8 min $\to$ warm turn prefill < 3.0 s (> 400× speedup).
  • Strict Memory Budget Compliance:
    • Base weights (PLE on SSD): ~64.2 GB resident
    • MTP draft sidecar: ~3.9 GB resident
    • 256k KV cache (q8_0): ~18.0 GB resident
    • Dedicated RAM prompt cache: ~8.0 GB resident
    • Total Peak RAM: 94.1 GB (comfortably under the 96 GB limit, leaving ~34 GB free system memory).

• Production Tooling & Containerization:

  • scripts/build-engine-halobox.sh: Automated build script for the qualified Vulkan engine.
  • scripts/launch-server-256k.sh: One-click production server script for 256k context with prompt caching.
  • scripts/launch-cli-256k.sh: Interactive 256k terminal CLI session.
  • scripts/launch-server-32k.sh: Low-latency 32k/40k daily driver server.
  • Dockerfile & docker-compose.yml: Ready-to-deploy container setup with kisak-mesa Vulkan drivers.

---

Getting Started

# Clone the repository
git clone https://github.com/julianmb/haloq38flash.git
cd haloq38flash

# Build the qualified engine
./scripts/build-engine-halobox.sh

# Launch the production 256k server with prompt caching
./scripts/launch-server-256k.sh

Or deploy via Docker:

docker compose up -d

Model Checkpoints

• Base Model: julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF (Qwen3.8-Flash-Next-IQ4_XS-PLE.gguf)
• MTP Sidecar: mtp-Qwen3.8-Flash-Next-Q8_0.gguf