Skip to content

v0.1.8 — Multi-Slot Multi-Agent, Slot Persistence, Switch-Mode Drift Fix

Latest
Latest

Choose a tag to compare

View all tags
@jaigouk jaigouk released this 05 Jun 10:35
· 21 commits to main since this release
v0.1.8
a076e45

This release lands the multi-slot multi-agent capability for hermes-agent (132 TPS aggregate at 3 concurrent agents on a 24 GB RTX 4090), per-slot KV-cache persistence so an agent can disconnect and resume without a full re-prefill, a drift-recovery fix for mode switch, comprehensive doc refresh + 8 new tutorials, CVE-clearing dep upgrades, and a wheel-build fix.

Re-tagged 2026-06-05 to include the wheel-build fix (drop redundant hatch force-include for compat/profiles). The initial v0.1.8 tag's release-day build failed in GitHub Actions; this re-tag includes commit 7eb8522 so uv build produces a clean wheel.

Highlights

  • Multi-agent with cont-batching (gpumod-8xaq) — new preset gemma4-26b-a4b-q4-multi runs Gemma 4 26B-A4B with --parallel 3 --cont-batching at 128K ctx per slot. Phase 2 measured 132 TPS aggregate vs 109 TPS single-slot. hermes-agent and code modes both swapped to it. (research)
  • Slot persistence (gpumod-8viu) — --slot-save-path enables POST /slots/{id}?action=save|restore for per-agent KV cache that survives disconnects. 3.8 ms save / 2.3 ms restore for 41 cached tokens; 3× prompt-eval speedup confirmed. (research)
  • mode switch drift recovery (gpumod-hrgg) — partitions running-or-sleeping into actively-running vs sleeping so to_start reconciles against actual systemd state rather than the DB record. A previously-silent no-op when DB == target but services drifted is now a clean restart.

What's New

  • Tutorials (gpumod-01lv) — 8 new pages on the public docs site:
    • Getting Started: Run your first service, Pick a mode for your workload
    • User Guide: Multi-agent with cont-batching, OpenAI-compatible clients, Modes deep dive, Host stability and gpumod doctor, Vision-enabled workflows, Benchmark your model
  • Reference doc refresh (gpumod-kk2j) — full rewrite of docs/internal/presets.md (40-preset catalog, multi-slot patterns, defense flags, vision vs text). Architecture, CLI reference, MCP workflows aligned.
  • Integration test (gpumod-hwbb) — tests/integration/test_switch_mode_partition.py exercises the real Registry → driver.status → partition path against a stubbed systemd backend (5 R-scenarios, 10 tests).
  • CVE-clearing dep upgrade — authlib 1.6.7→1.7.2, fastmcp 3.0.1→3.4.0, pyjwt 2.11.0→2.13.0, urllib3 2.6.3→2.7.0, pyasn1 0.6.2→0.6.3, python-multipart 0.0.22→0.0.32. Clears 10 Trivy HIGH/CRITICAL findings.
  • Build fix — removed a redundant tool.hatch.build.targets.wheel.force-include for compat/profiles that conflicted with the packages = ["src/gpumod"] directive on recent hatchling versions. Profile YAMLs are still shipped (via the packages directive).
  • Two new env vars — GPUMOD_MODES_DIR, GPUMOD_QUIESCE_SECONDS.

Important warnings (read before changing presets)

  • Never add --swa-full to a Gemma 4 multi-slot preset. Forces every layer to allocate full-ctx KV instead of Gemma's 1024-token sliding window; blows past the 24 GiB VRAM ceiling during cache allocation and triggers a silent host freeze (gpumod-8viu v1 incident, 2026-06-05 — hard reboot required). Slot save/restore works correctly without it.
  • Do not raise per-slot context above 128K on Gemma 4 26B-A4B. Phase 2 measured a continuous throughput cliff: 200K → 79 TPS, 256K → 10 TPS (Dev never finishes a turn in 600 s) even though VRAM still fits.

Upgrade

git pull --rebase
uv sync
gpumod mode sync
gpumod template install-all --yes

One-time host setup if you plan to use slot persistence:

mkdir -p ~/.cache/llama-slots

Quality

  • 2,721 unit + integration tests pass (94.36% coverage)
  • ruff check, ruff format --check, mypy --strict clean
  • mkdocs build --strict clean
  • uv build produces a clean wheel (gpumod-0.1.8-py3-none-any.whl, 119 files)
  • Trivy: 0 HIGH/CRITICAL vulns, 0 secrets, 0 misconfigurations

Ticket index

Ticket Type Title
gpumod-8xaq spike + feature Multi-agent capacity spike + N=3 cont-batching preset
gpumod-8viu spike + feature --slot-save-path + client wrapper pattern
gpumod-hrgg bug mode switch reports success but service doesn't start
gpumod-kk2j task Doc refresh — presets, architecture, cli, mcp-workflows
gpumod-hwbb task Integration test for switch_mode partition
gpumod-01lv task Author missing tutorials for getting-started + user-guide

Full diff: v0.1.7...v0.1.8

Assets 2
Loading