Skip to content

v0.6.1 — full cascade, vision/STT/generation tools, self-tuning flywheel, go-install fixed

Choose a tag to compare

View all tags
@dmmdea dmmdea released this 04 Jul 20:34
· 848 commits to main since this release
v0.6.1
bead81c

local-offload v0.6.1

The first public release since the initial drop — this tag brings the public repo up to the current private mainline (everything from the 0.4.x/0.5.x/0.6.0 internal cycles, plus the 2026-07-03 packaging & robustness patch).

Breaking / Upgrade

  • Module path fixed. The module is now github.com/dmmdea/local-offload, matching this repository, so go install finally works: 
    go install github.com/dmmdea/local-offload@latest
    If you previously installed under any older module path (local-offload-pp-cli, offload-harness), delete the old binary and reinstall with the command above. Import paths in forks must be updated to github.com/dmmdea/local-offload/....
  • Config search order is now: --config <path> > $LOCAL_OFFLOAD_CONFIG > ./config.json > ~/.local-offload/config.json > built-in defaults. If you relied on defaults being used despite a config.json sitting in the working directory, that file is now picked up.

Highlights

  • Full self-learning cascade — confidence-gated escalation (logprob class-mass margin + self-reported confidence), per-task conformal thresholds (calibrate), a logistic entry-tier router (train-router), health monitoring + circuit breakers (EWMA / Page-Hinkley / CUSUM, P95 timeouts), few-shot exemplars (optimize), and a kNN entry-tier prefilter — all trained offline and inference-free over the token ledger.
  • Shadow-labeling flywheel — optional capture of live calls, counterfactual replay through other tiers to manufacture training labels, and a rigorous adoption gate (paired-bootstrap CI) that only promotes a retrained artifact when it provably lowers error. The long-running MCP server hot-reloads nightly-retrained weights without a restart.
  • Vision tools — vqa, ocr, extract-image, assess-image on a local VLM tier (grammar-constrained, grounded).
  • Speech-to-text — transcribe (whisper.cpp; turbo + --hq high-quality model, VAD, SRT output) and video understanding — video-describe (frame sampling).
  • Local media generation (opt-in, single 8 GB GPU, one cross-process GPU slot): image (SDXL via ComfyUI), voice (Chatterbox TTS, zero-shot cloning), music (ACE-Step), video (Hunyuan 1.5 I2V — wired; quality renders want a bigger card), plus a dependency-free SVG data-viz kit (gauge / comparison-bar / chromatogram / icons).
  • Opt-in NVIDIA NIM remote tool (nim / offload_nim) — reach NVIDIA's hosted free-model catalog or a self-hosted NIM for the rare frontier-model task. Key from $NVIDIA_API_KEY env only (never a config field), never counted in the savings ledger; the local cascade is untouched.
  • 15 MCP tools over stdio + the same surface as a single-binary Go CLI.
  • Agent runtime library (internal/agent) — the sandboxed local-agent building blocks (policy cage, write cage, egress control, memory layer) used by the MCP server's agent path.

2026-07-03 packaging & robustness patch (ships in this tag)

  • Relative render-script paths resolve against the executable's directory, so generation tools work from any working directory.
  • Tilde expansion in all path-typed config fields; ./config.json joined the config search order.
  • Bad MCP tool arguments surface as a structured defer carrying the argument error (instead of a generic failure).
  • doctor diffs the live /v1/models roster against every configured model alias; models output is data-driven.
  • The ledger records each defer reason; ledger reports the top defer reasons.
  • The llama client's request budget is split from tier timeouts, and cold-model-swap timeouts no longer trip the circuit breaker.
  • Rune-safe context trimming (no more mid-UTF-8 cut points).
  • The savings ledger labels its estimate honestly: est_value_kept_local.

System requirements
OS Linux, macOS, Windows (agent sandbox cages are Linux-only; everything else is cross-platform)
Go 1.26+ (to build)
GPU NVIDIA, ~8 GB VRAM for the text + vision cascade
RAM / disk 32 GB+ system RAM and a fast SSD recommended (model weights, MoE CPU offload)
Serving a local llama.cpp server, ideally behind a model-swapper (llama-swap) on http://127.0.0.1:11436 with the Gemma-4 QAT family (E2B / E4B / 26B-A4B) — skill/scripts/detect.sh checks your box and skill/scripts/setup.sh installs the whole stack (both env-var driven)
Optional per feature ffmpeg on PATH (audio/video), a whisper.cpp server (STT), ComfyUI (image/music/video generation), Node.js (render scripts), $NVIDIA_API_KEY (hosted NIM tool only)

A defer ({"deferred": true, "reason": ...}) is a normal result, not an error — it means "do this one yourself."

Assets 3
Loading