Skip to content

v0.5.0.0

Choose a tag to compare

View all tags
@matthewdcage matthewdcage released this 02 Aug 22:13
· 70 commits to main since this release
v0.5.0.0
7bd2f26

0.5.0.0: routing engine consolidation + request-aware pools

Minor release: one failover engine for all API surfaces, unified model matching, audit remediation (F-30…F-53), and request-aware model pools for heterogeneous LAN fleets. Includes everything in 0.4.5.1 (menubar status header fix).

Upgrade note: This release changes routing outcomes in documented cases (embeddings guard, Messages upstream errors, batch_shard failover, pool overflow). Read the behaviour sections below before upgrading a live mesh.

Structural (invisible in normal use)

  • One routing engine — five duplicated proxy loops replaced by service/engine.py + four surface adapters; AttemptRecorder is the sole accounting path (F-24).
  • One model matcher — ModelResolver answers candidacy and invocation with the same algorithm (F-25).
  • Monolith split — service.py → 16 modules; CLI main.py → 80 lines + command subpackage (F-26).
  • Contract suite — 356 golden-vector tests gate routing behaviour; CI erosion check on engine.py.
  • Audit remediation (PR #41) — Anthropic streaming fix (F-30), streaming pre-flight errors (F-32), streaming telemetry parity (F-33), OpenAI/Anthropic error envelopes on /v1/*, SDK payload hardening, read gates on /metrics and /netllm/v1/client-env.

Behaviour you may notice

Embeddings guard (D4)

/v1/embeddings now returns 400 immediately for models classified as chat (name-heuristic). Unknown model names default to chat. Use [routing.model_aliases] with an embedding token in the alias if your encoder name is non-standard.

Messages upstream errors (D11)

/v1/messages forwards upstream 400/404 instead of flattening everything to 502 when backends are OpenAI-translated.

Streamed Messages fixes (D9)

  • message_start carries the requested model name (not the backend-resolved name).
  • Mid-stream errors terminate with message_stop.

Responses telemetry (D16)

Streamed /v1/responses (Codex CLI) now records success accounting — request counts, tokens, and per-backend routed_requests match chat.

batch_shard failover (D17)

When a shard-assigned backend fails, failover restarts at the candidate list head (more attempts, better recovery for shards on non-first backends).

Shard context on all surfaces (D5)

Shard headers apply to embeddings and Messages, not only chat.

Retry budget (D7)

Attempt cap includes injected cloud rows and Messages fallback tiers (bounded, not unbounded).

Request-aware model pools (D19)

Heterogeneous pools now match the requested model literally first; pool substitution to another allowlist model runs only when no pool member serves that name. Agent-hop requests skip pool substitution (exact_model_only). Prevents a local catch-all host from stealing traffic when another peer serves the requested model.

Provider payload adaptation

SDK-unknown params (top_k, etc.) are adapted at the OpenAI SDK layer for chat and embeddings; agent-hop terminating peers strip unknown fields consistently.

Telemetry / dashboard

  • docs/telemetry-api.md is normative; dashboard reads server total_tokens directly.
  • tests/contract/test_telemetry_contract.py gates documented keys.

Known issues (unchanged or filed)

  • F-38 partial — error envelopes fixed for HTTPException paths; malformed JSON still returns raw 500; upstream 401/429 on OpenAI routes may still map to 502.
  • F-57 — capability-guard 400 may echo post-rewrite model names from model_rewrites.
  • F-54 — dashboard.js remains a single file (no module split in this release).

Full operator detail: architecture/refactor/RELEASE-NOTES.md.

macOS (0.4.5.1 fix included)

Menubar status header reads live server.state — no more Agent stopped with Stop Agent visible after adopt.

Upgrade

macOS (notarized DMG when available):

git clone https://github.com/matthewdcage/llm-swarm-router.git
cd llm-swarm-router && git checkout v0.5.0.0
uv sync && uv pip install venvstacks
apps/netllm-mac/Scripts/build.sh release
packaging/scripts/macos-app-install.sh --source apps/netllm-mac/build/Stage/llm-swarm-router.app

Linux / Windows / CLI-only:

git checkout v0.5.0.0
uv sync
./netllm restart   # or systemctl --user restart netllm

Verify:

./netllm doctor
./netllm test
curl -sf http://127.0.0.1:11400/health
scripts/live-routing-smoke.sh   # maintainer LAN mesh (optional)

Mixed model pools: confirm [routing.model_pools.*] hosts lists every participating agent; run ./netllm status on each node after upgrade.

Test plan (release gate)

  • uv run pytest -q — full suite
  • uv run pytest tests/contract -q — 356 golden vectors
  • ./scripts/ci.sh lint — includes engine erosion gate
  • ./scripts/verify-before-pr.sh — macOS menubar when applicable
Assets 9
Loading