Skip to content

v0.4.1.0 — embeddings endpoint, capability-aware routing, smarter retries

Choose a tag to compare

View all tags
@matthewdcage matthewdcage released this 11 Jun 10:20
· 189 commits to main since this release
v0.4.1.0
7a8f156

0.4.1.0: embeddings endpoint, capability-aware routing, smarter retries

Feature release driven by a live two-machine swarm log audit: the router now serves POST /v1/embeddings, refuses chat requests against encoder/TTS models, stops wasting retries on backends that already failed, fixes an in-flight counter leak, and cleans up multi-homed hosts in peer scans.

New

  • POST /v1/embeddings — OpenAI-compatible embeddings on the same base URL (http://<host>:11400/v1). Routes to whichever backend serves the embedding model (Ollama, oMLX, LM Studio, vLLM, or a LAN peer agent via agent-hop) with the same failover and spillover as chat. Anthropic-format backends are excluded (the Anthropic Messages API has no embeddings standard); Anthropic-wired clients use the OpenAI surface for embeddings.
  • Model capability classification — GET /v1/models entries carry a capability field (chat, embedding, audio, rerank, other). Chat completions and Messages requests against embedding/TTS/encoder models return a clear 400 with a /v1/embeddings hint instead of burning the retry budget on upstream tokenizer.chat_template errors.
  • Case-insensitive model resolution — clients sending differently-cased names (e.g. gemma-…-4bit vs the served …-4Bit) now route, with the served ID's exact casing forwarded upstream and the requested name restored in the response. Alias keys in [routing.model_aliases] also match case-insensitively.
  • LMSTUDIO_API_KEY — LM Studio instances with API authentication enabled are now supported via env var or [[routing.backends]] api_key / api_key_env; the dashboard doctor flags auth-gated backends (401/403) with the fix.

Fixed

  • Retry budget reaches healthy peers — failover retries now skip backends that already failed the current request instead of re-hitting the same broken local backend, so spillover reaches an untried LAN peer before returning 502.
  • In-flight counter leak — merge_backends updates local backend rows in place rather than replacing them, so requests completing across a scan refresh decrement the live row. Previously in_flight could ratchet upward forever under load, distorting local_spillover decisions.
  • Duplicate peers in subnet scan — multi-homed hosts (Wi-Fi + Ethernet, DHCP drift) answering on several IPs with one agent_id now collapse to a single row preferring their reported listen URL, with other IPs listed under also_reachable_at. The scan also flags this machine's own row (self), which the dashboard labels "this machine".

Upgrade

macOS (menubar, recommended on macOS 26+):

git clone https://github.com/matthewdcage/llm-swarm-router.git
cd llm-swarm-router && git checkout v0.4.1.0
uv sync && uv pip install venvstacks
apps/netllm-mac/Scripts/build.sh release
packaging/scripts/macos-app-install.sh --source apps/netllm-mac/build/Stage/llm-swarm-router.app

CLI only: uv sync && ./netllm init && ./netllm serve — dashboard at http://127.0.0.1:11400/ui/

Verify embeddings:

curl -s http://127.0.0.1:11400/v1/embeddings \
  -H "Authorization: Bearer netllm-local" \
  -H "Content-Type: application/json" \
  -d '{"model":"<embedding-model-from-/v1/models>","input":"hello swarm"}'

Full guides: macos-install.md · editor-integration.md

Assets 11
Loading