Skip to content

v0.4.0: The swarm promise — two machines, three commands

Choose a tag to compare

View all tags
@matthewdcage matthewdcage released this 11 Jun 03:34
· 196 commits to main since this release
v0.4.0
05a10be
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

0.4.0: The swarm promise — two machines, three commands

Install-and-run now means it: netllm init --swarm on one machine, paste the printed netllm join command on every other, and the mesh forms with authentication and automatic same-model load spreading. No hand-edited config, no silent non-meshing.

Guided swarm setup

  • netllm init --swarm / --single — one question on a TTY (scripts stay non-interactive); swarm mode binds 0.0.0.0:11400, auto-generates swarm.cluster_token, selects local_spillover, and prints the exact join command
  • netllm join URL --token T — validates reachability and the token (401-aware), detects open-swarm/token mismatches, rejects self-joins, writes LAN bind + token + static peer
  • netllm swarm-token [--rotate] — show or rotate the pairing token

Load spreading that follows load

  • New local_spillover strategy (swarm default): serve locally below routing.spillover_max_local_in_flight concurrent requests (default 2), spill to the least-loaded LAN peer above it, never hop when the peer is just as busy — peer load travels in heartbeats plus an own-hops ledger
  • Loop-guarded agent hops: forwards carry x-netllm-local-only, so a peer running any distributing strategy can never bounce a request back into the mesh; peers advertise only models they serve directly (no transitive catalog echo)

Mixed-provider fleets

  • [routing.model_aliases] maps one canonical name to per-provider IDs (oMLX vs Ollama vs LM Studio naming); the proxy rewrites the upstream payload per backend and restores the canonical name in responses and streams
  • Clear 404s: unknown models fail fast with the live catalog instead of spraying every backend

Discovery that explains itself

  • Loopback-bound agents show in netllm peers as found but unreachable with the exact rebind fix
  • LAN-bound agents auto-run a one-shot subnet scan when mDNS finds no peers within 10 s
  • netllm doctor prints per-platform firewall commands (UDP 5353 + TCP 11400 for firewalld, ufw, netsh, macOS)

Performance

  • Provider scans (with their 1-token diagnose probes) are TTL-cached instead of running on every proxied request, with a cache-stampede guard
  • Health probes run off the event loop only when a probe could actually fire — fixes cross-agent status-page probe stalls

Verify (two machines)

# Machine A
./netllm init --swarm && ./netllm serve
# Machine B — paste the printed command
./netllm join http://<machine-A-ip>:11400 --token <printed> && ./netllm serve
# Either machine
./netllm peers && ./netllm models --lan
curl -s http://<each-machine>:11400/metrics | rg netllm_requests_total

Both request counters climb under parallel same-model load.

Compatibility

  • Existing configs are untouched: local_first, loopback bind, and all v0.3 strategies behave exactly as before (locked by contract tests); v0.3.x peers can heartbeat into v0.4.0 swarms
  • Standing acceptance harness: tests/test_e2e_two_agents.py (two real agents over HTTP in CI)

Previous release

Assets 11
Loading