Skip to content

CoderAI 0.2.21 — zero-config cluster, prefix-cache routing, metrics, signed images

Latest
Latest

Choose a tag to compare

View all tags
@nextime nextime released this 20 Sep 16:33
v0.2.21
36ae2ba

CoderAI 0.2.21 — zero-config cluster, prefix-cache routing, metrics, signed images

Complete orchestration, distribution and escalation of remotizable advanced inference.

Install from the packages

  • Linux: docker pull ghcr.io/nextime/coderai:0.2.21 then packaging/linux/run_oci.sh --nvidia -d
    (or --vulkan / --all; --host-network to find the other boxes over mDNS). Guide: docs/install-from-packages.md.
  • Windows (NVIDIA, Docker Desktop + WSL2): CoderAI-Setup-0.2.21.exe below, or
    irm https://raw.githubusercontent.com/nextime/coderai/master/packaging/windows/install-coderai.ps1 | iex in an elevated PowerShell.
  • Verify the image: cosign verify --key https://raw.githubusercontent.com/nextime/coderai/master/packaging/cosign.pub ghcr.io/nextime/coderai:0.2.21.
  • Offline: docker save ghcr.io/nextime/coderai:0.2.21 | gzip > coderai.tar.gz on a connected machine; docker load on the other side.
  • Capability images (one capability each, for pods, hosts and nodes), all signed: ghcr.io/nextime/coderai-{images,video,text,llama,vllm,engines,engines-kt,embeddings,ocr,ocr-paddle,tts,tts-xtts,stt,stt-nemo,stt-crisper,speaker,voice,audio,faceswap}:latest.

New

  • Zero-config cluster on a LAN: one shared cluster.token + mDNS discovery (_coderai._tcp); members auto-join as nodes, no URL and no per-node token. Nodes and engines accept the shared token; capability images join with CODERAI_CLUSTER_TOKEN / CODERAI_DISCOVERY=1.
  • Prefix-cache-aware routing: a conversation's next turn goes back to the engine or node that served it when the model is loaded in several places; several holders of a model are picked least-busy.
  • GET /metrics (Prometheus): requests and wall time per API key name / model / kind / status / engine, engines and nodes up, VRAM, in-flight, RPC servers, discovery, RunPod spend. Usage-by-key and by-model tables on the Cluster page.
  • Node logs from the head and re-push on return: a node that restarts gets its assigned models pushed again automatically.
  • Row split (llama.cpp tensor parallel) per model over local cards and RPC servers; vLLM tensor parallel may span nodes.
  • cosign-signed images: every GHCR image signed against packaging/cosign.pub.

Docs: docs/cluster.md · write-up: https://www.nexlab.net/articles/coderai-what-the-others-have/

Known limits: the multi-node paths are tested against fakes, two processes and a localhost rpc-server; mDNS was exercised with three announcers on one host. Nothing has yet crossed a real cable between two GPU machines. The Windows installer is unsigned and has not been run on a Windows machine.

Assets 5
Loading