CoderAI 0.2.21 — zero-config cluster, prefix-cache routing, metrics, signed images

CoderAI 0.2.21 — zero-config cluster, prefix-cache routing, metrics, signed images

Complete orchestration, distribution and escalation of remotizable advanced inference.

Install from the packages

• Linux: docker pull ghcr.io/nextime/coderai:0.2.21 then packaging/linux/run_oci.sh --nvidia -d
  (or --vulkan / --all; --host-network to find the other boxes over mDNS). Guide: docs/install-from-packages.md.
• Windows (NVIDIA, Docker Desktop + WSL2): CoderAI-Setup-0.2.21.exe below, or
  irm https://raw.githubusercontent.com/nextime/coderai/master/packaging/windows/install-coderai.ps1 | iex in an elevated PowerShell.
• Verify the image: cosign verify --key https://raw.githubusercontent.com/nextime/coderai/master/packaging/cosign.pub ghcr.io/nextime/coderai:0.2.21.
• Offline: docker save ghcr.io/nextime/coderai:0.2.21 | gzip > coderai.tar.gz on a connected machine; docker load on the other side.
• Capability images (one capability each, for pods, hosts and nodes), all signed: ghcr.io/nextime/coderai-{images,video,text,llama,vllm,engines,engines-kt,embeddings,ocr,ocr-paddle,tts,tts-xtts,stt,stt-nemo,stt-crisper,speaker,voice,audio,faceswap}:latest.

New

• Zero-config cluster on a LAN: one shared cluster.token + mDNS discovery (_coderai._tcp); members auto-join as nodes, no URL and no per-node token. Nodes and engines accept the shared token; capability images join with CODERAI_CLUSTER_TOKEN / CODERAI_DISCOVERY=1.
• Prefix-cache-aware routing: a conversation's next turn goes back to the engine or node that served it when the model is loaded in several places; several holders of a model are picked least-busy.
• GET /metrics (Prometheus): requests and wall time per API key name / model / kind / status / engine, engines and nodes up, VRAM, in-flight, RPC servers, discovery, RunPod spend. Usage-by-key and by-model tables on the Cluster page.
• Node logs from the head and re-push on return: a node that restarts gets its assigned models pushed again automatically.
• Row split (llama.cpp tensor parallel) per model over local cards and RPC servers; vLLM tensor parallel may span nodes.
• cosign-signed images: every GHCR image signed against packaging/cosign.pub.

Docs: docs/cluster.md · write-up: https://www.nexlab.net/articles/coderai-what-the-others-have/

Known limits: the multi-node paths are tested against fakes, two processes and a localhost rpc-server; mDNS was exercised with three announcers on one host. Nothing has yet crossed a real cable between two GPU machines. The Windows installer is unsigned and has not been run on a Windows machine.

CoderAI 0.2.20 — several machines as one

CoderAI 0.2.20 — several machines as one

Complete orchestration, distribution and escalation of remotizable advanced inference.

Install from the packages

• Linux: docker pull ghcr.io/nextime/coderai:0.2.20 then packaging/linux/run_oci.sh --nvidia -d
  (or --vulkan / --all). Guide: docs/install-from-packages.md.
• Windows (NVIDIA, Docker Desktop + WSL2): CoderAI-Setup-0.2.20.exe below, or
  irm https://raw.githubusercontent.com/nextime/coderai/master/packaging/windows/install-coderai.ps1 | iex in an elevated PowerShell.
• Offline: docker pull the image on a connected machine and docker save ghcr.io/nextime/coderai:0.2.20 | gzip > coderai.tar.gz; docker load it on the other side.
• Capability images (one capability each, for pods, hosts and nodes): ghcr.io/nextime/coderai-{images,video,text,llama,vllm,engines,engines-kt,embeddings,ocr,ocr-paddle,tts,tts-xtts,stt,stt-nemo,stt-crisper,speaker,voice,audio,faceswap}:latest.

New

• Cluster nodes: other CoderAI installs as engines of one front — polled, assigned models, pinned by name, routed by capability and load; Load/Unload from the head; live add/remove; /admin/cluster page.
• One GGUF over several machines: llama.cpp RPC — rpc-server processes managed by the front, per-model Cards on other machines; the bundled llama-cpp-python now carries the RPC backend (sm_80/86/89/90/120) and rpc-server from the same tree.
• vLLM on Ray / SGLang multi-node: per-model parallelism and node commands; the coderai-vllm image joins as a Ray worker.
• Work fan-out for every generation capability: n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches — split over every engine and node that has the model, merged back.
• Pipeline parts on other machines (video): text encoder, the Wan 2.2 low-noise expert and the VAE each on a named node.
• LoRA / QLoRA training over several machines: synchronous data parallel over torch.distributed, adapter gradients averaged every step.
• Pools of capability remotes and host machines with failover.
• No impossible pairs: the model form greys out engine/backend combinations that cannot work and refuses to save one.
• Direct TCP + pod TLS by default for CoderAI pod images; nineteen capability images on GHCR (from 0.2.18/0.2.19).

Docs: docs/cluster.md · write-up: https://www.nexlab.net/articles/coderai-cluster-several-machines-as-one/

Known limits: nodes, RPC, fan-out and DDP are tested against fakes, two processes and a localhost rpc-server; nothing has yet crossed a real cable between two GPU machines. The Windows installer is built and unsigned; it has not been run on a Windows machine yet.