Skip to content

CoderAI 0.2.20 — several machines as one

Choose a tag to compare

View all tags
@nextime nextime released this 19 Sep 10:16
· 6 commits to master since this release
v0.2.20
70c3d0c

CoderAI 0.2.20 — several machines as one

Complete orchestration, distribution and escalation of remotizable advanced inference.

Install from the packages

  • Linux: docker pull ghcr.io/nextime/coderai:0.2.20 then packaging/linux/run_oci.sh --nvidia -d
    (or --vulkan / --all). Guide: docs/install-from-packages.md.
  • Windows (NVIDIA, Docker Desktop + WSL2): CoderAI-Setup-0.2.20.exe below, or
    irm https://raw.githubusercontent.com/nextime/coderai/master/packaging/windows/install-coderai.ps1 | iex in an elevated PowerShell.
  • Offline: docker pull the image on a connected machine and docker save ghcr.io/nextime/coderai:0.2.20 | gzip > coderai.tar.gz; docker load it on the other side.
  • Capability images (one capability each, for pods, hosts and nodes): ghcr.io/nextime/coderai-{images,video,text,llama,vllm,engines,engines-kt,embeddings,ocr,ocr-paddle,tts,tts-xtts,stt,stt-nemo,stt-crisper,speaker,voice,audio,faceswap}:latest.

New

  • Cluster nodes: other CoderAI installs as engines of one front — polled, assigned models, pinned by name, routed by capability and load; Load/Unload from the head; live add/remove; /admin/cluster page.
  • One GGUF over several machines: llama.cpp RPC — rpc-server processes managed by the front, per-model Cards on other machines; the bundled llama-cpp-python now carries the RPC backend (sm_80/86/89/90/120) and rpc-server from the same tree.
  • vLLM on Ray / SGLang multi-node: per-model parallelism and node commands; the coderai-vllm image joins as a Ray worker.
  • Work fan-out for every generation capability: n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches — split over every engine and node that has the model, merged back.
  • Pipeline parts on other machines (video): text encoder, the Wan 2.2 low-noise expert and the VAE each on a named node.
  • LoRA / QLoRA training over several machines: synchronous data parallel over torch.distributed, adapter gradients averaged every step.
  • Pools of capability remotes and host machines with failover.
  • No impossible pairs: the model form greys out engine/backend combinations that cannot work and refuses to save one.
  • Direct TCP + pod TLS by default for CoderAI pod images; nineteen capability images on GHCR (from 0.2.18/0.2.19).

Docs: docs/cluster.md · write-up: https://www.nexlab.net/articles/coderai-cluster-several-machines-as-one/

Known limits: nodes, RPC, fan-out and DDP are tested against fakes, two processes and a localhost rpc-server; nothing has yet crossed a real cable between two GPU machines. The Windows installer is built and unsigned; it has not been run on a Windows machine yet.

Assets 5
Loading