Repository navigation
Releases
v0.3.5
Upgrade note: every node needs bootstrap (or pin) once
The agent control plane now speaks TLS with a key pinned per node (below). A machine on 0.3.5 will not send the cluster token to an address it has no pinned key for. After upgrading, run omarchy-cluster bootstrap user@host for each node (it reinstalls the agent and pins its key), or omarchy-cluster pin user@host for a node whose agent is already on 0.3.5. The agent needs the openssl command on the node to make its key.
Changes
Security: TLS control plane with pinned node keys (#98, #102)
- The agent on :8025 serves TLS with its own self-signed key, made at
install-agent.bootstrapandjoinread the node's certificate over ssh and pin its SHA-256 next to the recorded address. Before any request is written, the client completes the handshake and compares the key. A machine that takes over a recorded address, or a plain-HTTP listener there, receives no request and no token. An address with no pin is sent nothing. - New
omarchy-cluster pin user@hostrecords the key of a node that is already installed. - ssh now always checks the host key (
StrictHostKeyChecking=yes, never a reused control socket) when it reads the node's certificate, because that connection decides which key is trusted. If the check fails,bootstrapsays to run a plain ssh once and verify the key. The pin file is replaced atomically. - Not covered: the gateway (:8020), the engine and the rank ring are still plain HTTP, so prompts and answers cross the LAN unencrypted. Every agent route also needs the cluster token (#87, #91).
Joining and leaving nodes (#86, #94)
omarchy-cluster join CANDIDATEtakes a name fromdiscover(oruser@host), refuses a machine that is not Omarchy Linux, installs the agent and journals the join.omarchy-cluster leave CANDIDATEundoes a join: stops and disables the agent, removes the peer's copy of the token, drops this machine's trust record and journals it. Run twice, the second run changes nothing. A discovered node with no address is refused by name.bootstrapsets up new nodes over ssh and installs Avahi where it is missing (#58).
Placement and planning (#69, #78, #79, #81, #99, #103)
- A whole model on one node is tried before any split (#69). A missing route and too little memory are separate NO PLAN reasons (#78).
- A host that does not hold the model sizes it from the hub (#80).
- Placement counts the free disk on each node's model volume, and credits what the node already holds. A full node no longer hides a plan on the others, and the refusal says "not enough free disk" when only disk blocks (#99, #103).
- A remote node with no reachable address is refused before any rank starts (#100). The whole model's engine is reached at its own node, never at loopback (#75).
Models (#51, #84, #101)
- The Hugging Face cache follows
HF_HUB_CACHEandHF_HOME, and the snapshot that loads is whatrefs/mainnames, else the newest complete one, not the first by name (#101).models pullraises a normal error withouthuggingface_hub. models listandmodels pullstates are pinned by tests and documented (#84).
Serving: concurrency, sampling, ring (#57, #82, #93, #70, #76, #77, #104)
- The gateway serves concurrent requests with a bounded slot limit; the k > 1 gate switches itself off below the measured floor, and
/statusreports the running count (#57, #93). - The gateway accepts
temperature,top_pandseed; MLX ranks sample with a seeded key on rank 0 (#70, #82). - Sampled decode runs at greedy speed (#104, fixes #97): the pick is Gumbel-max over the seeded key instead of a full-vocabulary categorical draw, and with
0 < top_p < 1the nucleus is taken within the 4096 likeliest tokens and renormalised there. That is an approximation, not full-vocabulary nucleus sampling, even when the whole nucleus would fit inside 4096; an omittedtop_portop_p = 1bypasses the cap entirely. Measured on one Mac through the shipped decode path, greedy 146 tok/s, sampled 142 tok/s, sampled with top_p 107 tok/s (was 30 and 17 before, 13x slower than greedy). - Ring addresses follow
--links, and ranks retry ring start while a peer is not listening (#76, #77).
Interfaces
omarchy-cluster chat, a terminal chat over the gateway, with a session journal (#59, #92); a native model browser and chat window (#66); the cluster as a browser page (#60); a 0 Gb/s direction is shown as 0.00 (#90); Thunderbolt and RDMA-capable routes are labelled (#61).
Packaging and receipts
- An Arch package,
packaging/PKGBUILD(#85). - A whole-against-split receipt with the headroom stated and pinned (#83).
Full history: v0.3.4...v0.3.5
v0.3.4
Changes
omarchy-cluster models listshows the models you can load from the Hugging Face cache.omarchy-cluster models pull <repo>downloads one into the same cache. Thenserve --model <repo>loads it without extra steps. The gateway offers both asGET /modelsandPOST /models/pull. Pulling needshuggingface_hub; serving nodes never import it (#51).- The benchmarks page gains a matched CUDA and Vulkan comparison on a DGX Spark. CUDA won prompt processing in both matched windows, by 1.32x and 2.17x. That backs the CUDA default for NVIDIA ranks (#44).
- The README gains an index of the sibling projects (#40).
omarchy-cluster --version, the agent health reply and the node facts report the package version again. Since 0.3.3 they still said 0.3.2.
Full history: v0.3.3...v0.3.4
v0.3.3
Changes
- A model cache with a missing snapshots folder is now reported as a model miss instead of raising an error. Six new tests cover the resolver (#35).
- The hub validates heartbeat requests, and its boundaries have offline tests (#36).
- Agents send the required hub heartbeat identity at startup, and the startup contract has tests (#37).
- README and user guides refreshed (#38).
- On Linux, ranks start with HK_SYSMEM set to the node's RAM. This fixes the 8 GB ceiling on a 16 GB node (#39).
- The Sushi engine returns an error object when the upstream server is unreachable or fails, instead of raising.
Full history: v0.3.2...v0.3.3
v0.3.2
This release fixes three gateway and rank bugs. It also adds scoring and diagnostics to the ranks and pins the test runners.
Gateway and ranks
A malformed completion request now gets HTTP 400. The body must be a JSON object, and max_tokens must convert to an integer. Before, the connection closed with no reply (#31).
Each rank request now returns when its own inference ends. Before, a finished response could wait behind later queued requests and pass the gateway timeout (#32). Model execution is still serial.
Model placement follows the model that refs/main names in a cached Hugging Face repo. Before, it picked the first snapshot by name. That could plan with an old model's layer count (#30).
Ranks and serving
A teacher-forced /score job returns the per-token negative log-likelihood from rank 0. Ranks read NAME=VALUE lines from OMARCHY_LIVE_ENV before each job. serve --no-fit-check prints OVERRIDE and goes on when a node holds more than the planner's 90 percent rule (#29). The rank load line now prints the real rank number and the memory in use after load.
Receipts
A Muse Spark receipt for the full 62 layers comes with a quickstart, run scripts and result summaries (#28).
Tests and test jobs
Gateway tests cover the health route and the HTTP error replies (#33). The test jobs use pinned OS versions and one pytest release on Python 3.9 and 3.13 (#34).
omarchy-cluster v0.3.1
omarchy-cluster 0.3.1 lets serve start the MLX engine on more than two ranks, on a node that runs more than one rank, and send a raw prompt. It is three pull requests since v0.3.0. None of it has run on the 62-layer Muse Spark Open model. The multi-rank paths have not run on real nodes.
What changed
- N-rank serve (#23).
serve --nodes a,b,c --split 25,21,16starts one MLX rank per node, in rank order, for any number of ranks. There is no links file and no placement search. Rank 0 is the engine and runs the last layers. A node must hold its layers in its free memory, orservestops before it starts a rank. --ring-ip NAME=IPsets the ring address of a node whose first interface is not the one its neighbours reach. A name that is not in--nodesis refused.--rank-env NAME=VALUE(repeatable) sets an environment variable on every rank, for exampleMLX_DISABLE_COMPILE=1.- A macOS rank now gets the same
PYTHONPATHrule as a Linux rank, so it can find an adapter path. - Several ranks on one node (#25). A node may appear more than once in
--nodes. Each of its ranks is its own process on its own ring port, counting up from 52100. The layers of all of a node's ranks must fit its free memory together. One Vulkan process maps about 60 GiB, so a large laptop needs two ranks for a model bigger than that. - Raw prompts (#24). A request to
/v1/completionswith"raw": trueand a stringpromptis tokenized as exactly that text, with no chat template and no added special tokens. Rank 0 sends the same ids to rank 1. Requests withoutrawbehave as before. This lets a client end a prompt on the Muse Spark Open answer channel, which the chat template cannot do.
What was measured
- The suite on the release commit: 132 passed, 1 skipped.
- Each pull request added tests. I ran the tests of #25 against the code before it, and both fail there. The author of #24 reports the same for its test, and I did not rerun that one.
- Nothing else is measured in this release. There is no speed or load number.
Known limits
- Not run on the 62-layer Muse Spark Open model.
- N-rank serve and two ranks per node have not run on real nodes. I did not check that two ring ports on one node work with the MLX ring backend.
- The planner still does not charge hybrid recurrent state.
- The Muse Spark adapter hook registers the class only.
Pull requests
omarchy-cluster v0.3.0
omarchy-cluster 0.3.0 adds a way to serve a model that no single machine can hold through llama.cpp RPC, a placement planner that charges sliding-window and hybrid models the KV cache they really keep, and a hook that registers the Muse Spark adapter of omarchy-mlx in every rank. It also fixes two route-pinning cases.
What was measured
- Muse Spark Open 816B-A42B, a 175.6 GB GGUF (
UD-IQ1_S), served as one endpoint across a Mac Studio M1 Ultra (gateway, CPU) and two Omarchy Linux MacBook Pros, a 14-inch M2 Max running two servers and a 16-inch M1 Max. Loading took 857 s. A 107-token answer decoded at 3.04 tokens per second. The first request took 72.3 s to the first token. Later requests took 25.5 to 26.8 s for 158 to 161 prompt tokens. A Python function the model returned passed 13 of 13 checks. This is the GGUF route through llama.cpp, not an MLX run, and each prompt had one greedy sample. The README quickstart reproduces the setup. Receipt:receipts/2026-10-16-muse-spark-llamacpp-rpc/. - DeepSeek-Coder-V2-Lite 4-bit on one 16-inch M1 Max, current omarchy-mlx build: a median of 15.73 tokens per second over three 22-token replies. On a 13-inch M1 the same model fails to load at 8.07 GB of MLX allocations with 5.09 GB of RAM still available, and
placewrongly reports that it fits. Receipt:receipts/2026-10-11-single-host-baseline/.
What changed
serve --engine llamacppcharges sliding-window layers their window of KV, window times the server slots plus 512 tokens (#17), and takes extra llama-server arguments with--llama-arg(#18). For Muse Spark Open the KV charge at 32,768 tokens falls from 8.32 GB to 3.79 GB, and at 131,072 tokens from 33.29 GB to 10.23 GB. These are sums from the model's header, not a measured run.- The MLX planner charges sliding-window layers
min(context, window + 2047)tokens (#19), and only the attention layers of a hybrid model (#16). For Qwen3.8-27B the KV charge at 32,768 tokens falls from 8.0 to 2.0 GiB. This is arithmetic from the model config, with no placement run. - Every rank registers the omarchy-mlx Muse Spark adapter before it loads a model when
OMARCHY_MLX_SERVEnames the omarchy-mlxservedirectory (#21). This registers the class only. A split Muse Spark run on MLX has not been measured. - A route whose reverse direction failed is never pinned (#13), and a stored one-way pinned route fails closed (#14).
Known limits
- A split is for models that do not fit on one machine. It is not faster than one machine that can hold the model.
- The Muse Spark numbers are for 1.56 bits per weight through llama.cpp, three short prompts, and a night when other jobs ran on the laptops.
- The planner does not charge the recurrent state of a hybrid model. The state was not measured.
omarchy-cluster 0.1.0
First release.
omarchy-cluster runs one MLX model across Apple Silicon Macs on macOS and on Omarchy Linux, and serves it as one OpenAI-compatible endpoint.
Measured on 2026-10-06, greedy decoding. Details and caveats are in the README.
- GLM-4.5-Air-4bit (106B parameters, 60.1 GB) on a Mac Studio M1 Ultra (macOS, Metal) plus a MacBook Pro M1 Max 64 GB (Omarchy, Vulkan), 31 + 15 layers over Thunderbolt: 0.87 tokens per second, and both machines agree on every token. The model does not fit on the 64 GB laptop.
- DeepSeek-Coder-V2-Lite-Instruct-4bit: Mac Studio alone 68.2 tok/s. Mac Studio plus the M1 Max laptop 46 to 48 tok/s. Two Omarchy laptops 1.80 tok/s, with text identical to one laptop alone.
A split is not faster than one machine that can hold the model. It is for models that do not fit on one machine.
What is in 0.1.0:
discover,probe,status,place,serve,stop: mDNS discovery, route measurement, a fit check, and a two-rank mlx-lm pipeline started through token-authenticated node agents.- A layer split chosen from each node's measured speed and memory, or set with
--split. - An OpenAI-compatible gateway on :8020, per-step timing on request, and a contention watch in
GET /status.
Known limits:
- Two ranks, one request at a time.
- Linux nodes run 4-bit and 8-bit MoE models; 5-bit and 6-bit fail on the omarchy-mlx Vulkan backend today.
probedoes not pin a route between two Linux nodes yet.placedoes not know the Vulkan memory limit on 16 GB machines.
Install: see the README.