3 x Ideas from oMLX

[ianscrivener]

_{copied from oMLX release notes}

Distributed Serving

Added experimental distributed serving across Macs. Models can be split with tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac with byte-identical output. A 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary.

Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with memory-aware contiguous-layer placement, short-lived worker enrollment, ConnectX discovery, and NCCL verification. The current compatibility path keeps physical workers in the outer MLX Ring; the hierarchical Ring-to-NCCL gateway remains future work.

Added SSD-backed prompt reuse for distributed ranks. Each rank stores a process-lifetime chain of cache-boundary snapshots and restores only prefixes available on every rank. Incremental segments keep storage linear: a 12K-token GLM-5.2 chain used 1.17 GB instead of roughly 6.5 GB with cumulative copies.

[fxd0h]

The SSD prompt-reuse one translates directly. mflux already keeps an in-memory prompt_cache on some models, and MemorySaver knows to evict the encoder when a prompt is warm in it. Persisting those embeddings to disk, keyed by encoder and prompt, would let a repeated prompt skip the encoder load entirely. Lens is the extreme case: a 20B encoder feeding a 3.8B DiT, and the embedding it hands over is a few MB. Image workflows repeat prompts constantly (same prompt, new seeds, another session), so the hit rate would be high.

Distributed has a different economy here than oMLX's 225GB MiniMax. Our checkpoints are several times smaller: even full-precision FLUX.2 fits a 128GB Mac, and on a 32GB Air quantization gets you there before a second machine does. What they describe (liveness supervision, shard staging, per-rank guards) is also a resident server, and mflux is a CLI, so if someone wants it anyway it looks like the mflux.* plugin case from #554. The Metal+CUDA pool presupposes the plain CUDA target we don't have yet.

[ianscrivener]

yes, IMO 'SSD prompt cache' is a worthwhile feature to add

CUDA: All the HF MFlux-Community Models (75) have been created on CUDA and soon I'll be doing demo images on CUDA as well. So I'll do a document of the CUDA gotchyas for MFlux... but TLDR; MFlux works on CUDA with some caveats

"Distributed" is perhaps a media medium-term feature possibility. IMO MFlux should work for "normal people's Macs"... not just the expensive 32GB+ models. Many normal people have old macks lying around the house.

Use case: I have a 36GB Mac Studio, 16Gb MacBook Pro and a CUDA machine. In a perfect world, I'd share the inference load across my 3 GPU computes. Even MFlux's distribute on Mac silicon only would be good!