MiniMax-M3 on 2 nodes

Run MiniMax-M3 (426B-parameter MoE, ~23B active, 128 experts, 1M-token context) across two NVIDIA DGX Spark (GB10) nodes using llama.cpp with tensor/layer split over RPC — including a working native tool-calling fix (the hybrid chat template) that you will not find upstream yet.

The full BF16/MXFP8 checkpoints are far too large for a single 128 GB unified- memory Spark (the official MiniMax-M3-MXFP8 is ~444 GB), so this guide uses a sub-4-bit GGUF quant split across two boxes. With the recipe below you get a stable OpenAI-compatible endpoint with reasoning and function-calling.

TL;DR of the hard part: llama.cpp PR #24523 adds preliminary M3 support but its tool-call parser cannot read M3's native format (HTTP 500). M2's template parses but corrupts M3's generation. The fix is a hybrid template (M3 native + M2's tool-call format). See The hybrid template.

What you get

Metric	Value (UD-IQ4_XS, 2× DGX Spark)
Model	MiniMax-M3 426B MoE, UD-IQ4_XS GGUF (~194 GiB, ~97 GiB/node)
Decode (tg)	~10.7 tok/s
Prefill (pp)	~590 tok/s @ `--ubatch-size 2048` (8k-token prompt)
Context	65,536 (configurable; KV q8_0 ≈ 45 KB/token)
Tool-calling	✅ native, structured `tool_calls` via the hybrid template
Reasoning	✅ `<mm:think>` separated into `reasoning_content`
First load	~13–25 min (RPC streams the worker's layers; cached after)

Hardware / software prerequisites

Two DGX Spark (GB10) nodes, or any two CUDA boxes whose combined unified/ VRAM memory comfortably exceeds the chosen quant (~194 GiB for UD-IQ4_XS, so ~97 GiB free per node).
A fast, dedicated link between the nodes (the Sparks have a 200 GbE RoCE/QSFP link). Do not run the RPC traffic over WiFi — on the Spark the .local hostname often resolves to WiFi; always use the fast-link IP (e.g. 10.20.20.2). Difference is ~25× (7 vs 183 MB/s).
Passwordless SSH from the head node to the worker (the run script ssh's in to start rpc-server).
CUDA toolkit + cmake + a recent GCC (the Sparks ship aarch64 / GCC 13).
docker (only used to drop_caches as root via nsenter; optional).
~250 GB free disk on the head node for the GGUF.
Python 3 with gguf importable (ships in llama.cpp/gguf-py) for the template tool.

Terminology in this repo: HEAD = node 1 (runs llama-server, owns the API); WORKER = node 2 (runs rpc-server only).

Architecture

                         user / OpenAI client
                                  │  http :8891
                                  ▼
   HEAD (DGX Spark #1)                         WORKER (DGX Spark #2)
   ┌───────────────────────────┐   RPC :50052  ┌────────────────────────┐
   │ llama-server              │◄─────────────►│ rpc-server             │
   │  - reads the GGUF         │  200 GbE RoCE │  - holds ~half the      │
   │  - layers 0..N/2 on GPU   │   10.20.20.x  │    layers (N/2..N)      │
   │  - KV cache, sampling     │               │  - compute for its half │
   │  - HTTP / tool parsing    │               │                         │
   └───────────────────────────┘               └────────────────────────┘

--split-mode layer puts a contiguous range of layers on each device, so there is exactly one cross-node transfer per micro-batch (at the split boundary). That is why a larger --ubatch-size helps prefill: fewer, bigger transfers.

1. Build llama.cpp (both nodes)

Use the helper:

git clone https://github.com/<you>/minimax3-on-2-nodes
cd minimax3-on-2-nodes
CUDA_ARCH=121 bash scripts/build-llama-cpp.sh      # 121 = GB10 / sm_121

What it does (and why it matters):

Fetches PR #24523 ("Add preliminary MiniMax-M3 support") into a branch.
Merges origin/master — this pulls in two fixes that are required for tool-calling to not hard-crash:
- #24329 chat: harden peg-native tool call parsing
- #24653 chat: fix grammar generator bug ("left recursion detected") The M3 commit only touches src/models/… and the chat fixes only touch common/chat*, so the merge is conflict-free.
Builds with -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121.

Run it on both nodes, or build on the HEAD and copy the worker bits:

# on the WORKER, if you copied instead of building:
mkdir -p ~/llama.cpp-bin
cp ~/llama.cpp/build/bin/rpc-server ~/llama.cpp-bin/
cp ~/llama.cpp/build/bin/lib*.so*  ~/llama.cpp-bin/     # libggml*, libllama*, …

⚠️ Worker RPATH gotcha. The rpc-server binary has an empty RPATH, so cd ~/llama.cpp-bin && ./rpc-server fails with libggml.so.0: cannot open shared object file even though the .so are right there. It must be launched with LD_LIBRARY_PATH=~/llama.cpp-bin. The run script does this automatically.

2. (Optional) EAGLE3 speculative decoding

templates/eagle3-optional.diff contains the fc_norm patch for the Inferact/MiniMax-M3-EAGLE3 drafter. In practice the draft acceptance rate is ~37–39% and yields no net speedup for this quant, so the baseline runs without it. Skip unless you want to experiment (DRAFT=1 ./run-minimax-m3.sh).

3. Download the model (HEAD node)

bash scripts/download-model.sh                 # UD-IQ4_XS by default
# optional unattended watchdog for flaky links:
DST=~/models/MiniMax-M3-GGUF/UD-IQ4_XS bash scripts/download-supervisor.sh

Why a custom downloader instead of hf download? The large shards (~49 GB) are Xet-backed; on some setups hf_xet stalls (process alive, ~0 bytes/min) and HF_HUB_DISABLE_XET=1 rejects shards > ~48 GB. download-model.sh resolves each file's HF resolve URL to the cas-bridge presigned URL (plain HTTP, supports byte ranges) and pulls each shard with 16 parallel range requests (~20 MB/s aggregate), with per-part resume and curl -f (so an expired presigned URL can never silently corrupt a shard).

Other quants are available in the repo (QUANT=UD-Q4_K_XL ./download-model.sh). UD-IQ4_XS is the sweet spot that fits two Sparks with headroom.

4. Pre-flight checklist (HEAD node)

No other big model running on either node (docker ps, pgrep vllm). M3 needs ~97 GiB/node; you have almost no headroom.
Disable any OOM killer before launch. On the Spark, earlyoom will SIGTERM llama-server mid-load:
```
sudo systemctl stop earlyoom        # re-enable later if you like
```
Fast link up: ping -c2 10.20.20.2 should be < 1 ms.

5. The hybrid chat template (THE key fix)

This is the part that makes native tool-calling actually work.

The problem. M3's native template emits tool calls with a namespace token ]<]minimax[>[ before every XML tag plus a recursive to_xml argument encoder. llama.cpp's auto-derived "peg-native" parser can't parse it:

HTTP 500  "Failed to parse input at pos N: <]minimax[>[<tool_call>…"
HTTP 500  "The model produced output that does not match the expected peg-native format"

The trap. Swapping in models/templates/MiniMax-M2.jinja makes the parser happy (M2 uses a simple <minimax:tool_call><invoke><parameter> format), but M2's template forces <think> in the generation prompt while M3 was trained on <mm:think>. Result: M3 gets a foreign cue and degrades — truncated answers ("2+2" → "Simple math question." then stop) and prose narration instead of tool calls.

The fix — a hybrid. Keep M3's native template unchanged (roles, <mm:think>, adaptive generation prompt, <response> tool-result format) and change only the tool-call format to M2's parser-friendly style. M3 obeys the in-context format and the parser returns structured tool_calls, while M3's behaviour stays native. Verified: single call, multi-turn (call→result→answer), and reasoning (no <mm:think> leak into content).

templates/MiniMax-M3-hybrid.jinja is ready to use. To regenerate it for a different/updated GGUF:

python3 tools/make-hybrid-template.py \
  --gguf ~/models/MiniMax-M3-GGUF/UD-IQ4_XS/MiniMax-M3-UD-IQ4_XS-00001-of-00006.gguf \
  --out  ~/MiniMax-M3-hybrid.jinja

The tool extracts the native template from the GGUF and applies three precise transplants (token defs, the tool instruction block, the tool_calls renderer). It asserts each block matched exactly once, so it fails loudly if a future template changes shape.

6. Configure and run (HEAD node)

cp templates/MiniMax-M3-hybrid.jinja ~/MiniMax-M3-hybrid.jinja

# edit the CONFIG block of scripts/run-minimax-m3.sh, or override via env:
WORKER_HOST=10.20.20.2 \
WORKER_USER=$USER \
MODEL=~/models/MiniMax-M3-GGUF/UD-IQ4_XS/MiniMax-M3-UD-IQ4_XS-00001-of-00006.gguf \
TEMPLATE=~/MiniMax-M3-hybrid.jinja \
bash scripts/run-minimax-m3.sh

The script: kills stale instances, (optionally) drops page cache on both nodes, starts rpc-server on the worker with the right LD_LIBRARY_PATH, waits for its port, then execs llama-server with the tuned flags. First load streams the worker's ~97 GiB over RPC (~13–25 min); subsequent loads are faster thanks to rpc-server -c tensor caching.

Key flags (already set):

Flag	Why
`--rpc <worker>:50052 --split-mode layer`	distribute layers across the 2 nodes
`-ngl 999`	all layers on GPU
`-fa on`	flash attention
`--ubatch-size 2048 --batch-size 2048`	+26% prefill vs default 512 (fewer cross-node transfers); negligible extra memory
`--parallel 1`	one slot, full ctx to a single request (raise for concurrency)
`--cache-type-k/v q8_0`	KV in q8_0 to fit memory
`--jinja --chat-template-file …hybrid.jinja`	the fix
`--temp 1.0 --top-p 0.95 --top-k 40`	MiniMax-recommended sampling

7. Verify

curl -s http://localhost:8891/health
# tool-calling + more tests:

See llm-test/README.md for ready-to-paste smoke tests (plain chat, single tool call, multi-turn) and notes for benchmark harnesses (concurrency vs --parallel, reasoning/reasoning_content, thinking_mode).

Performance & tuning

Prefill scales with --ubatch-size: 512 → ~469 tok/s, 1024 → ~551 (+18%), 2048 → ~592 (+26%). Diminishing returns past 2048 (and you'd need a bigger --batch-size), so 2048 is the chosen sweet spot. Cost in memory was negligible here (fitting params to device memory auto-sizes the buffer).
Decode (~10.7 tok/s) is bounded by the dual-node RPC serialization and the dense-attention fallback — M3's native sparse attention is not implemented in llama.cpp. --ubatch-size does not affect decode.
Prompt cache is on by default — repeated prefixes (agent loops) are nearly free on subsequent turns.
Context: KV q8_0 ≈ 45 KB/token → 64k ≈ 2.8 GiB, 128k ≈ 5.6 GiB per the KV budget. Raising ctx eats into your thin per-node headroom.

Troubleshooting

Symptom	Cause / fix
`libggml.so.0: cannot open shared object file` on the worker	rpc-server empty RPATH → launch with `LD_LIBRARY_PATH=~/llama.cpp-bin` (run script does this)
Tool calls return HTTP 500 (`peg-native` / `Failed to parse`)	you're not using the hybrid template, or llama.cpp lacks the master chat fixes — rebuild per §1 and pass `--chat-template-file …hybrid.jinja`
Model gives truncated / narrated answers, no tool calls	you used M2's template instead of the hybrid — regenerate with `make-hybrid-template.py`
llama-server killed mid-load	an OOM killer (`earlyoom`) — stop it before launch
`rpc-server did not come up`	wrong worker IP (WiFi vs fast link), ssh not passwordless, or libs missing on worker
`hf download` stalls at ~0 B/min on big shards	Xet stall — use `scripts/download-model.sh` (parallel cas-bridge curl)
Throughput drops / RDMA collapse after a reboot	clean second reboot of both nodes (Spark mlx5 WC + nvidia-peermem quirk)

Credits & references

llama.cpp — PR #24523 (MiniMax-M3), #16932 (generalized XML tool-call parsing, incl. MiniMax-M2), #24329, #24653.
unsloth/MiniMax-M3-GGUF — the dynamic GGUF quants used here.
MiniMax — the MiniMax-M3 model.

The hybrid-template approach and the dual-node recipe in this repo are the contribution: they make M3's native tool-calling work end-to-end on a two-node llama.cpp RPC setup without degrading the model's behaviour.

Navigation Menu

Repository files navigation

MiniMax-M3 on 2 nodes

What you get

Hardware / software prerequisites

Architecture

1. Build llama.cpp (both nodes)

2. (Optional) EAGLE3 speculative decoding

3. Download the model (HEAD node)

4. Pre-flight checklist (HEAD node)

5. The hybrid chat template (THE key fix)

6. Configure and run (HEAD node)

7. Verify

Performance & tuning

Troubleshooting

Credits & references

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
llm-test		llm-test
scripts		scripts
templates		templates
tools		tools
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

MiniMax-M3 on 2 nodes

What you get

Hardware / software prerequisites

Architecture

1. Build llama.cpp (both nodes)

2. (Optional) EAGLE3 speculative decoding

3. Download the model (HEAD node)

4. Pre-flight checklist (HEAD node)

5. The hybrid chat template (THE key fix)

6. Configure and run (HEAD node)

7. Verify

Performance & tuning

Troubleshooting

Credits & references

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages