DGX Spark LLM Cluster

A complete setup kit for running distributed LLM inference on a 2-node NVIDIA DGX Spark cluster using Ray and vLLM.

What This Does

Turns two DGX Spark boxes into an LLM inference server with two operating modes:

• Split mode (default) — Independent vLLM instances (TP=1) on each node, load-balanced via proxy on port 9000
• Cluster mode — Ray tensor-parallel (TP=2) across both nodes, single API on port 8000
• OpenAI-compatible API — Drop-in replacement at http://spark-1:8000/v1 (or :9000 for load-balanced)
• Web chat interface — Open WebUI for interactive use
• AI agent & tool-calling support — Works with AI coding agents (Amplifier, etc.) via the OpenAI-compatible API with native tool calling
• Auto-start on boot — Systemd services with automatic recovery

Performance (split mode, Qwen3-Coder-Next-FP8): 0.3s short prompt latency, ~45 tok/s generation (from 11.6s / ~16 tok/s before optimizations).

Quick Start

Prerequisites

• 2x NVIDIA DGX Spark (or similar with GB10/GB200 GPUs)
• Ubuntu with NVIDIA drivers and Docker
• QSFP cable connecting the nodes
• Both nodes on the same LAN with SSH access between them

Installation

On spark-1 (head node):

git clone https://github.com/bkrabach/dgx-spark-cluster.git
cd dgx-spark-cluster
./install.sh --node-type head

On spark-2 (worker node):

git clone https://github.com/bkrabach/dgx-spark-cluster.git
cd dgx-spark-cluster
./install.sh --node-type worker

The installer auto-detects your username, IP addresses, and network interfaces. Override if needed:

./install.sh --head-ip 192.168.1.10 --worker-ip 192.168.1.11 --user myuser

Post-Install

Follow the instructions printed by the installer, then verify:

spark-check

Architecture

Split Mode (default)

┌─────────────────────────────────────┐     ┌─────────────────────────────────────┐
│           spark-1 (head)            │     │         spark-2 (worker)            │
│                                     │     │                                     │
│  ┌───────────────────────────┐      │     │   ┌───────────────────────────┐     │
│  │  vLLM Standalone :8000    │      │     │   │  vLLM Standalone :8000    │     │
│  └───────────────────────────┘      │     │   └───────────────────────────┘     │
│  ┌───────────────────────────┐      │ LAN │                                     │
│  │  Load-Balancing Proxy :9000│◄────╋─────╋──►(routes to both :8000 instances)  │
│  └───────────────────────────┘      │     │                                     │
│  ┌───────────────────────────┐      │     │                                     │
│  │     Open WebUI :8080      │      │     │                                     │
│  └───────────────────────────┘      │     │                                     │
└─────────────────────────────────────┘     └─────────────────────────────────────┘

Switch modes with sudo spark-mode split or sudo spark-mode cluster. Cluster mode uses Ray + QSFP 200Gb/s link for tensor-parallel traffic.

Usage

API Access

# Chat completion
curl http://spark-1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder-next",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# List available models
curl http://spark-1:8000/v1/models

Switch Models

sudo vllm-set-model Qwen/Qwen3-Coder-Next-FP8 2 131072

Operational Commands

Command	Description
spark-help	Quick reference
spark-check	Health check
spark-reset	Full cluster restart
journalctl -u vllm -f	View vLLM logs

Documentation

• INSTALL.md — Detailed installation guide
• docs/README.md — Operational reference
• docs/ARCHITECTURE.md — System design
• docs/TROUBLESHOOTING.md — Common issues

Key Details

GPU Memory

GB10 GPUs use unified memory shared with the system. Available GPU memory depends on what else is running. The default config uses --gpu-memory-utilization 0.85 which works when non-essential services are stopped. Reduce to 0.70-0.77 if running alongside other services.

Network

• LAN — Control plane, API access
• QSFP — 200Gb/s link-local for GPU tensor traffic (NCCL/UCX)

Services

Split mode (default):

Service	Node	Purpose
vllm-standalone	spark-1, spark-2	Independent vLLM instance (TP=1) per node
vllm-proxy	spark-1	Load-balancing proxy (:9000) across both nodes
open-webui	spark-1	Web chat interface

Cluster mode:

Service	Node	Purpose
vllm-ray-head	spark-1	Ray head in Docker
vllm	spark-1	vLLM model server (TP=2)
vllm-ray-worker	spark-2	Ray worker in Docker
open-webui	spark-1	Web chat interface

---

For AI Assistants

This repo is structured to be AI-friendly. If you're an AI assistant helping someone set up this cluster:

1. Run the installer — ./install.sh handles most configuration automatically
2. Check the context file — context/SYSTEM_CONTEXT.txt has operational details
3. Follow post-install steps — The installer prints what to do next
4. Verify with spark-check — Confirms all services are running

The installer uses templates with {{PLACEHOLDER}} syntax and auto-detects environment settings, so no manual file editing should be needed for standard setups.

License

MIT License — See LICENSE for details.