spark-llm-scripts

Scripts, patches, and Dockerfiles for running local LLM inference servers on the NVIDIA DGX Spark.

Hardware target: NVIDIA DGX Spark (GB10 Superchip, SM_121 / SM_120 arch, 128 GB unified memory)

Quick start

# Serve by model name — picks the best available script automatically
./run.sh                       # defaults to qwen3-coder-next
./run.sh qwen3-coder-next      # Qwen3-Coder-Next-FP8 via v23 (~46-47 tok/s)
./run.sh qwen3-next            # Qwen3-Next-NVFP4 (~65-70 tok/s)
./run.sh qwen35-coder

# Or run a specific script directly
bash servers/qwen3-coder-next-fp8/run-v23.sh

The run scripts auto-build patched vLLM files on first launch and mount them into the container.

---

Repository structure

spark-llm-scripts/
├── patches/                   # vLLM source patches (see patches/README.md)
│   ├── apply.sh               # Apply patches to a running container
│   ├── build.sh               # Generate .build/<version>/ from originals + .patch files
│   ├── README.md
│   └── vllm/
│       ├── v23/               # Patches for avarok/dgx-vllm-nvfp4-kernel:v23
│       └── v11/               # Patches for avarok/vllm-dgx-spark:v11
├── servers/                   # Docker run scripts for each model/image combo
│   ├── qwen3-coder-next-fp8/
│   │   ├── run-v23.sh         # avarok/dgx-vllm-nvfp4-kernel:v23 (recommended)
│   │   └── run-v11.sh         # avarok/vllm-dgx-spark:v11
│   ├── qwen3-coder-next-nvfp4/
│   │   └── run.sh             # avarok/dgx-vllm-nvfp4-kernel:v23 + MTP speculative decoding
│   ├── qwen3-next-nvfp4/
│   │   └── run.sh             # nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4
│   └── qwen35-coder-nvfp4/
│       └── run.sh             # vincentzed-hf/Qwen3-Coder-Next-NVFP4 (v11)
├── images/
│   └── dgx-vllm-nvfp4-kernel/
│       └── v23/Dockerfile     # Custom image with all patches baked in (alternative to -v mounts)
├── docs/
│   ├── TOOL_CALL_BUGS.md           # Full root-cause analysis of the 3 tool-call bugs
│   ├── trtllm-rc6-container.md     # TRT-LLM 1.3.0rc6 container — discovery notes & patch details
│   └── trtllm-local-venv.md        # TRT-LLM 1.3.0rc6 — local venv setup (no container)
├── .build/                    # (gitignored) Generated patched files for volume mounting
└── .gitignore

---

How the patching works

The patching approach keeps patch diffs readable in git while not bloating the repo with full copies of patched source files:

1. patches/vllm/<version>/<file>.py — original file extracted from the Docker image (committed)
2. patches/vllm/<version>/<file>.patch — unified diff of the fix (committed)
3. .build/<version>/<file>.py — generated by patch -o output orig patchfile (gitignored)
4. -v .build/...:/container/path:ro — patched file volume-mounted read-only into the container

Run scripts call patches/build.sh automatically on first run.

To rebuild manually:

bash patches/build.sh all    # builds both v23 and v11
bash patches/build.sh v23    # builds v23 only

---

DGX Spark images

Image	vLLM path in container	Use case
avarok/dgx-vllm-nvfp4-kernel:v23	/app/vllm/vllm/	FP8 and NVFP4 models — recommended
avarok/vllm-dgx-spark:v11	/opt/venv/lib/python3.12/site-packages/vllm/	Older, fallback

⚠️ The vLLM install path differs between v23 and v11. A wrong mount path means patches are silently ignored. Verify the active path with:

docker exec <container> python3 -c "import inspect, vllm.tool_parsers.qwen3coder_tool_parser as m; print(inspect.getfile(m))"

---

Servers

Script	Image	Model
servers/qwen3-coder-next-fp8/run-v23.sh	avarok/dgx-vllm-nvfp4-kernel:v23	Qwen/Qwen3-Coder-Next-FP8
servers/qwen3-coder-next-fp8/run-v11.sh	avarok/vllm-dgx-spark:v11	Qwen/Qwen3-Coder-Next-FP8
servers/qwen3-coder-next-nvfp4/run.sh	avarok/dgx-vllm-nvfp4-kernel:v23	Sehyo/Qwen3.5-122B-A10B-NVFP4 + MTP
servers/qwen3-next-nvfp4/run.sh	avarok/dgx-vllm-nvfp4-kernel:v23	nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4
servers/qwen35-coder-nvfp4/run.sh	avarok/vllm-dgx-spark:v11	vincentzed-hf/Qwen3-Coder-Next-NVFP4

---

Patches included

Patch	Versions	What it fixes
entrypoints/chat_utils.py	v23, v11	Tool call args left as JSON string — Jinja2 |items TypeError
tool_parsers/qwen3coder_tool_parser.py	v23, v11	IndexError on stream finish + arguments always streamed as {}
model_executor/layers/quantization/modelopt.py	v23	NVFP4 MTP layer exclusion — mtp.fc missed by wildcard → shape mismatch
model_executor/models/qwen3_5_mtp.py	v23	Clamp OOB token IDs from padded vocab during MTP draft sampling

See docs/TOOL_CALL_BUGS.md for full root-cause analysis.

---

TRT-LLM (MiniMax)

For serving saricles/MiniMax-M2.5-REAP-139B-A10B-NVFP4-GB10 via TRT-LLM 1.3.0rc6:

Method	Script/Doc	Notes
Container	serve_minimax.sh	Self-contained; applies patches at startup
Local venv	~/trtllm-venv/bin/trtllm-serve	No Docker; requires ABI shim setup

Documentation:

• docs/trtllm-rc6-container.md — container workflow, model format, scale math, all 18 patches explained
• docs/trtllm-local-venv.md — local venv setup, ABI shim, maintenance
• docs/trtllm-build.md — what trtllm-build is for and why it's not used for MiniMax

---

Building the custom Docker image

To bake patches into the image rather than volume-mounting them at runtime:

# From repo root
docker build \
  -f images/dgx-vllm-nvfp4-kernel/v23/Dockerfile \
  -t dgx-vllm-patched:v23 \
  .

---

Applying patches to a running container

# Auto-detects version from container image
bash patches/apply.sh qwen3-fp8-server

# Or specify explicitly
bash patches/apply.sh qwen3-fp8-server v23

Note: requires a container restart to reload the Python files after copying.

---

Clone

git clone git@github.com:scottgl9/spark-llm-scripts.git