DGX Spark research and tests — containers, benchmarks, and investigation notes for running large models on the NVIDIA DGX Spark (GB10, SM 12.1, 128 GB unified memory).
Entries address compatibility issues with CUDA 13.x, aarch64, and SM121 that aren't covered by upstream containers or documentation. Each folder is a self-contained topic; dates and environment details live inside each folder's README.
1. nvfp4/ — NVFP4 on DGX Spark (friction → working)
Single top-level home for the NVFP4 story on GB10 (only nv* topic folder). March 2026 memory-floor campaign (19 GB model, 50–120 GB → ~32 GB @ ~50 tok/s), then the software + driver/OS catch-up that made stock upstream viable. Former trees nvfp4-guide/, nvfp4-landscape/, nvfp4-memory/ live under this folder (guide.md, memory.md, archive/).
- Current narrative + what still holds:
nvfp4/README.md - Living launch YAMLs:
recipes/(not re-copied here) - Silicon deep dive:
fp4_SASS/(related FP4 silicon notes; separate topic) - Dated lab notes under
nvfp4/archive/(March 2026 stamps)
2. turboquant/ — TurboQuant 3-bit KV Cache Compression
Patches vLLM with TurboQuant KV cache compression (PR #38479). Builds on the NVFP4 stack — same model family, compressed KV cache.
- 240K token context at 64 GB memory, zero memory creep
- Faster than fp8 at long context (7.4 vs 5.0 tok/s at 64K)
- Needle-in-haystack recall tested across 1K–240K tokens
- Based on eugr/spark-vllm-docker with prebuilt SM121 FlashInfer
3. mamba-dev/ — mamba-ssm for aarch64
Working mamba-ssm + causal-conv1d build for DGX Spark. pip install mamba-ssm from PyPI is broken on aarch64 (x86_64 binaries + PyTorch ABI mismatch). This container builds from source.
- Loads NemotronH hybrid models (Mamba-2 + Attention) via transformers
- Tested with Nemotron-Nano-12B-v2-VL-BF16 (13.2B params on GPU)
4. nemo3-super-gguf/ — Nemotron-3-Super 120B via sm_121 llama.cpp
Native sm_121 llama.cpp build for Nemotron-3-Super 120B MoE. GGUF path documented because the NVFP4 checkpoint was blocked on vLLM — NemotronH LatentMoE uses relu² with separate projections and no fused act_and_mul MoE backend supports it.
- ~17 tok/s at Q4_K (66 GB weights, ~71 GB runtime)
- Ollama GGUF not compatible with upstream llama.cpp (different MoE tensor layout)
- NVFP4 attempt documented: three layers of failure (config, kernel, pip wheels)
5. dual-spark/ — Pairing two GX10 over a single CX-7 cable (200 Gb/s RDMA)
Field-tested runbook for pairing two NVIDIA DGX Sparks (ASUS Ascent GX10) over a single CX-7 direct-link cable. Covers the two failure classes that burn the most forum hours: power-path throttling (measure with gpu_stress.py — AC sag can clamp P8 under load; not all UPS bad) and dual-node connection issues (UFW, NCCL TCP-fallback, GID-index, half-bandwidth twin misconfig). Username models A (same user), B (two users + SSH map), and B→A conversion are documented.
- Symptom → fix table at the top of
dual-spark/DUAL_SPARK_SETUP.md - Visual cable / twin / PCI map:
dual-spark/TOPOLOGY.md(one cable ≈ 200 Gb/s total) - Verified: ~195–197 Gb/s RDMA aggregate (full single-cable budget)
- Patches eugr's launcher with NCCL multi-rail + RoCEv2 GID overrides
6. atlas/ — Atlas Inference Engine: head-to-head vs vLLM
Single-host benchmark of Atlas, a pure Rust+CUDA inference engine for GB10. Reproduced their published ~88 tok/s Nemotron-3-Nano-30B-A3B-NVFP4 number (88.34 max, 84.36 median, c=1). Same model, same prompts: Atlas single-stream beats vLLM by +63%, vLLM wins concurrency at c=4 (continuous batching). Atlas cold-start with cached weights: 20s vs vLLM 171s (8.5×). Includes reproducer (bench.py), raw JSON results, and the serve --help flag dump.
7. fp4_SASS/ — FP4 on GB10: silicon vs marketing
Evidence-first probe of what FP4 tensor-core paths actually work on sm_121a. NVIDIA's own ptxas refuses tcgen05.mma for sm_121a — the "1 PFLOP FP4" pipe is a silicon gap on GB10, not a software switch. Consumer warp-level FP4 (mma.sync.aligned.kind::mxf4.block_scale...) compiles cleanly and emits the same SASS opcode (OMMA.SF.16864.F32.E2M1.E2M1.E8) as the RTX 5060 Ti (sm_120a) — Spark and the 5060 Ti share the consumer-Blackwell FP4 silicon at the instruction level.
findings.md— full evidence trail with PTX/SASS citationskernel-patterns.md— patterns for integrating custom low-bit formats with consumer-Blackwell FP4, distilled from llama.cpp MMQreference/fp4_mma_reference.cu— working MXFP4/NVFP4 inline-PTX MMA, compiles cleanly on sm_120a + sm_121a- CuTeDSL 4.5.1+ now supports
sm_121aas a first-class JIT target (earlier 4.4.x "missing kernel images" reports no longer apply)
8. comfyui_spark_notes/ — ComfyUI wheel-shadowing gotchas on Spark
Two real pip-shadowing risks observed while running ComfyUI on Spark with the Triplany/comfyui-dgx-spark kit applied: PyPI's onnxruntime silently overwrites Jay0515's sm_121 GPU wheel (same top_level import path, different distribution names, no pip conflict detection), and pip install sageattention overwrites the local rebuild against current torch/CUDA.
- Shadowing mechanism traced to shared
top_level.txtdeclaration across distinct distributions get_available_providers()is the reliable detector — startup-log GPU-discovery warnings are misleading- Both findings verified against a live ComfyUI install on 2026-05-22
- Cross-references
fp4_SASS/for the sm_120 / sm_121 SASS-equivalence basis behind thesm_120rebuild target
9. recipes/ — Verified serving launch YAMLs
Spark Arena + local measured recipes (NVFP4 and related). Living “run this” counterpart to nvfp4/.
10. flightrec/ — GB10 benchmark flight recorder
Always-on Tier-1 machine-state recorder for Spark benches: GPU power/clock/throttle/energy, CPU, and memory pressure on one timeline. Invalidates throttled runs; includes mem_guard (OOM / mem-pressure backstop), smoke / watch / live monitoring arms, adaptive-N sampling and gates.
- Install:
uv pip install -e ".[dev]"fromflightrec/ - Primary target: DGX Spark (GB10); useful for OOM and power-path throttle triage
- Protocols:
flightrec/PROTOCOLS.md
| March 2026 lab notes | June 2026 baseline (nvfp4 pass) | |
|---|---|---|
| System | NVIDIA DGX Spark (GB10) | same |
| GPU | SM 12.1, 128 GB unified | same |
| CPU | 20-core ARM Grace (aarch64) | same |
| Driver / CUDA | 580.142 · 13.0/13.2 toolkit | 580.159.03 · 13.0.2 (DGX OS 7.5.0) |
Stamp driver · CUDA · image tag · model ID on any new measurement.
This is work on a single hardware configuration mostly. Results may not generalize to other setups. The TurboQuant container patches an unmerged vLLM PR — the API may change. Sharing what worked in case it helps others with similar hardware.
Related: llm-latent-bridge-negative-results — negative results on bridging LLM hidden states at shallow layers, measured on this hardware.
- eugr/spark-vllm-docker — Community vLLM container with prebuilt SM121 wheels
- vLLM PR #38479 — TurboQuant attention backend by vibhavagarwal5
- TurboQuant — Zandieh et al., Google Research, ICLR 2026
- turboquant-torch — Community PyTorch reimplementation
- NVIDIA DGX Spark Playbooks
- The DGX Spark community on NVIDIA Developer Forums
- Avarok-Cybersecurity/atlas — Pure Rust+CUDA inference engine, AGPL-3.0
- Atlas Discord community — collaborative benchmarking + maintainer responsiveness
- eugr's spark-vllm-docker networking deep-wiki — basis for the dual-spark §2 NCCL config
Topics span March–August 2026 — DGX Spark GB10 / SM121. See each folder for stamped baselines.