DeepSeek-v4-Flash EXL3 on one DGX Spark

_{by Mia'a AI Lab}

Single-node launcher for DeepSeek V4 Flash 0731 (EXL3/ExLlamaV3) with DSpark speculative decoding on one NVIDIA DGX Spark (GB10, SM121, 128 GiB unified memory).

Serves the 0xSero/deepseek-v4-flash-0731-spark build (3.0 bpw EXL3) via the sparkinfer (formerly b12x) kernel stack — a complete, self-contained Docker recipe tuned for speed and KV-cache headroom on a single device.

⚙️ Defaults changed (2026-08-21): the launcher start.sh serves the deep-context NVFP4 config — KV_RECORD=stock432 (native 432-byte records), GPU_MEMORY_UTILIZATION=0.94, MAX_MODEL_LEN=384000, MAX_NUM_SEQS=1, DSpark K5 healthy → 439,622-token KV pool (a 370,104-token context stress-tested with exact needle recall — see Stress test). The NVFP4 dual-cache prefill bugs behind the old either/or are fixed — full story in the internal postmortem (kept local, not in this repo).

• One DGX Spark, tensor-parallel 1 — no second node required (unlike the official FP4 build, which needs TP2 across two Sparks).
• DSpark K5 speculative decoding with a K64 draft model (MODE=dspark, fixed K5, K64 draft).
• nvfp4_ds_mla compressed KV cache with the B12X_MLA_SPARSE attention/MoE backend.
• Tuned CUDA-graph capture ([6,12,24]) so concurrent decode stays on captured graphs instead of falling back to eager.
• Long-prefill fairness (decode-starvation guard) enabled natively.
• Two upstream kernel backports applied as read-only bind-mounts (see Backports).
• Weights download fully locally into ./hf-hub (no remote machine involved). An optional LAN mode reuses one copy of the ~107 GB across machines via an SSHFS share (REMOTE_HOST — opt-in).

Two needle-in-a-haystack runs on 2026-08-21, both on fresh boots of the deep-context config (MAX_NUM_SEQS=1, KV_RECORD=stock432, util 0.94), each at ~96% of that boot's MAX_MODEL_LEN.

Method (both runs): a single large user prompt of random-word filler, generated so prefix caching cannot deduplicate it (cached_tokens: 0 — every token fresh KV); a secret passphrase planted at token ~20; a recall question at the very end; temperature 0, thinking disabled via chat_template_kwargs, max_completion_tokens 128.

Prefill throughput decays with depth — ~1,024 tok/s at the start of a request, ~350–614 tok/s past 300k accumulated — so a full-length 384k prefill takes ~10 minutes end-to-end.

These runs are the deepest exercise to date of the NVFP4 dual-cache prefill path fixed in image-patch/sparkinfer/ — the pre-fix kernel NaN'd on any prompt ≥ 7 tokens; these tests pushed 320k and 370k tokens through it with exact recall.

• Hardware: one NVIDIA DGX Spark (GB10, SM121, ≥128 GiB unified memory), GPU passthrough to Docker via the NVIDIA Container Toolkit.
• OS: Linux aarch64 (DGX OS). The runtime image is aarch64-only.
• EarlyOOM: disable it if present on the host (sudo systemctl disable --now earlyoom). The server intentionally holds ~94% of the 128 GiB unified memory, so a user-space OOM killer can't tell a healthy server from a leak — and may kill it mid-serve.
• Software: Docker Engine + Compose v2, curl, and ~110+ GiB free local disk. sshfs + fuse3 are only needed for the optional LAN sharing mode (auto-installed with sudo when missing; requires user_allow_other in /etc/fuse.conf).
• Network: internet access to HuggingFace for the one-time ~107 GB download. The download is fully local — no remote host required. (Optional LAN mode: set REMOTE_HOST/REMOTE_USER/REMOTE_SHARE_DIR to reuse a single copy across machines.) No HuggingFace login required; the repo and image are public. If you do hit HF rate limits or need a private repo, set the optional HF_TOKEN in start.sh (or via env / .env).

./start.sh      # start: deep-context NVFP4 (384k, DSpark) — writes compose.yml
./start.sh --no-wait   # start without waiting

First boot is intentionally long: pulls the image, downloads ~107 GB of weights locally (into ./hf-hub), coalesces TP4→TP1 losslessly, builds the K64 draft, and captures CUDA graphs. It is marked healthy only when the OpenAI-compatible endpoint responds.

ℹ️ No compose file is shipped in this repo — compose.yml is generated by start.sh on the first run and rewritten on every launch. It is gitignored, since the real config lives in the launcher. To produce it without starting anything: ./start.sh compose-gen. Do not hand-edit it.

Everything downloads on this machine; no remote host is involved in the default path:

• Default: ./start.sh auto-downloads on first boot into the local HF cache root ./hf-hub and coalesces into ./data/tp1. The losslessly coalesced serving checkpoint, the K64 draft, and the runtime caches all live on this machine.
• Optional offline prep: ./download.sh performs the same download+coalesce+verify standalone (also fully local). Once ./data/tp1/rank-sliced-tp1-manifest.json exists, boot skips the download/coalesce step entirely, so a network-free runtime follows — the manifest (not an env knob) is the gate.
• Optional LAN sharing: set REMOTE_HOST (+ REMOTE_USER / REMOTE_SHARE_DIR / MIA_MOUNT / HF_CACHE) to reuse a single weight copy across machines instead of downloading locally — the spark3 arrangement, preserved as an opt-in mode on main and unchanged on the mia-shared-setup branch.

The only hard requirement is free host RAM ≥ 114.3 GiB at launch (0.94 × 121.63 GiB; this UMA machine shares the 121.63 GiB between GPU and host) — stop an old container first and check free -h before launching. If a boot ever fails (KV check or otherwise), it stops after one failure (restart: on-failure:1 — it can never death-spiral the host); lower MAX_MODEL_LEN a notch and retry once, don't launch repeatedly while the host is loaded.

Concurrency is a single env override away: MAX_NUM_SEQS=4 ./start.sh. Note that the pool itself shrinks at higher sequence counts (the hybrid cache split shifts — a 2-seq boot observed ~337k total, ≈169k per slot), so depth trades against concurrency. The rest of the launcher is the deep-context config described in this README.

curl -sS http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
    "model": "deepseek-v4-flash-0731",
    "messages": [{"role":"user","content":"Write a correct Python function that returns the first n Fibonacci numbers."}],
    "temperature": 0, "max_completion_tokens": 256 }'

Served model name: deepseek-v4-flash-0731. API: http://127.0.0.1:8888/v1 (OpenAI-compatible chat/responses endpoints, DSpark spec decoding active).

Every tunable is an environment variable override: GPU_MEMORY_UTILIZATION=0.94 ./start.sh.

MAX_NUM_BATCHED_TOKENS is load-bearing twice. vLLM sizes the profiled activation peak (and therefore leftover KV) from it, and warmup captures the largest b12x compressed-MLA scratch, then lock_workspace() freezes that size. Lowering it to recover KV can still boot and even match prefill on a cold prompt, then crash later — typically a warm prefix-cache turn that attends a large width immediately:

AssertionError: Workspace is locked but allocation from 'b12x.py:…:_run_compressed_mla'
requires … MB, current size is … MB. Workspace growth is not allowed after locking.

Leave it at 8224. Do not treat it as a free memory knob. (Reported in #4; a 2048 trial recovered ~3.9 GiB of profiled peak but then hit the locked-workspace assertion under real traffic.)

./start.sh                   # applies ABLATE as set in start.sh (0 or 1)
ABLATE=1 ./start.sh          # same, env override for this boot

The server exposes an OpenAI-compatible API on http://127.0.0.1:8888/v1. It binds 0.0.0.0 by default (the container runs with network_mode: host), so from another machine on the LAN use the Spark's own address — http://<spark-ip>:8888/v1. There is no authentication in front of the port: keep it on a trusted network, or run SERVING_HOST=127.0.0.1 ./start.sh for local-only and reach it over an SSH tunnel. Recommended settings for any client:

"deepseek-v4-flash-spark-local": {
  "baseUrl": "http://127.0.0.1:8888/v1",
  "apiKey": "dummy",
  "api": "openai-completions",
  "authHeader": false,
  "auth": "none",
  "models": [
    {
      "id": "deepseek-v4-flash-0731",
      "name": "DeepSeek V4 Flash 0731 Spark · DSpark · 384k (local Spark)",
      "reasoning": true,
      "input": ["text"],
      "contextWindow": 384000,
      "maxTokens": 32768,
      "thinkingLevelMap": {
        "minimal": null, "low": null, "medium": null,
        "high": "high", "max": "max"
      },
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": true,
        "requiresReasoningContentOnAssistantMessages": true,
        "maxTokensField": "max_tokens",
        "thinkingFormat": "deepseek"
      }
    }
  ]
}

Two fixes from local-inference-lab/b12x (the current maintainer repo of the sparkinfer/b12x kernel stack) are applied onto the image's pinned kernel tree (272a84bd) as read-only bind-mounts in image-patch/sparkinfer/:

• #150 — preallocate W4A16 route histograms for CUDA-graph capture.
• #228 — keep graphed tiny-decode routes with inactive expert ids in range (prevents out-of-range reads on graph padding).

The 0xSero image is pinned and no newer build (with newer kernel commits) is published yet, which is why these are backported locally. Remove the image-patch/sparkinfer/ mounts from start.sh to return to stock kernels.

EXL3 — @turboderp_ and @BrandonMusicKy are the core people behind EXL3. The EXL3 path used here with vLLM is based on the work of turboderp (ExLlama) and was adapted to this customized trellis quantization format by Brandon and other members of the RTX 6K Discord. Format source: turboderp-org/exllamav3.

• Hugging Face checkpoint this launcher downloads: 0xSero/deepseek-v4-flash-0731-spark (REAP-K216, EXL3 3.0 bpw, Trellis) and the upstream 0xSero/DeepSeek-V4-Flash-0731-EXL3-3.0bpw. That is the full extent of the 0xSero attribution — the hosted repo this recipe pulls weights from, not EXL3 itself.
• Runtime image: ghcr.io/0xsero/deepseek-v4-flash-0731-spark-sparkinfer (NVIDIA vLLM 26.02 base)
• Kernel stack: local-inference-lab/b12x (sparkinfer / formerly b12x)
• Design reference: MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark (2-node TP2 recipe; our speed/KV work derives from its methodology)
• Refusal direction (files/direction_r1.pt): thank you drowzeys/keys-DeepSeekV4-Flash-GA-0731-Dspark-Abliterated-Anchored-Tensors

The packaging/orchestration glue in this repository is licensed under the MIT License. The runtime image, model weights, and upstream libraries are covered by their own licenses.