qwen3.5-122B-A10B-on-spark

Qwen3.5-122B-A10B (hybrid GDN + mamba + 128-expert MoE, ~10B active) on a single NVIDIA DGX Spark (GB10 / SM121, 128 GB / 119 GiB unified) under vLLM, with DFlash block-diffusion speculative decode and a dense-bandwidth patch stack — measured end-to-end, with a per-token bandwidth model behind the numbers.

The default dense profile serves a purpose-built hybrid INT4+FP8 checkpoint, bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid, downloaded ready-to-run — no local checkpoint build.

Status: working end-to-end, one-shot install. On real agent tool-call turns, DFlash decode reaches a median ~81 tok/s on GB10 — about 2× the native MTP-2 head (~40 tok/s) on the same workload, and above albond's fully-patched MTP recipe (51.58 tok/s end-to-end), the recipe this project builds on (see Building on the albond recipe). DFlash acceptance is task-dependent (it block-drafts 12 tokens in one parallel forward), so the gain is largest on structured / tool-call / code traffic and falls to parity on open-ended prose.

The default dense profile bundles a dense-bandwidth stack (hybrid INT4+FP8 shared experts + int8 lm-head) on top of DFlash, downloaded as a prebuilt checkpoint: +28 % on no-spec / base decode (28.2 → 36.0 tok/s) and equal to plain DFlash on high-acceptance agent traffic (per the amortization law below), at a modestly smaller KV pool. The dflash profile drops the dense patches for the largest KV pool and the same ~81 tok/s on agents.

• Model (default): bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid — the prebuilt hybrid the dense profile downloads and serves: INT4 (AutoRound/GPTQ) routed experts + attention, calibrated FP8 shared experts, BF16 embeddings / head; ~67 GiB safetensors (not GGUF — vLLM serves it directly). Built from Intel/Qwen3.5-122B-A10B-int4-AutoRound (INT4 base) + Qwen/Qwen3.5-122B-A10B-FP8 (FP8 shared experts). The dflash / base / mtp profiles serve the plain Intel/…int4-AutoRound instead.
• Drafter: z-lab/Qwen3.5-122B-A10B-DFlash — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), sharing the model's embed_tokens + lm_head, ~1.6 GiB.
• Engine: vLLM 0.23, sm121 build with the DFlash PRs, via the prebuilt image ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix. No host build — the runtime patches in runtime/ are applied at serve time.
• Hardware: NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s.

Quick start

On a DGX Spark with Docker and the NVIDIA container runtime:

curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash -s -- --start

This command:

1. Verifies the host (aarch64, GB10 / SM121, Docker GPU access, free disk).
2. Pulls the sm121 vLLM image (~40 GiB, one-time).
3. Downloads the hybrid checkpoint (~67 GiB) and the DFlash drafter (~1.6 GiB) into the HF cache.
4. Starts the dense profile on :8000, waits until READY (~3 min), and runs the "capital of France" smoke test (asserts "Paris").

For the plain-DFlash agent profile (largest KV pool, no dense patches, serves the Intel INT4 checkpoint):

./install.sh --start --profile dflash

To reuse a checkpoint already on disk and skip the download:

# an existing checkpoint directory (mounted read-only at /model):
./install.sh --start --model-dir /path/to/checkpoint
# or reuse an existing HF cache (download becomes a no-op if already present):
./install.sh --start --hf-home /mnt/big/hf

Preview without running: append --help.

Hardware requirements

Validated on	NVIDIA DGX Spark (GB10, SM121, 128 GB / 119 GiB unified)
Likely to work	other Blackwell with --force (untested)
Runtime	Docker + NVIDIA container runtime (docker run --gpus all)
Disk	≥ 75 GiB free (image + weights); ≥ 150 GiB with --build-hybrid
OS	aarch64 Linux (Grace)
Memory	128 GB / 119 GiB unified holds the model + DFlash drafter + KV

GB10 is detected via nvidia-smi --query-gpu=compute_cap returning 12.1; other hardware requires --force.

Memory and context (defaults tuned for up to 3 concurrent streams)

The intended deployment is a single user with up to 3 concurrent decode streams (for example, a main agent thread plus subagents). A single stream can use the full 262 144 context; the defaults assume concurrent streams stay under ~100 k tokens each. The default operating point is one stream with no contention; two to three streams are additive (see the per-workload table below).

Attention KV is small for this model (~24 KiB/token — 12 of 48 layers are full attention, with GQA num_key_value_heads=2), but the per-sequence GDN/mamba state and the activation reserve dominate the footprint, so the usable pool is far smaller than a KV-only estimate suggests. The values below are measured on the target hardware at the shipped defaults (gpu-mem 0.82, ctx 262144, seqs 3):

Measurement (default dense profile)	Value
Free memory at READY	~16 GiB (responsive, no swap; ~15 GiB under peak 3-stream load)
GPU KV cache pool	426,610 tokens (dflash profile: 456,664)
Max concurrency at full 262 144	1.63× (dflash: 1.74×)

The dense pool was lifted from 376,518 → 426,610 tokens (+13 %) at the same 0.82 headroom by reclaiming over-reserved memory rather than spending headroom: the int8 lm-head frees its now-dead bf16 copy (~1.4 GiB — the DFlash drafter shares the same int8 lm_head, and tie_word_embeddings=False, so it is genuinely unused after quantization), and VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 returns the CUDA-graph over-estimate (~0.6 GiB; actual capture is ~0.14 GiB, drawn from the wide 0.82 headroom). Both are validated coherent with full drafter acceptance (4–12).

Decode-only throughput (streaming, excludes prefill), by workload and concurrency:

Workload	1 stream	2 streams	3 streams	aggregate @ 3
prose	48.8	38.4	30.8	92.5
code	66.9	55.3	43.0	129
agentic (real 6 k tool-call ctx)	121.8	98.9	66.7	200

(tok/s per stream; aggregate is the sum across streams. Each added stream lowers per-stream throughput and raises the aggregate. The single-context agentic 121.8 exceeds the ~81 headline, which is the median over 10 varied real turns including longer, slower-prefilling contexts.)

A typical load — three streams under ~100 k each (≈ <300 k tokens) — fits the 426 k pool with margin, and a single stream can still reach the full 262 144 context. At gpu-mem 0.88–0.89 the static footprint leaves only ~5 GiB free; the host then swaps and requests stall. 0.82 is the validated value (~16 GiB free on dense, ~15 GiB under peak load). Defaults (override via flags or environment variables):

Flag / env	Default	Note
--gpu-mem / GPU_MEM	0.82	~14 GiB free (validated); 0.88+ over-subscribes and swaps
--ctx / CTX (MAX_MODEL_LEN)	262144	model native max; a single stream can reach any length up to this. Costs only KV-pool sizing — the CUDA-graph compile range tracks max-batched-tokens, not ctx
--max-num-seqs / MAX_NUM_SEQS	3	concurrent-stream cap; the pool holds ~1.6× (dense) / ~1.7× (dflash) a full-262 k context, ample for <100 k streams
--max-batched-tokens / MAX_BATCHED_TOKENS	8192	chunked-prefill chunk, kept below ctx so a long prefill does not batch all at once

The default operating point is a single stream (no contention; ~81 tok/s on agent turns). Two to three concurrent streams are additive: per-stream throughput decreases as the decode batch grows (more routed-expert traffic per step) while aggregate throughput rises — a throughput-versus-latency trade, not a safety limit. A fourth simultaneous stream requires --max-num-seqs 4 (additional streams queue rather than fail) or a smaller --ctx.

Unified-memory OOM hard-freezes the host, and the vLLM profiler can undershoot peak by a couple of GB. Bring the server up under scripts/monitor.sh (OOM auto-kill guard) the first time at any new gpu-mem or ctx.

Seeing a smaller KV pool / less free memory than the numbers above? The Spark's 128 GB is unified — the GNOME desktop and maintenance daemons can hold ~10–15 GiB that would otherwise go to the KV cache. Drop to a headless multi-user.target with Entrpi/dgx-spark-serving-mode (sudo serving-mode serve) to hand that memory back to the model — it directly raises the usable pool and context length.

Startup time

Time to READY is ~3 min (vs ~12 min on the default loader). --load-format fastsafetensors (default) reads the 62–71 GiB straight to the device — ~32 s vs ~8 min for the default mmap-backed per-tensor read + redundant CPU→GPU copy, which is pathologically slow on GB10's current kernel (the copy is also physically pointless on unified memory). It falls back to nogds automatically — no GPUDirect Storage hardware required; override with LOAD_FORMAT=auto. The remaining ~2.5 min is torch.compile (~42 s) + CUDA-graph capture (~2 min); the compile cache is persisted at $HF_HOME/.vllm_cache, so boots after the first skip the compile.

Profiles

Selected with --profile:

Profile	Stack	Best for	Measured
dense (default)	hybrid INT4+FP8 + int8 lm-head + DFlash n=12	general — downloads the prebuilt hybrid; ≈ dflash on agents, +28% on base	36.0 base (+28%) · 59.0 albond-bench · ~81 Hermes
dflash	INT4 + DFlash n=12	agent path; largest KV pool (456k vs 426k)	~81 tok/s Hermes · 53.7 albond-bench
base	plain INT4, no speculative decode	airtight baseline	28.2 tok/s c=1
mtp	INT4 + native MTP-2 head	comparison	~40 tok/s Hermes

The server is OpenAI-compatible (/v1/chat/completions with tool calls + SSE, /v1/completions, /v1/models) and serves under the model name qwen.

Tool calling is enabled by default (--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3), so clients can send tools with tool_choice="auto". Qwen3.5 emits the XML tool format, so qwen3_xml is the correct parser — hermes returns 200 but with empty tool_calls (the call lands in content instead). <think> reasoning is split into reasoning_content. Override with TOOL_PARSER= / REASONING_PARSER= (empty disables; qwen3_coder also works).

Network: the server binds 0.0.0.0 in --net=host, so it is reachable from the LAN at http://<spark-ip>:8000 — the 127.0.0.1 in the examples is just the local default. There is no auth; put it behind a reverse proxy / firewall for shared use.

Benchmarks

All single-stream (c=1), temperature 0, GB10. "Hermes" regenerates the next assistant turn over 10 real conversations from a live agent's state.db (73 % tool-calls); "albond-bench" is albond's end-to-end harness (completion tokens / total wallclock incl. prefill, 5 prompts, run 1 discarded — directly comparable to the published 51.58).

DFlash vs MTP, same harness, unpatched

Workload (accept len)	base no-spec	MTP-2	DFlash n=12
Prose (~2.3)	28.2	33.7	33.2 (use n=4)
Code (~5.4)	28.2	40.5	54.5
Counting (~11)	28.2	43.7 (MTP caps at acc 3)	124.5
Hermes, real turns (8.3)	—	39.9	~81
albond-bench e2e (6.5)	—	—	53.7

MTP-2 drafts 2 tokens sequentially, so acceptance caps at ~3; DFlash block-drafts 12 in one parallel forward, accepting 5–11 on predictable / agent traffic and running ~2× ahead. The two tie only on low-acceptance prose. Unpatched DFlash (53.7) already clears albond's fully-patched MTP (51.58) under the same end-to-end method.

Dense-bandwidth stack (dense profile)

Two independent always-on levers, ported to vLLM 0.23 as runtime patches: hybrid INT4+FP8 (BF16 shared experts → calibrated FP8) and int8 lm-head (the 248 320-row vocab projection → int8 w8a16 GEMV, ~2× the bf16 read).

Config	base (acc 1)	DFlash spec, albond-bench (acc 6.4)	Hermes (acc 8.3)
INT4 baseline	28.2	53.7	~81
+ hybrid-FP8	30.4 (+7.8%)	57.0 (+6.1%)	~80
+ int8 lm-head	32.7 (+16%)	—	—
+ both	36.0 (+28%)	59.0 (+10%)	~80–87 (noise)

Building on the albond recipe

This project builds on albond's DGX-Spark Qwen3.5-122B recipe — the first working high-throughput recipe for this model on Spark, and the reference used for the comparisons here. On eugr's vLLM 0.19.1 fork, albond established:

• the rebuilt hybrid INT4+FP8 checkpoint (BF16 dense → calibrated FP8),
• the INT8 lm-head patch (the single largest dense-bandwidth lever),
• MTP-2 native speculative decode, and
• the end-to-end benchmark methodology (completion tokens / total wallclock, incl. prefill), reproduced verbatim in scripts/bench_albond.py.

This project carries that recipe forward to the latest vLLM and a stronger drafter, and composes the community patches there:

1. Dense levers forward-ported to vLLM 0.23. albond's patches target the 0.19 fork and do not apply cleanly. The hybrid-FP8 dispatch is re-expressed against 0.23's maybe_update_config(model_name, hf_config=…) quant-config hook; the INT8 lm-head is reintegrated from scratch (the prior port zeroed the lm-head weight — which corrupts the DFlash-shared head — and looped per-row for batch > 4, which is slower under speculative decode; both are fixed in patch_int8_lmhead_v3.py).
2. MTP-2 replaced by the DFlash block-diffusion drafter, running on the hybrid 122B in vLLM via the KV-unify fix. DFlash block-drafts 12 tokens in one parallel forward versus MTP's sequential head (acceptance-capped at ~3), running ~2× ahead on agent / code traffic.
3. Dense levers composed with DFlash rather than MTP.

Comparison — albond's end-to-end method, same hardware class

Stack	Spec	Dense patches	e2e tok/s
albond (published)	MTP-2	hybrid-FP8 + INT8 lm-head + PR#38325	51.58
this project — dflash	DFlash n=12	none	53.7 (+4%)
this project — dense	DFlash n=12	hybrid-FP8 + INT8 lm-head	59.0 (+14%)

On the real agent workload (decode-only, regenerating live tool-call turns), DFlash's parallel block-drafting runs further ahead of MTP's sequential head: ~81 vs ~40 tok/s.

Note on the comparison. The 51.58 is the published figure for albond's stack (vLLM 0.19 + MTP); the figures here are for this stack (vLLM 0.23 + DFlash). Both use the same end-to-end harness on the same hardware class (DGX Spark / GB10) — a best-on-each-stack comparison, not a single-variable controlled run. Unpatched DFlash (53.7) already exceeds the fully-patched MTP result, so the dense stack is additional headroom rather than the source of the difference.

The amortization law

The dense levers cut always-on weight reads (shared experts and lm-head, read every token). Under speculative decode the verify forward reads those weights once and amortizes them across the accepted block, so the gain shrinks as acceptance rises — monotonically, across the curve:

dense stack uplift:   +28%  (base, accept 1)
                  →   +10%  (albond-bench, accept ~6.4)
                  →   ~0%   (Hermes, accept ~8.3)

For the agent path (dflash), DFlash's high acceptance already saturates the dense levers; the remaining bottleneck is routed-expert verify-batch reads, which no dense-weight quantization touches. For base / low-acceptance serving (dense), the stack is a real +28 %. Full derivation in docs/FINDINGS.md.

Under the hood: the five runtime patches

vLLM is unmodified on disk; runtime/serve.sh edits the installed package in-place before vllm serve (idempotent, sentinel-guarded):

Patch	Effect	Required by
patch_unify2.py	scale-block KV-cache unify so the hybrid GDN+mamba target absorbs the drafter's attention spec (the upstream assert cannot)	DFlash (any spec profile)
patch_prefix_align.py	makes resolve_kv_cache_block_sizes' mamba back-off align-aware so prefix caching coexists with DFlash (uses the GCD hash block size, 2240, instead of the LCM that fails the coordinator assert)	prefix caching ON (default)
patch_inc_hybrid.py	adds an INCConfig.maybe_update_config override that detects FP8 dense layers in the hybrid checkpoint and dispatches Fp8LinearMethod for shared_expert	dense
patch_int8_lmhead_v3.py	replaces the lm-head matmul in _get_logits with a batched int8 w8a16 Triton GEMV (keeps the bf16 weight for the shared drafter)	dense
patch_fla_shmem.py	allows the FLA GDN chunk kernels to use large tiles on sm121's 99 KiB shmem (prefill / TTFT only; harmless)	always (free)

The rationale for the unify patch, the requirement that the drafter run FLASH_ATTN (non-causal), and the full vLLM-vs-SGLang history are in docs/FINDINGS.md.

Repo layout

install.sh                 One-shot installer (curl | bash | --help)
runtime/                   Mounted read-only at /host inside the container:
  serve.sh                   vLLM serve wrapper (applies the patches, then serves)
  patch_unify2.py            DFlash KV-unify fix
  patch_inc_hybrid.py        hybrid INT4+FP8 dispatch
  patch_int8_lmhead_v3.py    int8 lm-head GEMV
  patch_fla_shmem.py         FLA sm121 big-tile (prefill)
  mtp_serve.sh               MTP-2 comparison serve
scripts/                   Host-side helpers:
  monitor.sh                 Container-startup monitor with OOM auto-kill guard
  bench_decode.py            Decode-only tok/s (excludes TTFT)
  bench_albond.py            albond's e2e method (comparable to 51.58)
  hermes_bench.py            Real agent turns from ~/.hermes/state.db
  conc_workloads.py          Per-workload concurrency sweep (decode-only)
  run_bank.sh                prose / code / counting / hermes bank
tools/
  build-hybrid-checkpoint.py Builds the hybrid INT4+FP8 checkpoint (--build-hybrid)
  inspect_ckpt.py            Reports which layers are INT4 / BF16 / FP8
  validate_*.py              Standalone correctness checks for the patches
docs/
  FINDINGS.md                Full investigation, methodology, and the
                             amortization-law derivation

Reproducing

# default: the dense profile (downloads the prebuilt hybrid) + smoke test:
./install.sh --start
# (to build the hybrid checkpoint locally instead of downloading it:)
#   ./install.sh --build-hybrid && ./install.sh --start

# plain-DFlash agent path (serves the Intel INT4 checkpoint, largest KV pool):
./install.sh --start --profile dflash

# benchmarks (run on the host against the server; requires: pip install requests):
python3 scripts/bench_decode.py --base-url http://127.0.0.1:8000 --model qwen \
        --prompt "Write a detailed essay about the history of tea."
python3 scripts/bench_albond.py http://127.0.0.1:8000 "dflash"     # e2e, vs 51.58
python3 scripts/hermes_bench.py --base-url http://127.0.0.1:8000   # real agent turns
python3 scripts/conc_workloads.py --base-url http://127.0.0.1:8000 # concurrency sweep

# MTP comparison:
./install.sh --start --profile mtp

Related work

Project	Role
bleysg/...int4-fp8-hybrid	the default served checkpoint — this project's prebuilt hybrid INT4+FP8
vLLM	the inference engine; served unmodified-on-disk
Intel/...int4-AutoRound (INT4 base) · Qwen/...FP8 (FP8 donor) · z-lab/...DFlash (drafter)	upstream weights the hybrid + dflash profiles build on
albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4	the MTP + hybrid-FP8 + int8-lm-head recipe; benchmark reference and source of the ported dense levers
Entrpi/ds4-on-spark	sibling repo, same hardware, different model (DeepSeek-V4-Flash via ds4)
Modal: Speculative decoding is all you need	the DFlash block-diffusion drafter and the task-dependent-acceptance framing

Acknowledgements

• albond — the foundation this project builds on: the hybrid INT4+FP8 checkpoint, the INT8 lm-head, MTP-2, and the end-to-end benchmark methodology, forward-ported and recomposed here onto vLLM 0.23 + DFlash.
• z-lab / Modal — the DFlash drafter and the block-diffusion speculative-decode work.
• Intel/AutoRound — the INT4 target quantization.
• vLLM and the AEON sm121 image maintainers — the engine and the DFlash-enabled GB10 build.
• eugr/spark-vllm-docker — the vLLM-on-Spark base much of this ecosystem started from.

License

MIT — see LICENSE. The patches are original; vendored third-party files (tools/build-hybrid-checkpoint.py) retain their upstream attribution.