Real-world benchmark of Qwen 3.5-27B / Qwen 3.6-27B / Gemma 4-31B Dense models on NVIDIA DGX Spark (GB10), using a unified llama.cpp b8922 runtime. Speed, Japanese commonsense reasoning (JCommonsenseQA), and Vision-Language Model (VLM) tasks across 6 Dense + 3 MoE configurations — all measured on the same hardware with the same build, in April 2026.
Why this repo exists: Almost every "Qwen 3.6 vs Gemma 4" comparison online is a table of marketing benchmarks copied from official model cards. This repo is what actually happens when you run them on real DGX Spark hardware with llama.cpp, with publicly reproducible scripts.
This is a translation and extension of the original Japanese article on Qiita: DGX Spark で Dense 27-31B 対決 — Qwen 3.5 / 3.6 と Gemma 4 を BF16・Q4_K_M で並べてみた.
- Q4_K_M Dense reaches usable speed on DGX Spark. tg = 10–12 tok/s for conversational use. BF16 sits at 3.8–4.5 tok/s — too slow for daily use, as previously reported. The Q4_K_M / BF16 ratio is 2.6–2.8×.
- JCommonsenseQA (Thinking OFF): Gemma 4-31B leads by a small margin. 98.12% in BF16, 97.77% in Q4_K_M. But across all 11 measured configurations, the spread is only 2.14 points (24 questions) — quantization and Dense/MoE differences sit within noise.
- Quantization (BF16 → Q4_K_M) costs only 0.2–0.6 points on JCQ. Qwen 3.6-27B even gains 0.18 points when quantized (96.69% BF16 vs 96.87% Q4_K_M) — likely noise (2 questions), but a useful reminder that "BF16 is always better" doesn't always hold.
- Long-prompt (pp512 → pp16384) degradation: Qwen series −3% to −6%, Gemma 4 −12% to −13%. Qwen scales smoothly to 16K; Gemma 4 drops more.
- VLM (Q4_K_M only): JSON extraction parses 100% across all 3 Dense models. PPE detection is brittle — Qwen 3.5 and Gemma 4 hit 1/3, Qwen 3.6 hits 0/3 (3 samples is small, but Qwen 3.6 specifically fails the PPE schema).
This article compares Dense and MoE (Mixture of Experts) models. They have fundamentally different internal structures and performance characteristics:
- Dense models: every parameter is used for every token. Behavior is predictable and quality tends to be stable, but compute cost scales linearly with parameter count.
- MoE models: only a subset of "experts" is selected per token. Compute is cheaper per token, so they're typically much faster, though output quality can vary across tasks.
The common rule of thumb is "MoE for speed, Dense for stability", but the gap in practice depends heavily on hardware and quantization. This article tests both Dense (Qwen / Gemma) and MoE on the same DGX Spark hardware with the same llama.cpp build — to see whether the assumption "Dense is too slow to be practical on bandwidth-bound hardware" still holds, particularly under Q4_K_M quantization.
Two earlier benchmark articles by the same author intentionally excluded Dense models from comparison:
- Gemma 4 vs Qwen 3.5 MoE Showdown on DGX Spark (2026-04-06, Japanese)
- Qwen 3.6 vs 3.5 Real-World Comparison on DGX Spark (2026-04-18, Japanese)
The reason: on DGX Spark's bandwidth-bound architecture, Gemma 4-31B Dense was projected at 4.4 tok/s theoretical and measured at 3.7 tok/s (per the NVIDIA Developer Forum measurement). Too slow for everyday use.
| Model | Architecture | Active params | Per-token read | Theoretical tok/s |
|---|---|---|---|---|
| Qwen 3.5-35B-A3B MXFP4 | MoE | 3B | ~3 GB | ~91 |
| Gemma 4 26B-A4B F16 | MoE | 3.8B | ~8 GB | ~34 |
| Gemma 4 31B BF16 | Dense | 30.7B | ~62 GB | ~4.4 |
But here's the thing: Qwen 3.6-27B Dense Q4_K_M measured 11.75 tok/s on DGX Spark in the most recent single-model comparison article. That changes the picture. Q4_K_M cuts per-token memory traffic to roughly 1/4 of BF16 — pulling Dense back into the 10+ tok/s range.
The hypothesis: Q4_K_M Dense might be a new sweet spot for DGX Spark. This article measures it across 3 Dense models × 2 quantizations to find out.
Theoretical tok/s = memory bandwidth ÷ memory size of active parameters
DGX Spark memory bandwidth: ~273 GB/s
Gemma 4-31B BF16: 273 ÷ ~62 GB ≈ 4.4 tok/s → measured 3.82 tok/s (87% efficiency)
Gemma 4-31B Q4_K_M: 273 ÷ ~17 GB ≈ 16 tok/s → measured 10.65 tok/s (67% efficiency)
Qwen 3.6-27B BF16: 273 ÷ ~54 GB ≈ 5.0 tok/s → measured 4.54 tok/s (91% efficiency)
Qwen 3.6-27B Q4_K_M: 273 ÷ ~17 GB ≈ 16 tok/s → measured 11.85 tok/s (74% efficiency)
BF16 reaches 87–91% of theoretical bandwidth. Q4_K_M plateaus at 67–74% — likely due to compute overhead in the dequantization kernels. Even so, Q4_K_M Dense achieves 10–12 tok/s in practice, which is comfortable for interactive use.
This benchmark uses b8922 (commit 13d36cf89, CUDA, SM121a). Roughly 30
tags shipped during the b8892–b8922 window. Notable items relevant to local
inference:
- server: heap-buffer-overflow fix (CVE-2026-21869, #22267) — out-of-bounds when
n_discardis negative. Apply this if you expose llama-server externally. - server: anthropic API prefix caching fix (#21793) — prefix cache misses on the Anthropic-compatible API.
- server: SWA-full logic fix (#22288) — Sliding Window Attention full-expansion behavior, relevant to Gemma family.
- CUDA: fuse relu + sqr (#22249) — minor; regression test on the 6 configs in this article showed differences within measurement noise.
- HIP: graphs default ON (#22254) — AMD ROCm default behavior change (relevant if you also run AMD).
- SYCL: oneAPI 2025.3.3 + MoE mul_mat_vec_q fusion (#21920) — Intel GPU updates.
For DGX Spark specifically, the b8892 → b8922 performance impact on these 6 Dense configurations is within measurement noise (Qwen 3.6-27B Q4_K_M regression test: pp512 825.89 → 818.80, tg128 11.90 → 11.88). The CVE fix is the practical reason to upgrade.
| Component | Detail |
|---|---|
| Hardware | NVIDIA DGX Spark (GB10, Grace + Blackwell, 128 GB unified memory) |
| OS | NVIDIA DGX OS (Ubuntu 24.04.3 LTS base, aarch64) |
| Kernel | 6.14.0-1015-nvidia |
| CUDA | 13.0 |
| llama.cpp | b8922 (commit 13d36cf89, CUDA, SM121a, source build) |
| Quality benchmark | JCommonsenseQA v1.1 (1,119 questions, 5-choice, 3-shot) |
| VLM benchmark | Caption / JSON extraction / PPE detection (8 images) |
Build command:
cmake -B build \
-DGGML_CUDA=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON \
-DCMAKE_CUDA_ARCHITECTURES=121 \
-DGGML_CUDA_F16=ON \
-DGGML_NATIVE=ON \
-DCMAKE_BUILD_TYPE=Release
cmake --build build -j $(nproc) \
--target llama-server llama-bench llama-cli llama-mtmd-cliCMAKE_CUDA_ARCHITECTURES=121 is mapped internally to 121a, the native
arch for GB10.
| # | Model | Quantization | Weight size | Source |
|---|---|---|---|---|
| 1 | Qwen 3.5-27B (Dense) | Q4_K_M | 15.58 GiB | unsloth |
| 2 | Qwen 3.6-27B (Dense) | Q4_K_M | 15.65 GiB | unsloth |
| 3 | Gemma 4-31B (Dense) | Q4_K_M | 17.39 GiB | ggml-org |
| 4 | Qwen 3.5-27B (Dense) | BF16 | 50.10 GiB | unsloth |
| 5 | Qwen 3.6-27B (Dense) | BF16 | 50.10 GiB | unsloth |
| 6 | Gemma 4-31B (Dense) | BF16 | 57.18 GiB | ggml-org |
JCQ measured on all 6 configurations. VLM measured on the 3 Q4_K_M configurations only — VLM with BF16 takes ~200s per sample, which is prohibitively expensive for the sample count needed.
For the MoE-side comparison data (Sections 5-4 and 7-2), three additional models from the previous articles were re-measured on b8922 to align builds:
- Qwen 3.5-35B-A3B MoE MXFP4 (35B total / 3B active)
- Qwen 3.6-35B-A3B MoE MXFP4 (35B total / 3B active)
- Gemma 4-26B-A4B MoE F16 (26B total / 3.8B active)
llama-bench is the benchmark tool bundled with llama.cpp. It measures raw
inference throughput without the API overhead of llama-server. Two metrics:
- pp (prompt processing) — how fast the model ingests existing prompt tokens
- tg (token generation) — how fast the model generates new tokens
Run command:
llama-bench -p 512 -n 128 -ngl 99 -fa 1 -mmp 0 -r 5 -m <model.gguf>-ngl 99: offload all layers to GPU (DGX Spark's unified memory holds BF16 comfortably)-fa 1: enable Flash Attention-mmp 0: disable mmap (reduces measurement noise)-r 5: average across 5 runs
- BF16 leads on prompt processing across all models (833–905 tok/s); Q4_K_M is 5–12% behind (746–823 tok/s)
- Qwen 3.6-27B BF16 is the fastest at 904.8 tok/s across all 6 configs
- Gemma 4-31B sits ~8% slower than Qwen 27B in both BF16 and Q4_K_M — directly tracks the parameter count gap (30.7B vs 26.9B)
- Q4_K_M is 2.6–2.8× faster on tg (11 tok/s vs 4 tok/s)
- Qwen 3.5 and Qwen 3.6 are nearly identical (11.94 vs 11.85) — same architecture family, expected
- Gemma 4-31B is ~11% slower on tg than Qwen 27B series
| Model | Quant | pp512 (tok/s) | tg128 (tok/s) |
|---|---|---|---|
| Qwen 3.5-27B | Q4_K_M | 820.68 ± 5.53 | 11.94 ± 0.01 |
| Qwen 3.6-27B | Q4_K_M | 823.49 ± 7.86 | 11.85 ± 0.01 |
| Gemma 4-31B | Q4_K_M | 746.31 ± 2.29 | 10.65 ± 0.01 |
| Qwen 3.5-27B | BF16 | 860.67 ± 7.42 | 4.50 ± 0.00 |
| Qwen 3.6-27B | BF16 | 904.80 ± 6.42 | 4.54 ± 0.00 |
| Gemma 4-31B | BF16 | 833.90 ± 5.82 | 3.82 ± 0.00 |
How much does pp throughput drop on long prompts? This matters for RAG and long-document summarization (8K–16K context).
The previous MoE comparison showed Qwen 3.6 MoE shrinking the pp512 → pp16384 degradation from −6.2% to −2.4%. Does Dense behave similarly?
- Qwen 3.5 / 3.6-27B (BF16 and Q4_K_M) scale smoothly to 16K, with degradation in the −3% to −6% range
- Gemma 4-31B (BF16 and Q4_K_M) drops −12% to −13% — likely the SWA + Global Attention hybrid architecture imposing higher long-context cost
- Qwen 3.6 shows slightly less degradation than 3.5 (−6.2% → −4.1%) — same direction as the MoE generation gap, but smaller in magnitude
| Prompt length | Q3.5 Q4 | Q3.6 Q4 | G4 Q4 | Q3.5 BF16 | Q3.6 BF16 | G4 BF16 |
|---|---|---|---|---|---|---|
| pp512 | 812 | 801 | 740 | 898 | 898 | 850 |
| pp2048 | 808 | 800 | 723 | 914 | 913 | 872 |
| pp4096 | 802 | 805 | 706 | 919 | 913 | 843 |
| pp8192 | 793 | 797 | 684 | 894 | 893 | 806 |
| pp16384 | 762 | 768 | 650 | 870 | 866 | 738 |
| Degradation (512→16K) | −6.2% | −4.1% | −12.2% | −3.1% | −3.6% | −13.2% |
All configurations maintain 650–870 tok/s at pp16384 — Dense 27–31B + Q4_K_M is workable for RAG and long-document workloads on DGX Spark.
| Aspect | Result |
|---|---|
| tg | Q4_K_M is 2.6–2.8× faster (11 tok/s vs 4 tok/s) |
| pp (short) | BF16 wins by 5–12%, but Q4_K_M's 700–820 tok/s is more than enough |
| pp (16K) | Qwen series −3% to −6%, Gemma 4 −12% to −13% — Qwen scales better |
Bottom line: use Q4_K_M for everyday workloads and reach for BF16 only when quality demands it. This mirrors the MoE-side conclusion (use MXFP4 by default) — the "quantize first" approach holds for Dense too.
Three MoE models from previous articles were re-measured on b8922 to align builds. With the same hardware and same build, you can directly compare how active-parameter scale affects tg / pp.
- MoE (Qwen 3.5/3.6-35B-A3B, 3B active) wins tg overwhelmingly at 60+ tok/s — bandwidth-bound impact is small with 3B active params
- Gemma 4-26B-A4B MoE F16 (3.8B active) is slower than MXFP4 at 26.80 tok/s — but still 2× faster than Dense Q4_K_M (10–12 tok/s). Active size × quantization combine to drive speed
- Dense Q4_K_M (11–12 tok/s) is slower than MoE but practical for chat. Dense BF16 stays at 4 tok/s — too slow for daily use
| Model | Architecture | Quant | pp512 (tok/s) | tg128 (tok/s) | Source |
|---|---|---|---|---|---|
| Qwen 3.5-35B-A3B | MoE | MXFP4 | 2,300.99 | 63.13 | This article (re-run) |
| Qwen 3.6-35B-A3B | MoE | MXFP4 | 2,289.18 | 62.61 | This article (re-run) |
| Gemma 4-26B-A4B | MoE | F16 | 1,456.66 | 26.80 | This article (re-run) |
| Qwen 3.6-27B | Dense | BF16 | 904.80 | 4.54 | This article |
| Qwen 3.5-27B | Dense | BF16 | 860.67 | 4.50 | This article |
| Gemma 4-31B | Dense | BF16 | 833.90 | 3.82 | This article |
| Qwen 3.6-27B | Dense | Q4_K_M | 823.49 | 11.85 | This article |
| Qwen 3.5-27B | Dense | Q4_K_M | 820.68 | 11.94 | This article |
| Gemma 4-31B | Dense | Q4_K_M | 746.31 | 10.65 | This article |
- MoE dominates on tg (62–63 tok/s on MXFP4). These numbers match the previous article's 63.17 / 62.91 within noise — b8892 → b8922 didn't move them.
- MoE pp is ~2.5× Dense BF16 — at 35B total parameters, MoE has more compute resources for prompt processing than Dense BF16 (27–31B).
- Dense Q4_K_M vs MoE F16 (Gemma 4-26B-A4B) tg ratio is 2.5× (10.65 vs 26.80). Active 3.8B (F16) has roughly the same per-token memory traffic as Active 30.7B (Q4_K_M), and that's reflected directly in speed.
On bandwidth-bound hardware like DGX Spark, "same parameter scale, MoE is always faster" is structurally true — confirmed across these 9 configurations. Dense 27–31B Q4_K_M is fine for chat. If you want maximum speed, MoE wins.
What does JCommonsenseQA measure?
JCommonsenseQA v1.1 is a 5-choice multiple-choice benchmark for Japanese commonsense reasoning, 1,119 questions, evaluated with 3-shot prompting (3 example Q&A pairs followed by the actual question). It's the de facto standard for evaluating Japanese LLM quality — coding and math need separate benchmarks, but JCQ is a solid first-pass quality check for "can the model handle everyday Japanese correctly?".
llama-server launch:
~/llama.cpp-b8922/build/bin/llama-server \
-m <model.gguf> \
--host 0.0.0.0 --port 8080 \
-ngl 99 -fa on -c 4096 \
--jinja \
--reasoning-format deepseek \
--chat-template-kwargs '{"enable_thinking":false}' \
--temp 0.7 --top-p 0.8 --top-k 20-ngl 99: full GPU offload-fa on: Flash Attention enabled-c 4096: context length (3-shot fits well within this)--jinja: use the model's built-in template (required for thinking control)--chat-template-kwargs '{"enable_thinking":false}': Thinking OFF- Sampling: Qwen-recommended values (also applied to Gemma 4 for consistency)
Evaluation uses jcq_bench.py (same as previous
articles) against the OpenAI-compatible API on llama-server.
- Gemma 4-31B leads (BF16 98.12%, Q4_K_M 97.77%) — beats every Qwen BF16 config
- Quantization (BF16 → Q4_K_M) costs 0.2–0.6 points — close to the noise floor on a 1,119-question test
- Qwen 3.6-27B BF16 (96.69%) is lower than Q4_K_M (96.87%) — a 2-question difference (likely noise) but a useful counter-example to the "BF16 is always better" assumption
| Model | Quant | Correct / 1,119 | Accuracy |
|---|---|---|---|
| Gemma 4-31B | BF16 | 1098 | 98.12% |
| Gemma 4-31B | Q4_K_M | 1094 | 97.77% |
| Qwen 3.5-27B | BF16 | 1092 | 97.59% |
| Qwen 3.5-27B | Q4_K_M | 1085 | 96.96% |
| Qwen 3.6-27B | Q4_K_M | 1084 | 96.87% |
| Qwen 3.6-27B | BF16 | 1082 | 96.69% |
Observed trends:
- Gemma 4-31B's lead (1 point or so over Qwen) likely comes from the 3B parameter advantage (31B vs 27B), or because JCQ aligns with Gemma's training-data emphasis on Japanese commonsense. Hard to disentangle from a single benchmark.
- Quantization impact stays small — consistent with the previous Gemma 4 F16 vs Q4_K_M result (0.11 point difference). Knowledge-style multiple choice doesn't penalize aggressive quantization much.
- Qwen 3.6-27B's BF16 vs Q4_K_M reversal also showed up in the previous single-model comparison — a reproducible observation, even if the magnitude is at noise level.
- Qwen 3.5 vs 3.6 difference is in the noise (0.1 point). The MoE comparison showed the same pattern — the generation jump didn't move JCQ.
Reference values from other models measured on the same DGX Spark setup (all Thinking OFF):
- Gemma 4-31B BF16 is #1 across all articles (98.12%) — beats Nemotron 3 Super 120B (97.86%)
- Dense > MoE trend holds: Dense 27–31B sits at 96.7–98.1%, MoE 35B-A3B at 95.98–96.16%
- Quantization differences are small — Q4_K_M sometimes beats BF16. Choice of size and architecture matters more than choice of quantization
| Model | Quant | JCQ OFF | Source |
|---|---|---|---|
| Gemma 4-31B Dense | BF16 | 98.12% | This article |
| Nemotron 3 Super 120B | Q4_K_M | 97.86% | Previous article (single-model) |
| Gemma 4-31B Dense | Q4_K_M | 97.77% | This article |
| Qwen 3.5-27B Dense | BF16 | 97.59% | This article |
| Qwen 3.6-27B Dense | UD-Q4_K_XL | 97.05% | Previous article (single-model) |
| Qwen 3.5-27B Dense | Q4_K_M | 96.96% | This article |
| Qwen 3.6-27B Dense | Q4_K_M | 96.87% | This article |
| Qwen 3.6-27B Dense | BF16 | 96.69% | This article |
| Gemma 4-26B-A4B MoE | F16 | 96.51% | Earlier article ① |
| Qwen 3.5-35B-A3B MoE | MXFP4 | 96.16% | Earlier articles ① ② |
| Qwen 3.6-35B-A3B MoE | MXFP4 | 95.98% | Earlier article ② |
Across these 11 configurations, the spread between best (Gemma 4-31B BF16: 98.12%) and worst (Qwen 3.6-35B-A3B MoE: 95.98%) is 2.14 points, equivalent to 24 questions out of 1,119. JCQ being a 5-choice Japanese commonsense benchmark, models in the 20B+ range tend to converge to a similar ceiling — quantization choice and Dense/MoE selection produce noise-level differences within this range.
That said, three rough patterns emerge:
- Gemma 4-31B sits at the top (BF16 98.12%, Q4_K_M 97.77%) — both at least 0.3 points above the next entry
- Dense > MoE direction holds (Dense 27–31B at 96.7–98.1%, MoE 35B-A3B at 95.98–96.16%), but the gap is at most 2 points
- Quantization impact stays under 0.6 points — and reversals (Q4_K_M > BF16) do happen, as in Qwen 3.6
JCQ is a knowledge / commonsense task. It's not enough on its own to declare "which model is better" — it should be combined with coding (HumanEval+ etc.) and long-context coherence benchmarks for a full picture.
What does the VLM benchmark measure?
VLMs are multimodal LLMs that take image + text inputs together. This article runs three task types to probe their behavior:
- Caption generation — free-form description of an image (classic VLM eval axis)
- JSON extraction — structured output with a specified schema (location / subjects / technologies). Tests how reliably the model follows schema. Practical for business applications.
- PPE (Personal Protective Equipment) detection — given a workplace photo, identify safety equipment compliance and report it as JSON. Tests domain-specific structured output. The hardest of the three.
Captions are evaluated qualitatively (latency + sample output check). JSON and PPE are scored on parse rate and content validity.
The image set is identical to previous articles: 5 trade-show photos (CEATEC 2025, etc., taken by the author) and 3 stock workplace photos for PPE detection.
llama-server launch (with mmproj added):
~/llama.cpp-b8922/build/bin/llama-server \
-m <model.gguf> \
--mmproj <mmproj.gguf> \
--host 0.0.0.0 --port 8080 \
-ngl 99 -fa on -c 8192 \
--jinja--mmproj: multimodal projector (image encoder weights). Required for image input-c 8192: context length raised because image + prompt uses 3–5K tokens
- JSON extraction parses 100% across all 3 models — schema-following is stable on Dense Q4_K_M
- PPE detection: Qwen 3.5 / Gemma 4 = 1/3, Qwen 3.6 = 0/3 — Qwen 3.6 emits valid JSON but doesn't fit the PPE schema's content requirements
- Latency: Qwen series < Gemma (Caption 89.6–90.4s vs 102.7s) — Caption hits the
max_tokens=1024cap, so the tg gap (11.85 vs 10.65 tok/s) shows up directly
| Model | Caption (s) | JSON (s) | PPE (s) | JSON parse | PPE parse |
|---|---|---|---|---|---|
| Qwen 3.5-27B Q4_K_M | 89.60 | 49.05 | 82.86 | 100% | 33.3% |
| Qwen 3.6-27B Q4_K_M | 90.35 | 40.86 | 89.18 | 100% | 0.0% |
| Gemma 4-31B Q4_K_M | 102.66 | 68.40 | 93.61 | 100% | 33.3% |
3 samples is not enough to draw firm conclusions. But for "image + domain-specific structured output", Qwen 3.5 / Gemma 4 handled the PPE schema better than Qwen 3.6 in this set.
The 3 MoE models from previous articles, re-measured on b8922 alongside the Dense Q4_K_M results. VLM is sensitive to llama.cpp build updates (mmproj fixes ship frequently), so same-build comparison is critical.
- MoE Caption latency is dramatically shorter (17.98–21.45s) — 3B active params makes generation fast, even when hitting
max_tokens=1024 - JSON parse rate: Dense Q4_K_M leads (100% across all 3). MoE drops: Qwen 3.5 80%, Qwen 3.6 60%, Gemma 4-26B-A4B 40%
- PPE detection is brittle for everyone (0–33%). It's a model-capability question, not a quantization-or-architecture question
| Model | Architecture | Quant | Caption (s) | JSON parse | PPE parse | Source |
|---|---|---|---|---|---|---|
| Qwen 3.5-35B-A3B | MoE | MXFP4 | 17.98 | 80.0% | 33.3% | This article (re-run) |
| Qwen 3.6-35B-A3B | MoE | MXFP4 | 21.45 | 60.0% | 0.0% | This article (re-run) |
| Gemma 4-26B-A4B | MoE | F16 | 40.14 | 40.0% | 33.3% | This article (re-run) |
| Qwen 3.5-27B | Dense | Q4_K_M | 89.60 | 100% | 33.3% | This article |
| Qwen 3.6-27B | Dense | Q4_K_M | 90.35 | 100% | 0.0% | This article |
| Gemma 4-31B | Dense | Q4_K_M | 102.66 | 100% | 33.3% | This article |
- JSON extraction stability is higher on Dense Q4_K_M (100% across the board). MoE has parse failure modes that Dense doesn't — Gemma 4-26B-A4B F16 drops to 40%.
- Gemma 4-26B-A4B F16 JSON parse was 0% on b8665 (per the earlier article). That climbed to 40% by b8922 — likely the result of mmproj-side fixes during the intervening builds.
- Caption latency gap of 4–5× between MoE and Dense is dominated by hitting
max_tokens=1024. To compare output length fairly, raisemax_tokensor constrain via prompt. - PPE detection: Qwen 3.6 fails (0%) on both architectures (MoE and Dense). Qwen 3.6 has a specific weakness on the PPE schema (3 samples is small, but the pattern is reproducible across architectures).
For "image + structured output" workloads: Dense Q4_K_M for JSON reliability (100% parse), MoE for response latency. That's the operational trade-off.
- Default to Q4_K_M — tg in the 10–12 tok/s range, JCQ delta ≤ 0.6 points vs BF16, VRAM footprint stays under 17 GiB so you can run other tools alongside.
- BF16 is reasonable when accuracy is paramount, but Qwen 3.6-27B's BF16 < Q4_K_M reversal is a reminder to always benchmark your specific use case.
- Gemma 4-31B BF16 leads JCQ at 98.12% — if you can fit 57 GiB and tolerate 3.82 tok/s tg, this is your pick.
- If BF16 is overkill, Gemma 4-31B Q4_K_M (97.77% JCQ, 10.65 tok/s tg) is the best speed/quality balance in this set.
- Among Q4_K_M Dense, Qwen 3.5-27B was the most consistent in this run — shortest Caption latency, 1/3 PPE parse rate.
- Sample size is small (3 PPE images). Validate on your own image set before production — Qwen 3.6's PPE schema struggle may not generalize to other domains.
| Aspect | Result |
|---|---|
| Dense Q4_K_M practicality | tg 10–12 tok/s — workable for chat |
| Q4_K_M / BF16 tg ratio | 2.6–2.8× (Q4_K_M wins) |
| JCQ Thinking OFF | Gemma 4-31B leads (BF16 98.12%, Q4_K_M 97.77%) |
| Quantization quality cost | 0.2–0.6 points — within noise |
| Long pp degradation | Qwen series −3% to −6%, Gemma 4 −12% to −13% |
| VLM JSON extraction (Dense Q4_K_M) | All 3 models parse 100% |
| VLM PPE detection (Dense Q4_K_M) | Qwen 3.5 / Gemma 4 = 1/3, Qwen 3.6 = 0/3 |
| Speed gap vs MoE (cross-article) | tg: MoE is 5–6× faster than Dense Q4_K_M |
| VLM stability vs MoE (cross-article) | Dense Q4_K_M wins on JSON parse (100% vs 40–80%) |
The earlier "Dense is too bandwidth-bound to be practical on DGX Spark" stance doesn't hold once Q4_K_M is in the picture. MoE is still the speed king on this hardware, but Dense Q4_K_M is now a real choice, not just a theoretical option.
For anyone running 27–31B Dense on DGX Spark, hopefully these numbers help narrow the choice.
This repository contains the scripts and configuration to reproduce the
benchmarks. See the scripts/ directory:
models.conf— model pathsbench_all.sh— llama-bench runner (short and long)vlm_all.sh— VLM evaluatorjcq_bench.py— JCQ evaluator (OpenAI-compatible API, 3-shot)run-20260425.sh— top-level orchestrator (Dense 6 configs)run-20260425-moe.sh— MoE 3 configs (for cross-article comparison)make_charts/— chart generation scripts (English-labeled)
# Build llama.cpp b8922 first (see Section 3)
# Edit scripts/models.conf to point to your GGUF files
# Run the full Dense benchmark suite
./scripts/run-20260425.sh b8922
# Run the MoE re-measurement (optional, for cross-article comparison)
./scripts/run-20260425-moe.sh b8922Raw output is in results/.
MIT — feel free to fork, reuse, or adapt the scripts and chart generators for your own benchmarks.
@nabe2030 (Makoto Watanabe). Companion articles on Qiita.
- shamily/gemma4-llama-dgx-spark — Gemma 4 family benchmark on DGX Spark (Gemma-only, more sizes)
- nabe2030/gemma4-vs-qwen35-dgx-spark — earlier MoE-side comparison from the same author
- llama.cpp DGX Spark Performance Discussion — community leaderboard
- NVIDIA Developer Forum: Gemma 4 Day-1 Inference on DGX Spark
- 2026-04-25 — Initial release covering Dense 6 configs + MoE 3 cross-article re-measurements