Performance of llama.cpp on NVIDIA DGX Spark #16578
Replies: 32 comments 119 replies
-
|
Thanks for the benchmark! I would like to request additional benchmark for a very popular model GLM-4.5-Air-FP8: and quants for it:
|
Beta Was this translation helpful? Give feedback.
-
|
Saw the benchmark results. Thank you so much for the work! Appreciate very much. |
Beta Was this translation helpful? Give feedback.
-
|
Hi. It would be great to see a Qwen Next 80B benchmark for these two models: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Not support yet with open pr currently |
Beta Was this translation helpful? Give feedback.
-
Yeah I really want to see the performance of a specific model comparing full 16 bit precision, Q8, Q4, FP4 and FP8. None the less, thank you for the wonderful data! |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Getting similar performance with my Farmework Desktop. Thanks for helping my FOMO. |
Beta Was this translation helpful? Give feedback.
-
I'm sure the connectx-7 200GB networking has something to do with the pricing difference :) |
Beta Was this translation helpful? Give feedback.
-
Interesting, the Asus GB10 seems to run with 240W power adapter, much higher than the DGX Spark. I wonder if you will get more performance given the higher power intake. |
Beta Was this translation helpful? Give feedback.
-
I havent seen the specs. But its possible ASUS just used a power adapter with a high enough rating for the device? For example, I can plug a 90watt compatible power adapter into my 45watt laptop. It will pull what it needs to. |
Beta Was this translation helpful? Give feedback.
-
|
@bartlettroscoe i benched gpt-oss 120b on Framework Desktop a couple months ago: geerlingguy/ai-benchmarks#21 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
with "correct" rocm and build I get:
|
Beta Was this translation helpful? Give feedback.
-
|
Can you run the classic llama 2 7B Q4_0 so it can be compared on the chart? |
Beta Was this translation helpful? Give feedback.
-
|
Super interesting, thanks for sharing, Georgi!
Could you please help me understand: Does "-d" mean KV cache length before the "-p" prefill happens? What does "-ub" define, eg batch size? |
Beta Was this translation helpful? Give feedback.
-
Yes.
Yes. |
Beta Was this translation helpful? Give feedback.
-
|
Could you add llama2-7b result to #15013? |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you! So whats the sense of a dgx spark? I mean sure it has 128gb memory, but i can offload bigger models between 96gb vram and the rest to normal Ram (CPU)... Its too expensive for what it offers. If the DGX Spark would be around 2k, like the Ryzen Max 395+ Mini-PC's it would be fine and okay. PS: And a Mac Mini/Studio is a much better option at 4k usd/eur, compared to a DGX Sparc. |
Beta Was this translation helpful? Give feedback.
-
|
Guys, please dont take fp4 or fp8 as a win. Let me explain: Comparing embedding Models is actually great, because you can simply query the resulting vector database and see the quantisation impacts. From my tests, no matter which Model, be it Qwen3-Embedding or BGE-M3 or anything else, the impact of Quantisation is Huge! FP32 is Amazing So you Guys want to tell me that FP4 is a win? I didnt tested fp4 tho, not even fp8, so i cant say for sure. Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
It depends on the model. In many cases, in my experience FP4 does a fantastic job. Also NVFP4 has the potential to be amazing. So is it situational? Sure, it can be. But I don't think it's something that can be ignored. Also, FP8 is also great, I have found little reason to not use it. |
Beta Was this translation helpful? Give feedback.
-
|
I'd agree that everyone should eval for their particular downstream tasks rather than just trusting perplexity or KLD. When running quants on my 405B model I ran JA MT-Bench evals and was surprised to find a bigger difference with FP8-Dynamic than IQ3_M. @icsy7867 I know you're just theory-crafting instead of running tests, but see my PRO 6000 TensorRT/NVFP4 benchmark below, but there is zero throughput benefit from NVFP4. Maybe related to NVIDIA/TransformerEngine#2255 - I never use TensorRT and it's impossible to build so I just used the latest docker for my tests (tensorrt-llm/release:1.2.0rc0) but I've put my full scripts/details online so it's easy for anyone to rent any GPU they want to check any configuration/variation for themselves. |
Beta Was this translation helpful? Give feedback.
-
|
yes it rely depend on model. for exemple I get for Mistral-SMAL:
|
Beta Was this translation helpful? Give feedback.
-
I appreciate the edit you did there. But you arent wrong, I wish I had a Blackwell gpu to test. But I am surprised the 6000 Pro doesnt have a speedup there from the FP4 tensor cores. Your data is much appreciated though, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov Are there llama.cpp benchmarks for the AGX Thor? It seems it's similar offering but Nvidia markets it as twice as fast. There are no official detailed spec sheet for the DGX Spark to make a comparison to the Thor (2560 cuda cores and 92 tensor cores), but Nvidia claims 2PLOPS (sparse FP4) for the Thor and 1PFLOPS (sparse FP4) for the Spark. |
Beta Was this translation helpful? Give feedback.
-
|
Quick tldr: Thor is sm_110 (formerly sm_101) with the datacenter-style tensor cores - including tensor memory. And no raytracing cores. While Spark is sm_121 with the full consumer Blackwell feature set. Thor and Spark have relatively similar memory bandwidth. The Thor CPU is much slower. Vector throughput on Thor is 1/3rd of the one on DGX Spark but you get twice the matrix throughput. Thor has 4 cursed Synopsys 25GbE NICs (set to 10GbE by default, see https://docs.nvidia.com/jetson/archives/r38.2/DeveloperGuide/SD/Kernel/Enable25GbEthernetOnQSFP.html as it doesn't have auto-negociation of the link rate) exposed via a QSFP connector providing 4x25GbE while Spark systems have regular ConnectX-7. Thor uses a downstream L4T stack instead of regular NVIDIA drivers unlike Spark. But at least the CUDA SDK is the same unlike prior Tegras. Oh and you get less other IO too. Side note: might be better to also consider GB10 systems from OEMs. Those are available for cheaper than AGX Thor devkits too. |
Beta Was this translation helpful? Give feedback.
-
I don't have one unfortunately, hoping whoever does will run those benchmarks.
This is a very weird and interesting tradeoff. |
Beta Was this translation helpful? Give feedback.
-
@woachk does "tensor memory" here refer to TMEM? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. |
Beta Was this translation helpful? Give feedback.
-
|
Here're my two cents - AGX Thor bench results: 
And Asus Ascent GX10 ( dgx spark): 
I want to point out one thing - I discovered that THOR gives a 22-28% speed boost on Generative AI (video/image generation)! |
Beta Was this translation helpful? Give feedback.
-
|
For those curious about Thor performance gpt-oss-20b-gguf# ./bin/llama-bench -m /workspace/models/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 | 2008.85 ± 4.18 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 | 60.85 ± 0.17 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1862.13 ± 4.80 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 55.03 ± 0.06 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1740.90 ± 3.24 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 53.58 ± 0.18 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 1446.75 ± 3.01 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 52.49 ± 1.94 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 1193.93 ± 0.72 |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 48.33 ± 0.04 |
build: f9fb33f2 (6771)Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF# ./bin/llama-bench -m /workspace/models/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF/qwen3-coder-30b-a3b-instruct-q8_0.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 | 1654.25 ± 1.80 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 | 44.26 ± 0.11 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1410.87 ± 2.22 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 39.46 ± 0.04 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1228.69 ± 1.78 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 36.88 ± 0.13 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 985.39 ± 7.04 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 33.55 ± 0.01 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 686.45 ± 0.93 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 26.92 ± 0.05 |
build: f9fb33f2 (6771)gpt-oss-120b# ./bin/llama-bench -m /workspace/models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 | 967.20 ± 6.04 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 | 42.00 ± 0.09 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 932.85 ± 2.33 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 38.81 ± 0.04 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 892.28 ± 2.88 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 39.22 ± 1.05 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 827.57 ± 1.28 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 37.77 ± 0.01 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 677.70 ± 1.06 |
| gpt-oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 34.02 ± 0.02 |
build: f9fb33f2 (6771) |
Beta Was this translation helpful? Give feedback.
-
|
That commit only applies the change to |
Beta Was this translation helpful? Give feedback.
-
|
I did a quick one off build where I removed the conditional around the scheduling block to force spin and I do see a consistent improvement. Just looking at power draw there is a probably at least another 10-20% performance untapped on thor beyond moving it to the spin scheduler. Currently looks like we are mostly cpu bound. 
Llama-bench Test Results (Qwen3moe 30B)schedule Default Spin Improvement (%) Average improvement: 1.86% Llama-batched-bench Test ResultsPP=4096: PP=8192: Spin schedule
Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes
Test: llama-bench
| model | size | params | backend | ngl | n_ubatch | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 | 1700.05 ± 2.02 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 | 45.67 ± 0.11 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d4096 | 1446.22 ± 3.54 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d4096 | 40.64 ± 0.05 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d8192 | 1257.35 ± 0.75 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d8192 | 38.09 ± 0.09 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d16384 | 992.37 ± 1.89 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d16384 | 33.62 ± 0.01 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | pp2048 @ d32768 | 687.30 ± 0.48 |
| qwen3moe 30B.A3B Q8_0 | 30.25 GiB | 30.53 B | CUDA | 99 | 2048 | 1 | tg32 @ d32768 | 27.05 ± 0.03 |
Test: llama-batched-bench
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 4096 | 32 | 1 | 4128 | 2.537 | 1614.38 | 0.789 | 40.54 | 3.327 | 1240.92 |
| 4096 | 32 | 2 | 8256 | 4.949 | 1655.30 | 1.301 | 49.18 | 6.250 | 1320.87 |
| 4096 | 32 | 4 | 16512 | 9.887 | 1657.09 | 1.663 | 76.98 | 11.550 | 1429.62 |
| 4096 | 32 | 8 | 33024 | 19.739 | 1660.11 | 2.289 | 111.86 | 22.027 | 1499.25 |
| 4096 | 32 | 16 | 66048 | 39.464 | 1660.65 | 3.279 | 156.14 | 42.743 | 1545.23 |
| 4096 | 32 | 32 | 132096 | 78.936 | 1660.49 | 5.033 | 203.46 | 83.968 | 1573.16 |
| 8192 | 32 | 1 | 8224 | 5.314 | 1541.47 | 0.839 | 38.14 | 6.153 | 1336.50 |
| 8192 | 32 | 2 | 16448 | 10.614 | 1543.68 | 1.396 | 45.86 | 12.009 | 1369.61 |
| 8192 | 32 | 4 | 32896 | 21.220 | 1544.24 | 1.888 | 67.79 | 23.108 | 1423.59 |
| 8192 | 32 | 8 | 65792 | 42.394 | 1545.87 | 2.792 | 91.68 | 45.187 | 1456.01 |
| 8192 | 32 | 16 | 131584 | 84.800 | 1545.66 | 4.206 | 121.73 | 89.006 | 1478.37 |
| 8192 | 32 | 32 | 263168 | 169.577 | 1545.87 | 6.867 | 149.11 | 176.444 | 1491.51 | |
Beta Was this translation helpful? Give feedback.
-
|
For prompt processing there's a lot more on the table but that means switching to tcgen05 MMA instructions. (Which is a separate instruction set than the regular tensor core one) And there's also the matter of using lower precision MMAs in general |
Beta Was this translation helpful? Give feedback.
-
|
I believe that Thor doesn't support tcgen05 because it doesn't have tensor-memory |
Beta Was this translation helpful? Give feedback.
-
|
Thor does have tensor memory - it uses the data centre tensor cores (it's sm_110[a]), Spark does not. See https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma |
Beta Was this translation helpful? Give feedback.
-
|
Would love to see accuracy of the same models on main banchmarks running in DGX as they will vary on different HW & FW in addition to the speed. As its clearly sing here https://artificialanalysis.ai/models/gpt-oss-120b/providers |
Beta Was this translation helpful? Give feedback.
-
|
Added AIME25 evals in the post. |
Beta Was this translation helpful? Give feedback.
-
|
Please bench the full Qwen3 coder model |
Beta Was this translation helpful? Give feedback.
-
|
There isn't any measurable benefits in terms of quality compared to |
Beta Was this translation helpful? Give feedback.
-
|
I am just impressed that it might run at all. It's there any bench on fine-tuning? |
Beta Was this translation helpful? Give feedback.
-
|
Would love to see this this cluster setup in the comparison table too |
Beta Was this translation helpful? Give feedback.
-
|
AFAICT this is vaporware. |
Beta Was this translation helpful? Give feedback.
-
|
On the subject of Spark and Thor, I have been looking for alternatives to TensorRT for python-free and community driven inference engine. I'm looking to leverage nvfp4 tensor cores , and wonder if there's any project or folks working to support those in llama.cpp? |
Beta Was this translation helpful? Give feedback.
-
|
The whole Blackwell product range, from the RTX 5050 onwards to the B200/300 through iGPUs |
Beta Was this translation helpful? Give feedback.
-
|
That said: NVIDIA/TransformerEngine#2255 |
Beta Was this translation helpful? Give feedback.
-
|
Just as an FYI, I don't have a Spark but I tested NVP4 on an RTX PRO 6000 (Llama 3.1 8B Instruct). NVP4 w/ TensorRT does not perform better than llama.cpp at bs=1, and at higher concurency, doesn't take a lead until c=32. I didn't test quality loss, but from a pure throughput perspective, I don't think the current NVFP4 implementation is particularly good. Certainly not worth all the custom quanting and other hassles...
|
Beta Was this translation helpful? Give feedback.
-
|
@lhl what's the prefill sequence length in the profiles above? |
Beta Was this translation helpful? Give feedback.
-
|
This is using a standard vLLM bench - ShareGPT w/ prefill 1024 and decode 128 I believe. If you have a specific use case it's probably best to just trying the device directly - I think they're available for a buck or two on Vast or Runpod. I think the compute is particularly strong for a client card. For example, the PRO 6000 actually beats an H100 on our Whisper inference sweeps. (Still trains much slower though) Here's my LLM sweep scripts (and raw results) btw: https://github.com/AUGMXNT/speed-benchmarking/tree/main/nvfp4 |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov - what flags did you use to compile for DGX Spark? Also, did you set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1? It does seem to offload layers to GPU properly, but nvtop/nvidia-smi shows host memory utilization growing to quite large numbers (more than 100GB and then it all goes to GPU memory). In comparison, my Strix Halo PC loads the same model 5x faster. My numbers: Without GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: Model loading time - 1 minute 44 seconds using this command: build/bin/llama-server -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -ngl 999 -ub 2048Benchmarks: build/bin/llama-bench -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048
build: 03792ad (6816) With GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: Model loading time: 49 seconds
For comparison, from my GMKTek Evo X2 (AMD AI MAX+ 395), same llama.cpp build, compiled with HIP: Model loading time: 25 seconds (8 seconds if still in caches!!!)
Any ideas? You benchmarks look closer to what I'd expect from this device. And long loading time makes me think that it is doing some extra mallocs/copying. |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov - yep, confirmed that it's the kernel. Most likely this option which appeared in 6.17: NO_PAGE_MAPCOUNT.
build: 7db35a7 (6922) This is on Fedora 43 Server with the new compiled kernel. On DGX Spark, of course :) |
Beta Was this translation helpful? Give feedback.
-
|
@eugr Thanks for the follow-up. I have added a section to the post referencing your findings about model load performance. |
Beta Was this translation helpful? Give feedback.
-
|
One more thing to note: mmap performance is still mediocre even with 6.17.1, so you have to use --no-mmap flag! Loading gpt-oss-120b on 6.17.1 kernel:
Same for vLLM, btw. Loading Qwen3-Next-80B-A3B-FP8 takes 8 minutes 44 seconds with default parameters and 1 minute 30 seconds with --safetensors-load-strategy eager (equivalent of --no-mmap), but the downside is much higher RAM consumption by vllm. Good thing that doesn't happen with llama.cpp! |
Beta Was this translation helpful? Give feedback.
-
|
Another update: DGX Spark just received an update to 6.14 kernel (6.17 is expected in January). Good news is that model loading is now significantly improved - takes 27 seconds on my machine for gpt-oss-120b compared to 68 seconds on previous 6.11. Still not as fast as 6.17, but close. Mmap performance remains bad, so don't use it. |
Beta Was this translation helpful? Give feedback.
-
|
Another improvement, suggested by Nvidia folks - increase NVME read-ahead buffer: On kernel 6.17 improves both mmap performance (from almost 2 minutes to 30 seconds) and no-mmap performance (from 20 to 15 seconds). Weirdly enough, works well only with llama.cpp, vllm model loading remains bad. On kernel 6.14 (now standard after a recent update), doesn't affect mmap performance, but improves no-mmap performance similar to 6.17. |
Beta Was this translation helpful? Give feedback.
-
|
Throughput is not the only metric. We need to take into account that different HW/FW produce different accuracy for the same model. Can someone test popular LLMs like gpt-oss? |
Beta Was this translation helpful? Give feedback.
-
|
I was reading this and wonder if the loading speed differences are because under DGX OS it uses older gcc/llvm that doesn't target gb10, and under Fedora it targets gb10: Since LLVM version 21 and gcc version 15, it is also possible to specifically target DGX Spark with -mcpu=gb10. This enables optimizations for the Cortex-X925 and Cortex-A725 cores as well as the optional crypto extension and is thus preferred over a more generic -march switch. When compiling on the Spark itself, this architecture is automatically selected with -march=native. |
Beta Was this translation helpful? Give feedback.
-
|
All right, a little dry facts I spotted. Chat on M4 Max with 64GB RAM
Chat on Spark with CUDA
llama-bench -m /Users/dev/Library/Caches/llama.cpp/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768 llama-bench -m /home/dev/.cache/llama.cpp/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768I guess thats because Metal runs better than CUDA on smaller context if I understand results correctly
This sounds like a memory‑bandwidth issue, which on the DGX Spark is limited to only 273 GB/s. Details
Tokens Per Secat 16 Nov 2025 On MacBook Pro M4 Max:
The result 117.32 tokens/secSteps On DGX Spark
The result 84.67 tokens/secConclusionPrice Rate: 6000/4000 => 1.5 I Would bet the Mac Studio M4 Max with 128GB RAM (at price ~5000 USD) would be better for price rate comparison for headless machines, but I |
Beta Was this translation helpful? Give feedback.
-
|
Well, yes, M4 Max has double memory bandwidth compared to Spark, so inference will be faster. Prompt processing, however, is faster on Spark because of better GPU. |
Beta Was this translation helpful? Give feedback.
-
|
Well, someone had to do it :) Running Unsloth/Qwen3-VL-235B-A22B-Instruct:Q4_K_XL on dual Sparks with full context (but tested up to 32K): eugr@spark:~/llm/llama.cpp$ build/bin/llama-bench -m ~/.cache/llama.cpp/unsloth_Qwen3-VL-235B-A22B-Instruct-GGUF_UD-Q4_K_XL_Qwen3-VL-235B-A22B-Instruct-UD-Q4_K_XL-00001-of-00003.gguf --rpc 192.168.177.12:15001 -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes
build: dd0f321 (7121) I also tried gpt-oss-120b on dual Sparks briefly - haven't tried llama-bench yet, but on test generation I've got 47 t/s on dual Sparks vs. 57 t/s on a single one. I feel like the performance can be improved if RPC backend gets NCCL support, as TCP/IP stack adds a lot more latency - we are talking 1-2 microsecond latency as measured by |
Beta Was this translation helpful? Give feedback.
-
|
@rgerganov - suddenly realized that I was running VLLM over Ethernet all this time. Still over the 200G connection, but NCCL was not using Infiniband due to missing libraries in my container. After I fixed that, got improvement and also a good comparison between NCCL using ethernet and IB/RoCE over the same physical interface => reinforces the fact that there is a dramatic difference in latency. The example below is a "fast" model - the one where network latency becomes a bottleneck. TL;DR: Using tensor parallel for the cluster:
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 --gpu-memory-utilization 0.5 --host 0.0.0.0 --port 8888 -tp 2 --distributed-executor-backend rayUsing NCCL over Ethernet: Using NCCL over Infiniband Single vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 --load-format fastsafetensors --gpu-memory-utilization 0.5 --host 0.0.0.0 --port 8888vllm bench serve --backend vllm --model RedHatAI/Qwen3-30B-A3B-NVFP4 --endpoint /v1/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100 --port 8888 |
Beta Was this translation helpful? Give feedback.
-
|
@eugr thanks a lot for these benchmarks, using the infiniband transport is clearly a huge boost compared to ethernet. I will start thinking how to abstract the transport layer in the RPC backend, so we can support other transports like infiniband/RDMA in addition to plain tcp/ip |
Beta Was this translation helpful? Give feedback.
-
|
@rgerganov - one other observation: disabling CPU idle states (or at least deep idle states) improves llama.cpp RPC performance on Spark due to keeping CPU cores in awake state => avoiding wake up latency penalty. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. How do you disable CPU idle states and how much improvement do you observe? The RPC backend is currently using blocking I/O and there is no way we can use busy loops unless we switch to async I/O. |
Beta Was this translation helpful? Give feedback.
-
sudo cpupower idle-set -D 0Although, it may be too aggressive, I may try -D 1. As for the difference, here is ping. eugr@spark:~/llm/llama.cpp$ ping 192.168.177.12
PING 192.168.177.12 (192.168.177.12) 56(84) bytes of data.
64 bytes from 192.168.177.12: icmp_seq=1 ttl=64 time=0.860 ms
64 bytes from 192.168.177.12: icmp_seq=2 ttl=64 time=1.12 ms
64 bytes from 192.168.177.12: icmp_seq=3 ttl=64 time=1.26 ms
64 bytes from 192.168.177.12: icmp_seq=4 ttl=64 time=1.16 ms
64 bytes from 192.168.177.12: icmp_seq=5 ttl=64 time=1.18 ms
64 bytes from 192.168.177.12: icmp_seq=6 ttl=64 time=1.16 ms
64 bytes from 192.168.177.12: icmp_seq=7 ttl=64 time=1.14 ms
64 bytes from 192.168.177.12: icmp_seq=8 ttl=64 time=0.828 ms
64 bytes from 192.168.177.12: icmp_seq=9 ttl=64 time=0.327 ms
64 bytes from 192.168.177.12: icmp_seq=10 ttl=64 time=1.02 ms
64 bytes from 192.168.177.12: icmp_seq=11 ttl=64 time=1.05 ms
^C
--- 192.168.177.12 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10099ms
rtt min/avg/max/mdev = 0.327/1.010/1.264/0.250 msAfter disabling idle states: eugr@spark:~/llm/llama.cpp$ ping 192.168.177.12
PING 192.168.177.12 (192.168.177.12) 56(84) bytes of data.
64 bytes from 192.168.177.12: icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from 192.168.177.12: icmp_seq=2 ttl=64 time=0.032 ms
64 bytes from 192.168.177.12: icmp_seq=3 ttl=64 time=0.018 ms
64 bytes from 192.168.177.12: icmp_seq=4 ttl=64 time=0.022 ms
64 bytes from 192.168.177.12: icmp_seq=5 ttl=64 time=0.017 ms
64 bytes from 192.168.177.12: icmp_seq=6 ttl=64 time=0.029 ms
64 bytes from 192.168.177.12: icmp_seq=7 ttl=64 time=0.017 ms
64 bytes from 192.168.177.12: icmp_seq=8 ttl=64 time=0.015 ms
64 bytes from 192.168.177.12: icmp_seq=9 ttl=64 time=0.029 ms
64 bytes from 192.168.177.12: icmp_seq=10 ttl=64 time=0.025 ms
^C
--- 192.168.177.12 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9223ms
rtt min/avg/max/mdev = 0.015/0.030/0.101/0.024 msPerformance difference with gpt-oss-120b: build/bin/llama-bench -m ~/.cache/llama.cpp/ggml-org_gpt-oss-120b-GGUF_gpt-oss-120b-mxfp4-00001-of-00003.gguf --rpc 192.168.177.12:15001 -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0
build: 583cb83 (7157) WITH DISABLED IDLE STATES
build: 583cb83 (7157) |
Beta Was this translation helpful? Give feedback.
-
|
network topology: |
Beta Was this translation helpful? Give feedback.
-
|
Any chance this can be extended to an environment with mixed GPUs (Nvidia, AMD and possibly Metal, CPU)? NCCL and RCCL are not interoperable so would sending RPC messages across RoCE help? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
There is a reasonably priced MikroTik CRS812 DDQ switch that (in theory) may allow connecting up to 8 DGX Sparks. It has 2 x 400 GbE ports, 2 x 200 GbE and 8 x 50 GbE ports. I think two Sparks can be connected directly to 2 x 200 GbE ports, there are QSFP-DD to 2xQSFP56 splitter cables (400GbE to 2 x 200GbE) to connect four more to 2 x 400 GbE ports and perhaps with cables like this it would even be possible to connect two more Sparks via remaining 8 x 50 GbE ports. Not sure if this all would work out of the box, but it certainly looks promising. Has anyone tried it? |
Beta Was this translation helpful? Give feedback.
-
|
I had a guy message me on X who had got one for review and needed help setting it up. I think he got it working with a pair of DGX's eventually. the trick was there were some very specific settings to get the cable working. Here was the specific detail that he had missed : "2x200G, 4x100G, 8x50G, 8x25G, 2x40G, 8x10G and 8x1G: Must be set with a forced speed mode and auto-negotiation disabled." Probably not worth it as the performance of two of them doesn't seem that amazing but it would be a fun project if you don't mind spending the $$. I suspect that switch gets loud - although perhaps you could replace the fans with nocturnas |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov link to dgx-spark.md benchmark results in the top post is broken (it's missing dgx-spark directory). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Wow, the latest Blackwell optimizations in llama.cpp made noticeable bump in prompt processing on DGX Spark:
build: f5acfb2 (7535) |
Beta Was this translation helpful? Give feedback.
-
|
is anyone seeing slowdowns over time with llama-server and glm 4.7 flash on the spark? I have a small sample prompt that I use to get a feel for token generation speed of a model. I've only seen this with glm 4.7 flash so far. Happy to open an issue but I feel like I don't have enough info to be useful. It is repeatable however. I'd use another model but glm 4.7 flash is really really good with opencode. It is the best agent programming model that I have used so far. |
Beta Was this translation helpful? Give feedback.
-
|
The issue is most likely that you are using the unified KV cache with multiple server slots (the default). This is not yet optimized for the CUDA backend and leads to a slowdown with time. The simplest workaround is to disable the unified kv cache with: llama-server ... -np 1 |
Beta Was this translation helpful? Give feedback.
-
|
perfect I will add that. Thank you very much! |
Beta Was this translation helpful? Give feedback.
-
|
@ggerganov would love to get your feedback. We're adding benchmarks to llama.cpp including the experimental NCCL support. When NVIDIA started shipping DGX Spark in mid-October 2025, the pitch was basically: “desktop box, huge unified memory, run big models locally (even ~200B params for inference).” The fun part is how quickly the software + community benchmarking story evolved from “here are some early numbers” to a real, reproducible leaderboard. On Oct 14, 2025, ggerganov posted a DGX Spark performance thread in llama.cpp with a clear methodology: measure prefill (pp) and generation/decode (tg) across multiple context depths and batch sizes, using llama.cpp CUDA builds + llama-bench / llama-batched-bench. Fast forward: the NVIDIA DGX Spark community basically acknowledged the recurring problem (“everyone posts partial flags, then nobody can reproduce it two weeks later”), we've agreed on our community tools for runtime image building, orchestration, recipe format and launched Spark Arena on Feb 11, 2026. Top of the board right now (decode tokens/sec): |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Qwen3.5 35B A3B Q8 An "older" one Qwen3-Coder-Next Q8 Possible to leverage NVFP4 optimization available on the GB10? For reference I see people benching with mostly vllm here: https://spark-arena.com/leaderboard NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 seen at tg128@65535 70 tok/sec Recent news shows the Qwen3.5 serie support very well aggressive quantization (https://x.com/i/status/2025951400119751040) Meaning that this TQ1_0 could even been considered to run on the GB10. PS: qwen3-next recent perfs on Spark here |
Beta Was this translation helpful? Give feedback.
-
|
@gelim Where do you get your vllm data from? On the site that you posted, I see this: 
Which is 2192.54 t/s at 32k context. |
Beta Was this translation helpful? Give feedback.
-
This is with: ...built on a DGX Spark: Benchmark results are: |
Beta Was this translation helpful? Give feedback.
-
This is awesome to see! Any chance you could try and nvfp4 on the GB10? :D |
Beta Was this translation helpful? Give feedback.
-
|
no nvfp4 on the gb10 yet. |
Beta Was this translation helpful? Give feedback.
-
I took this one. 
There are others available with better tg speed (46-47 tok/sec) and worst pp speed (~2k ish). Their difference is:
E.g: diff between https://spark-arena.com/api/recipes/c091e5aa-6f0d-453e-aaa7-8c63b30f9df5/raw and https://spark-arena.com/api/recipes/80fced6b-fec1-4657-aa5b-adaaab7d09bb/raw |
Beta Was this translation helpful? Give feedback.
-
|
For latest llama.cpp build on DGX Spark, can people share their build flags? Curious whether we need to specify 120 or 121 for -DCMAKE_CUDA_ARCHITECTURES |
Beta Was this translation helpful? Give feedback.
-
|
I am not doing anything fancy. here is my rebuild script : https://github.com/openmarmot/tech-notes/blob/main/nvidia/DGX_Spark/llamacpp/rebuild_llama_cpp.sh |
Beta Was this translation helpful? Give feedback.
-
|
I am seeing faster pp and tg speed with latest builds, even versus just 3 days ago! Prior: Current: |
Beta Was this translation helpful? Give feedback.
-
|
Speaking about Spark performance, I stumbled on that: https://www.reddit.com/r/LocalLLaMA/s/Z0RzA1NAPJ This is claiming big perf boost by allowing K=64 tiles size hence fitting inside the 99KB SMEM for consumer grade Blackwell (like the Spark's GB10). This seems huge whenever full NVFP4 support will land. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
by looking a bit more at the PTX instruction in asm volatile(
"mma.sync.aligned.kind::mxf4.block_scale.scale_vec::2X.m16n8k64.row.col.f32.e2m1.e2m1.f32.ue8m0 "
"{%0, %1, %2, %3}, {%4, %5, %6, %7}, {%8, %9}, {%0, %1, %2, %3}, "
"%10, {0, 0}, %11, {0, 0};"
: "+f"(Dxi[0]), "+f"(Dxi[1]), "+f"(Dxi[2]), "+f"(Dxi[3])
: "r"(Axi[0]), "r"(Axi[1]), "r"(Axi[2]), "r"(Axi[3]), "r"(Bxi[0]), "r"(Bxi[1]), "r"(a_scale), "r"(b_scale));
[...]would confirm that K=64 is properly used in code path related to MMA for Blackwell architecture. |
Beta Was this translation helpful? Give feedback.
-
|
I'm skeptical of the post you mentioned. In general, relying on reddit posts is not recommended |
Beta Was this translation helpful? Give feedback.
-
|
in second sight it seems out of topic for llama.cpp. As I'm not really knowledgeable on the codebase at first I had the impression it could be beneficial. Glad to hear if my second comment by digging in the code is as well out of topic. |
Beta Was this translation helpful? Give feedback.
-
|
That is correct spot for mxfp4 multiplication, nvfp4 would also similar. I would be extremely surprised if cutlass/flash-infer didn't use this. |
Beta Was this translation helpful? Give feedback.
-
Compile logCFLAGS="-O3 -mcpu=native -mtune=native -fomit-frame-pointer -pipe" \
CXXFLAGS="-O3 -mcpu=native -mtune=native -fomit-frame-pointer -pipe" \
cmake -S . -B build \
-DGGML_CUDA=ON \
-DGGML_CUDA_F16=ON \
-DGGML_CUDA_FORCE_MMQ=ON \
-DGGML_NATIVE=ON \
-DGGML_LTO=ON \
-DGGML_OPENMP=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CUDA_ARCHITECTURES=121aQwen3-Coder-Next-MXFP4_MOEsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3-Coder-Next-GGUF/snapshots/ce09c67b53bc8739eef83fe67b2f5d293c270632/Qwen3-Coder-Next-MXFP4_MOE.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3-Coder-Next-GGUF/snapshots/ce09c67b53bc8739eef83fe67b2f5d293c270632/Qwen3-Coder-Next-MXFP4_MOE.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
Qwen3.5-35B-A3B-MXFP4_MOEsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3.5-35B-A3B-GGUF/snapshots/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-MXFP4_MOE.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3.5-35B-A3B-GGUF/snapshots/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-MXFP4_MOE.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
Qwen3.5-122B-A10B-MXFP4_MOEsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3.5-122B-A10B-GGUF/snapshots/51eab4d59d53f573fb9206cb3ce613f1d0aa392b/MXFP4_MOE/Qwen3.5-122B-A10B-MXFP4_MOE-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--Qwen3.5-122B-A10B-GGUF/snapshots/51eab4d59d53f573fb9206cb3ce613f1d0aa392b/MXFP4_MOE/Qwen3.5-122B-A10B-MXFP4_MOE-00001-of-00003.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
gpt-oss-20b-UD-Q4_K_XLsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--gpt-oss-20b-GGUF/snapshots/d449b42d93e1c2c7bda5312f5c25c8fb91dfa9b4/gpt-oss-20b-UD-Q4_K_XL.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--gpt-oss-20b-GGUF/snapshots/d449b42d93e1c2c7bda5312f5c25c8fb91dfa9b4/gpt-oss-20b-UD-Q4_K_XL.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
gpt-oss-120b-UD-Q4_K_XLsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--gpt-oss-120b-GGUF/snapshots/ff1a82da6ad466e32284fa3d2b86694db3204789/UD-Q4_K_XL/gpt-oss-120b-UD-Q4_K_XL-00001-of-00002.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--gpt-oss-120b-GGUF/snapshots/ff1a82da6ad466e32284fa3d2b86694db3204789/UD-Q4_K_XL/gpt-oss-120b-UD-Q4_K_XL-00001-of-00002.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
NVIDIA-Nemotron-3-Nano-4B-UD-Q4_K_XLsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--NVIDIA-Nemotron-3-Nano-4B-GGUF/snapshots/8e81be55c5aa3d63bb82b6ceec62d50805d9e1bb/NVIDIA-Nemotron-3-Nano-4B-UD-Q4_K_XL.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--NVIDIA-Nemotron-3-Nano-4B-GGUF/snapshots/8e81be55c5aa3d63bb82b6ceec62d50805d9e1bb/NVIDIA-Nemotron-3-Nano-4B-UD-Q4_K_XL.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapmain: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
NVIDIA-Nemotron-3-Super-120B-A12B-MXFP4_MOEsrc/llama-cpp/build/bin/llama-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/snapshots/036038fb30334a2d56a146c6f0d4871ab5edccbb/MXFP4_MOE/NVIDIA-Nemotron-3-Super-120B-A12B-MXFP4_MOE-00001-of-00003.gguf -fa 1 -d 0,4096,8192,16384,32768 -p 2048 -n 32 -ub 2048 -mmp 0ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
build: 7c20367 (8580) src/llama-cpp/build/bin/llama-batched-bench -m /home/sparky/.cache/huggingface/hub/models--unsloth--NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/snapshots/036038fb30334a2d56a146c6f0d4871ab5edccbb/MXFP4_MOE/NVIDIA-Nemotron-3-Super-120B-A12B-MXFP4_MOE-00001-of-00003.gguf -fa 1 -c 300000 -ub 2048 -npp 4096,8192 -ntg 32 -npl 1,2,4,8,16,32 --no-mmapggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB): main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 20, n_threads_batch = 20
|
Beta Was this translation helpful? Give feedback.
-
|
Adding KV cache quantization data to this thread. Tested --cache-type-k / --cache-type-v with f16, q8_0, and q4_0 on Nemotron 3 Nano 30B A3B (Q4_K_XL) at 128K context. Key finding: q4_0 KV cache has a devastating performance cliff at 64K+ context on Spark. Prompt processing drops 92.5% (282.7 to 21.3 tok/s) due to dequantization overhead. q8_0 is the sweet spot: 2x compression with under 5% speed hit at all context lengths. Interestingly, q4_0 uses ~6% MORE RSS than f16 on Spark's unified memory. The per group scale/zero point metadata overhead exceeds the compression savings. Config Context Prompt tps Gen tps RSS For most Spark workloads, f16 is the right default. 128GB unified memory means there's no KV cache memory pressure to solve. The exception is extreme concurrency or 500K+ context, where q8_0 makes sense. Full writeup: https://www.linkedin.com/pulse/i-benchmarked-kv-cache-quantization-my-dgx-spark-heres-nathan-maine-szxtc |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Overview
This document summarizes the performance of
llama.cppfor various models on the new NVIDIA DGX Spark.Benchmarks include:
pp) and generation (tg) at various context depths (d)Models:
gpt-oss-20bgpt-oss-120bQwen3 Coder 30B A3BQwen2.5 Coder 7BGemma 3 4B QATGLM 4.5 AirFeel free to request additional benchmarks for models and use cases.
Benchmarks
Build with:
Using the following commands:
Results
https://github.com/ggml-org/llama.cpp/blob/master/benches/dgx-spark/dgx-spark.md
Evals
gpt-oss-120b,high:93.75%History
2025 Oct 14 (b6761)7ea15bb Initial numbers2025 Oct 15 (b6767)5acd455 Improved decode via CUDA: Changing the CUDA scheduling strategy to spin #165852025 Oct 26 (b6845)73a48c9 Various improvements + disabled mmap2025 Nov 09 (b6989)eeee367 Various improvements2026 Feb 05 (b7946)3795cc1 Various improvementsModel loading performance
More info
Beta Was this translation helpful? Give feedback.
All reactions