Compare NVIDIA GPUs on different tasks

The main goal of this repo is to benchmark the NVIDIA DGX Spark (also known as GB10) against the NVIDIA RTX 4090 on a series of ML/AI tasks.

Why these two GPUs?

They are the two GPUs I own.

But the tests could easily be extended to other GPUs.

Most of the analysis in benchmark_analysis/ was performed by GPT-Codex simply to compare the results in benchmark_results/ (this holds the ground truth for a series of tests in .csv files).

These benchmarks are mostly targeted at simple single user usage rather than multiple users at the same time.

For example, a single developer/researcher working on their own machine.

None of these tests assess the quality of the outputs created by the models. They are only focused on speed.

Benchmark overview

• LLM/VLM inference - Compare LLM/VLM inference speed with llama.cpp and vLLM inference engines across various models.
• LLM training - Compare LLM full fine-tuning training speed with a relatively small LLM (Gemma-3-270M).
• Image generation - Compare image generation speeds with Flux.2-klein-4B and Z-Image-Turbo.
• Object detection - Fine-tune an object detection model.

Results: See the results in benchmark_analysis/README.md.

Summary

One line: RTX 4090 faster on training and inference by ~3-4x but cannot run larger models or very long contexts like the DGX Spark (24GB VRAM vs 128GB VRAM).

Performance

The NVIDIA DGX Spark (GB10) is a like a minivan.

Whereas the NVIDIA RTX 4090 is like Ferrari.

What I mean by this is that for any compute intensive task, the RTX 4090 is generally faster (on training and inference) than the DGX Spark on an order of 3-4x.

However, there are several benchmarks (e.g. running gpt-oss-120b) where the RTX 4090 simply can't because it doesn't have enough space (VRAM).

So the DGX Spark while generally slower than the RTX 4090 can run much larger models with a much larger context.

Getting started and machine footprint

DGX Spark is one plug and play (it is a full system in one box). Go from unboxing to running models in ~30 mins. Especially with the NVIDIA DGX Spark website tutorials.

RTX 4090 is a GPU card only and requires a custom built PC to house it (this took me and my friend ~3-4 hours) as well as software setup (~2-3 hours with the help of AIs).

The DGX Spark also has a small footprint, similar to the size of a small textbook.

The RTX 4090, depending on your PC build will have a much larger footprint.

Which one?

• You: Want quickest possible setup to go from 0 to running AI models on my desktop = DGX Spark.
• You: Want fastest GPU for fine-tuning/running smaller models and don't mind building a PC = RTX 4090.
• You: Want to run large models (e.g. >8B parameters) and want lots of context tokens = DGX Spark.
• You: Want to create large amounts of synthetic data with large models + distill synthetic data into smaller models via fine-tuing = Get both (DGX Spark for synthetic data generation -> RTX 4090 for small model fine-tuning).

Specs

GPU specs

The two GPUs I'm focused on are the NVIDIA RTX 4090 and NVIDIA DGX Spark.

These are their comparative specs.

Notice the RTX 4090 has more CUDA cores (about 3x) and more memory bandwidth (about 4x) but lower overall memory capacity.

Metric (AI/ML oriented)	NVIDIA RTX 4090	NVIDIA DGX Spark
NVIDIA official specs page	https://www.nvidia.com/en-au/geforce/graphics-cards/40-series/rtx-4090/	https://www.nvidia.com/en-au/products/workstations/dgx-spark/
CUDA cores	16,384 CUDA cores	6,144 CUDA cores
Device class	Discrete desktop GPU	Desktop AI computer (Grace Blackwell “GB10” superchip system)
Peak “AI” advertised throughput	1321 AI TOPS	1000 AI TOPS (theoretical FP4 using sparsity)
Best-supported low precision for tensors	Supports FP8 tensor format (Ada 4th-gen Tensor Cores)	Supports FP4 (5th-gen Tensor Cores)
FP8 peak (vendor-stated)	NVIDIA states “1.3 petaFLOPS” for AI inference with FP8 (Ada architecture PDF)	Not clearly published as FP8 PFLOPS in the official datasheet/user guide (only FP4/TOPS headline is stated)
FP32 (non-tensor) peak	83 TFLOPS (NVIDIA product page)	Not disclosed as a peak FP32 figure in the official datasheet/user guide
GPU memory capacity	24 GB GDDR6X	128 GB unified LPDDR5x (CPU+GPU coherent unified memory)
Memory bandwidth (device memory)	1008 GB/s	273 GB/s. Source (datasheet PDF)
VRAM Practicality	VRAM can be a hard limit for bigger LLMs without offload/quantization (24GB VRAM)	Officially positioned for ~200B parameter models on one unit; up to ~405B with two systems linked
CPU / host included	Requires a separate PC/CPU	Built-in 20-core Arm CPU (10 Cortex-X925 + 10 Cortex-A725)
Interconnect / scaling	Desktop PCIe	Includes ConnectX-7 SmartNIC and is designed to link two DGX Spark units for bigger models
Power	450W board power/TDP widely reported	Datasheet lists power consumption as TBD
Storage	Not applicable (GPU only)	1TB or 4TB NVMe M.2 (self-encryption)
Networking	Not applicable (GPU only)	10 GbE, Wi-Fi 7, ConnectX-7. Source (datasheet PDF)
Best fit (typical AI workflow)	Strong single-GPU performance for training/fine-tune/inference that fits in 24GB VRAM, high memory bandwidth helps throughput-sensitive workloads	Best when workloads benefit from large unified memory (large models + context) and an integrated AI workstation setup

My deep learning PC specs

I built this at the start of 2024 with my friend Michael.

Chances are some parts have newer versions in 2026 (time of running the benchmarks), however, these are the actual parts I used.

Part Type	Part Name
CPU	Intel Core i9-14900K 3.2 GHz 24-Core Processor
CPU Cooler	Noctua NH-U12A chromax.black 60.09 CFM CPU Cooler
Motherboard	Gigabyte Z790 AORUS MASTER EATX LGA1700 Motherboard
Memory	Corsair Vengeance 96 GB (2 x 48 GB) DDR5-6400 CL32 Memory
Storage	Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 x4 NVME Solid State Drive
Video Card	NVIDIA Founders Edition GeForce RTX 4090 24 GB Video Card
Case	Corsair 7000D AIRFLOW ATX Full Tower Case
Power Supply	Corsair AX1600i 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply

Requirements and Setup

• Create a Python 3.12 virtual environment (can use conda or uv if you want) and install dependencies:

python3.12 -m venv .venv
source .venv/bin/activate
pip install torch transformers datasets diffusers

• Install llama.cpp - https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md
  • Setup llama server - ggml-org/llama.cpp#16938
• Install vLLM - https://docs.vllm.ai/en/stable/getting_started/installation/gpu/
• Helpful: DGX Spark user guide and setup settings - https://build.nvidia.com/spark

LLM/VLM Inference

The main goal here is to test token/s throughput and output.

We're going to use two different inference engines here.

vLLM and llama.cpp.

We could use native Hugging Face Transformers, however, this is often much slower than an optimized inference engine.

If you want to just use an LLM locally, I'd suggest opting for something like llama-server as it offers a nice chat interface.

If you want to run large inference jobs and unlock batch inference, use vLLM.

vLLM benchmarking (text only and image + text)

Setup steps:

• Install Docker: https://docs.docker.com/engine/install/ubuntu/
• Setup vLLM container on GPU via NVIDIA: https://build.nvidia.com/spark/vllm/instructions
• Run benchmarks .sh scripts, these are slightly modified versions of the vLLM multimodal benchmarking examples: https://docs.vllm.ai/en/stable/benchmarking/cli/#multi-modal-benchmark

Note

All tests done with NVIDIA vLLM container version: 25.12.post1-py3, this requires a docker installation.

So you can use: export LATEST_VLLM_VERSION=25.12.post1-py3

Example command for starting a vLLM server:

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} vllm serve "MODEL_NAME" --gpu-memory-utilization 0.8 --max_model_len 32000

Then you can run the tests, these are executed via a bash script vllm_image_and_text_testing.sh or vllm_text_only_testing.sh which runs a set of tests:

• 0 images + random range of input/output tokens
• 1 images + random range of input/output tokens (note: images are of various sizes randomly sampled to enable real-world test-like conditions)
  • These tests are designed to mimic image + text throughput as well as text only throughput

100 prompts each (~1000 input, ~300 output) with 10 warmup prompts (these are not measured in final output performance).

Nemotron3-Nano-FP8 (text only)

Source: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

Note: Requires nano_v3_reasoning_parser.py.

Download nano_v3_reasoning_parser.py (required for using Nemotron v3 versions):

wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8/resolve/main/nano_v3_reasoning_parser.py

Start server:

docker run -it --gpus all \
  -p 8000:8000 \
  -v $(pwd)/nano_v3_reasoning_parser.py:/workspace/nano_v3_reasoning_parser.py \
  nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 32000 \
  --port 8000 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3 \
  --kv-cache-dtype fp8

Run benchmark:

./vllm_text_only_testing.sh nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

Qwen3-VL-32B-Instruct-FP8 (text and image)

Note: You could only run this on the NVIDIA DGX Spark (too large for the RTX 4090).

Tests done: 0 images and 1 image input.

Start vLLM server:

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} vllm serve "Qwen/Qwen3-VL-32B-Instruct-FP8" --gpu-memory-utilization 0.8 --max_model_len 32000

Benchmark:

./vllm_image_and_text_testing.sh Qwen/Qwen3-VL-32B-Instruct-FP8

Qwen3-VL-30B-A3B-Instruct-FP8 (text and image)

Note: You could only run this on the NVIDIA DGX Spark (too large for the RTX 4090).

Start server:

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} vllm serve "Qwen/Qwen3-VL-30B-A3B-Instruct-FP8" --gpu-memory-utilization 0.8 --max_model_len 32000

Benchmark:

./vllm_image_and_text_testing.sh Qwen/Qwen3-VL-30B-A3B-Instruct-FP8

Qwen3-VL-8B-Instruct-FP8 (text and image)

Note: This model is compatible with the RTX 4090 and DGX Spark.

Start server:

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} vllm serve "Qwen/Qwen3-VL-8B-Instruct-FP8" --gpu-memory-utilization 0.8 --max_model_len 8000

Benchmark:

./vllm_image_and_text_testing.sh Qwen/Qwen3-VL-8B-Instruct-FP8

openai/gpt-oss-20b (text only)

Note

gpt-oss-20b & gpt-oss-120b models require the OpenAI Harmony package: https://github.com/openai/harmony

This model is compatible with both the RTX 4090 & DGX Spark.

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:25.12.post1-py3 \
bash -c "pip install openai-harmony && vllm serve 'openai/gpt-oss-20b' --gpu-memory-utilization 0.8 --max_model_len 8000"

Benchmark:

./vllm_text_only_testing.sh openai/gpt-oss-20b

openai/gpt-oss-120b (text only)

Note: You could only run this on the NVIDIA DGX Spark (too large for the RTX 4090).

docker run -it --gpus all -p 8000:8000 nvcr.io/nvidia/vllm:25.12.post1-py3 \
bash -c "pip install openai-harmony && vllm serve 'openai/gpt-oss-120b' --gpu-memory-utilization 0.8 --max_model_len 32000"

Benchmark:

./vllm_text_only_testing.sh openai/gpt-oss-120b

llama.cpp running in a server

Setup:

• See the guide for creating a llama.cpp server on DGX Spark: https://build.nvidia.com/spark/nemotron/instructions
• See the docs for llama.cpp HTTP Server: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

gpt-oss-20b-GGUF (text only)

Link: https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-UD-Q4_K_XL.gguf

Download model:

hf download unsloth/gpt-oss-20b-GGUF gpt-oss-20b-UD-Q4_K_XL.gguf --local-dir ~/models/gpt-oss-20b-gguf/

Run a server:

./bin/llama-server \
  --model ~/models/gpt-oss-20b-gguf/gpt-oss-20b-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 \
  --port 30000 \
  --n-gpu-layers 99 \
  --ctx-size 16384 \
  --threads 8 \
  --jinja \
  --temp 1.0 \
  --top-p 1.0 \
  --top-k 0

Run the text only benchmark:

python bench_llama_cpp_text_only.py

Qwen3-VL-8B-GGUF (text and image)

Source: https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF/blob/main/Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf

Download a model:

hf download unsloth/Qwen3-VL-8B-Instruct-GGUF \Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf --local-dir ~/models/
qwen3-vl-8b-gguf/

For multimodal, download the multimodal projector:

hf download unsloth/Qwen3-VL-8B-Instruct-GGUF \mmproj-BF16.gguf --local-dir ~/models/
qwen3-vl-8b-gguf/

Run a server (with multimodal capabilities):

./bin/llama-server \
    --model ~/models/qwen3-vl-8b-gguf/Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf \
    --mmproj ~/models/qwen3-vl-8b-gguf/mmproj-BF16.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --presence-penalty 1.5 \
    --ctx-size 8192 \
    --host 0.0.0.0 \
    --port 30000

With flash attention on:

./bin/llama-server \
    --model ~/models/qwen3-vl-8b-gguf/Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf \
    --mmproj ~/models/qwen3-vl-8b-gguf/mmproj-BF16.gguf \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --flash-attn auto \
    --presence-penalty 1.5 \
    --ctx-size 8192 \
    --host 0.0.0.0 \
    --port 30000

Run the text only benchmark:

python bench_llama_cpp_text_only.py

Run the multimodal (text + image) benchmark (requires a multimodal model):

python bench_llama_cpp_text_and_image.py

Nemotron-3-Nano

Source: https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF/blob/main/Nemotron-3-Nano-30B-A3B-UD-Q8_K_XL.gguf

Note: Requires 40GB download for Q8 model. Could use the Q4 model for less memory requirements.

Download a model:

hf download unsloth/Nemotron-3-Nano-30B-A3B-GGUF \
  Nemotron-3-Nano-30B-A3B-UD-Q8_K_XL.gguf \
  --local-dir ~/models/nemotron3-gguf

Run a server:

./bin/llama-server \
  --model ~/models/nemotron3-gguf/Nemotron-3-Nano-30B-A3B-UD-Q8_K_XL.gguf \
  --host 0.0.0.0 \
  --port 30000 \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --threads 8

Run the text only benchmark:

python bench_llama_cpp_text_only.py

LLM fune-tuning

• Overview: Fully fine-tune a small language model (Gemma-3-270m) on ~1k samples for 3 epochs.
• Source: https://www.learnhuggingface.com/notebooks/hugging_face_llm_full_fine_tune_tutorial (converted notebook into a Python script)
• See: bench_train_llm_fine_tune.py for settings.

Run the benchmark:

python bench_train_llm_fine_tune.py

Image generation

• Overview: Generate ~100 images with random input prompts time how long each takes to generate.
• Source: black-forest-labs/FLUX.2-klein-4B & Tongyi-MAI/Z-Image-Turbo.
• See: bench_z_image_turbo.py & bench_flux.py for settings.

Run the Flux.2 benchmark:

python bench_flux.py

Run the Z-Image-Turbo benchmark:

python bench_z_image_turbo.py

Training an object detection model

• Overview: Fine-tune an object detection model (RT-DETRv2) on a custom dataset of ~1.1k images for 10 epochs.
• Source: https://www.learnhuggingface.com/notebooks/hugging_face_object_detection_tutorial (converted notebook into a Python script)
• See: bench_train_object_detection.py for settings.

python bench_train_object_detection.py

Extras

• EXO can run LLMs with Mac Studio + NVIDIA DGX Spark combined for optimal inference (DGX Spark for prefill, Mac Studio for decode) - https://blog.exolabs.net/nvidia-dgx-spark/

Notes and potential extensions

• Extension: I have a Mac Mini M4 Pro, I wonder how it goes with MLX? That will be something to look into in the future.
• Defaults where possible: I tried to use the default settings on all available repos if they were available. In the future, benchmarks would likely take into account different specific settings for different scenarios (e.g. long context input/long context output etc).
• Extension: Upgrade the amount of tokens used in context to see how the devices perform under higher load (e.g. large codebase in the context window).
• Extension: Use llama-bench for benchmarking llama.cpp servers: https://blog.steelph0enix.dev/posts/llama-cpp-guide/#llama-bench
• Extension: Get some numbers on NVFP4 (in my brief research, it's still not as fleshed out as I'd like), see example: https://build.nvidia.com/spark/nvfp4-quantization/instructions
• Analogy for DGX Spark vs RTX 4090 = RTX 4090 is a Ferrari, fast but not much storage. DGX Spark is like a minivan, plenty of storage, not that fast.
• Does Docker slow down inference? (e.g. using NVIDIA's approaved/signed Docker container but does this make things slower?), hat tip: @ibrahimadiallo7444
  • Using this container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm?version=25.12.post1-py3
• Check these docs for possible improvements: https://docs.vllm.ai/en/stable/configuration/optimization/#configuration
• For multimodal, is it best for image/text first?
• For large inference runs, if you wanted to use the same prompt over and over again on different samples, what is the best way to cache the prompt?
• For different image sizes in vLLM, getting different processing speeds, this was done intentionally to reflect the different image sizes in the wild
  • In practice, if you wanted to maintain the highest throughput, you would likely normalize all the image sizes (make them the same sizes)
• Why not GLM 4.7 Flash? At the time of recording (late January 2026, vLLM support GLM 4.7 Flash on their main branches but I'm using NVIDIA's vLLM container)
  • "vLLM and SGLang only support GLM-4.7-Flash on their main branches."
  • I'd expect the inference to be similar speeds to Qwen3-VL-30B-A3B-Instruct-FP8
  • Unsloth makes GLM Flash easier - https://unsloth.ai/docs/models/glm-4.7-flash
• Could I increase maximum request concurrency? Currently set to default of 10 but perhaps this could be higher/better?
• If you have any tips of what settings I could use to improve vLLM througput, please let me know.
• Try another inference engine? SGLang? How does this go?
  • See here: https://build.nvidia.com/spark/sglang/overview
• First time running NVIDIA-Nemotron3-Nano, thinking was enabled so it output 10x more output tokens total compared to GPT-OSS-120B.
• Better flash attention settings = better results?
  • Flash attention 2 seems questionable on the DGX Spark (as of January 2026)
  • Depending on your platform/task, look into the best settings for that task
• Leave a comment for which model you'd like to see benchmarked next (optional: and benchmarked what on), I'll add it to the repo