Gemma 4 DGX Spark Performance Benchmarks

This repository contains the official performance benchmarks and capability demonstrations for the Gemma 4 model family (E2B, E4B, 26B-A4B MoE, and 31B).

The tests were conducted on a DGX Spark system (NVIDIA GB10, 121GB VRAM) using the official optimized weights and transformers v5.5.0. These results demonstrate the resolution of early-access bottlenecks, particularly the "MoE Paradox," where the sparse 26B model now significantly outperforms the 31B dense model.

Source: A Visual Guide to Gemma 4

Key Results

Task	E2B-it (TPS)	E4B-it (TPS)	26B-A4B MoE (TPS)	31B Dense (TPS)
Text Generation	28.00	15.69	12.52	3.40
Thinking Mode (CoT)	28.73	15.93	12.48	3.40
Vision (Multimodal)	27.69	15.67	12.24	3.22
Function Calling	28.76	16.10	11.55	3.44

Repository Structure

Core Scripts

• gemma4_benchmark_master.py: The unified benchmark engine. It handles model loading (official HF paths), automated testing across 7 capability pillars, and auto-generates demonstration notebooks.
• install.sh: A shell script to set up the environment, install the correct version of transformers and dependencies (using uv).

Artifacts & Results

• master_benchmark_results.json: The complete raw data from the latest runs, including precise TPS metrics and the actual text/code generated by each model.
• gemma-4-*-core-capabilities.ipynb: Replicated Jupyter Notebooks for each model size, showing hand-on examples of vision, thinking mode, and tool use.

Documentation

• results_summary.md: A high-level overview of the performance metrics.
• moe_investigation.md: A technical deep-dive into how the official weights resolved previous MoE architectural bottlenecks.
• medium_article.md: A full-length article summarizing the market impact and technical takeaways of the Gemma 4 series.

Installation

1. Ensure you have uv installed (or the script will attempt to install it for you).
2. Run the installation script:

   chmod +x install.sh
   ./install.sh

3. Hugging Face Login: You will need access to the Gemma 4 models on Hugging Face.

   huggingface-cli login

Running the Benchmarks

You can run the full suite or target specific models using the master script.

Run all models:

python gemma4_benchmark_master.py

Run a specific model (e.g., E4B):

python gemma4_benchmark_master.py E4B

Capability Pillars Tested:

1. Creative Text: Single-turn poem generation.
2. Analytical Text: Multi-turn technical explanations.
3. Thinking Mode: Native Chain-of-Thought reasoning using <|think|> tokens.
4. Vision (Single): High-resolution image description.
5. Vision (Multi): Comparative analysis across interleaved images.
6. Visual Logic: Biological trait validation (Object counting/attributes).
7. Function Calling: Zero-shot tool use for weather and agentic tasks.

Technical Notes

• Precision: All tests are run in bfloat16.
• Framework: Requires transformers >= 5.5.0 for native Gemma 4 MoE support.
• Thinking Mode: Enabled via enable_thinking=True in the processor's chat template.