🏆 Community Benchmark Challenge — Share Your Hardware Results vs llama.cpp

[shifulegend]

Why We're Asking

Project Zero is a from-scratch CPU-only LLM inference engine in C — no GPU, no Python, no external dependencies. We've benchmarked extensively on two machines (Intel i5-11300H laptop and a Xeon server), but every CPU is different. Memory bandwidth, L3 cache size, SIMD tier, DRAM configuration — all of these affect inference speed in ways that are hard to predict analytically.

We're building a community benchmark table to:

1. Understand how Project Zero performs across the hardware landscape
2. Compare directly against llama.cpp on the same machine (the most meaningful comparison)
3. Identify where Project Zero is fast, slow, or broken on hardware we haven't tested
4. Guide optimization priorities — if most community hardware is Zen 3 AMD with no VNNI, that changes what we build next

This is a crowdsourced leaderboard. Your result matters.

---

What to Run

Step 1 — Clone and build (30 seconds)

git clone https://github.com/shifulegend/project-zero
cd project-zero
make release

Requirements: GCC or Clang, pthreads, libm. Nothing else.

Step 2 — Get a model

BitNet (recommended — fastest, smallest download):

# Download from: https://huggingface.co/microsoft/bitnet-b1.58-2B-4T
# Convert to .bin format (one-time, ~2 min):
python3 tools/convert_hf_bitnet.py \
  --model models/microsoft-bitnet-b1.58-2B-4T/ \
  --output models/bitnet-b1.58-2B-4T.bin
python3 tools/convert_tokenizer.py \
  --input  models/microsoft-bitnet-b1.58-2B-4T/tokenizer.json \
  --output models/bitnet-b1.58-2B-4T_tokenizer_proper.bin

DeepSeek-V2-Lite-Chat GGUF (tests MoE path):

# Download: DeepSeek-V2-Lite-Chat-Q4_K_S.gguf from HuggingFace
# No conversion needed — load directly

Step 3 — Run the benchmark

# BitNet benchmark (use --verbose to see SIMD/classifier detected)
./adaptive_ai_engine \
  --model models/bitnet-b1.58-2B-4T.bin \
  --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin \
  --prompt "List the first 20 prime numbers:" \
  --max-tokens 100 \
  --temperature 0.0 \
  --threads 4 \
  --verbose

# Repeat with --threads 1, 2, 4, 8 (or your physical core count)
# Note the tok/s from each run

# DeepSeek benchmark (if you have the GGUF)
./adaptive_ai_engine \
  --model models/deepseek-v2-lite-chat-Q4_K_S.gguf \
  --prompt "What is the capital of France?" \
  --max-tokens 30 \
  --temperature 0.0 \
  --threads 4 \
  --verbose

Step 4 — Run llama.cpp on the same model (for comparison)

# Build llama.cpp if you don't have it:
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make -j4

# BitNet via llama.cpp:
./llama-bench -m /path/to/DeepSeek-V2-Lite-Chat-Q4_K_S.gguf -t 4 -p 0 -n 50 -r 3

# Or for generation:
./llama-cli -m /path/to/model -t 4 -n 50 --temp 0 \
  -p "List the first 20 prime numbers:"

Step 5 — Post your results below using this template

---

📋 Result Template

Copy and fill this in your reply:

### My Result

**Hardware:**
- CPU: [e.g. Intel Core i7-12700H, AMD Ryzen 9 5900X, Apple M2 Pro]
- Cores: [physical cores / logical threads — e.g. 6P+2E / 20T]
- RAM: [total GB, type, channels — e.g. 32 GB DDR5-4800 dual-channel]
- L3 Cache: [e.g. 24 MB]
- OS: [e.g. Ubuntu 22.04, macOS 14.5, Windows 11 WSL2]
- Compiler: [gcc --version output]

**SIMD Detected by Project Zero:** [from --verbose header, e.g. AVX-512 VNNI / AVX2 / NEON]
**Classifier Auto-Selected:** [e.g. INT4 / INT8 / BF16]

**Project Zero — BitNet b1.58-2B-4T:**
| Threads | tok/s |
|---------|-------|
| T=1 | |
| T=2 | |
| T=4 | |
| T=8 | |
Best config: T=?, SIMD=?, CLS=? → **X.XX tok/s**

**llama.cpp — Same model, same machine:**
| Threads | TG tok/s (token generation) |
|---------|------------------------------|
| T=4 | |

**Project Zero vs llama.cpp ratio:** X.X× [faster/slower]

**DeepSeek-V2-Lite-Chat Q4_K_S (if tested):**
| Engine | T=4 tok/s |
|--------|-----------|
| Project Zero | |
| llama.cpp | |

**Notes / Observations:**
[Anything unusual? Crashes? Wrong output? Unexpected SIMD selection?]

---

📊 Community Results — Running Table

I'll update this table as results come in.

Contributor	CPU	RAM	SIMD	PZ BitNet (best)	llama.cpp BitNet	PZ/llama ratio
(your result here)

---

Reference: Our Own Benchmarks (Baseline)

Machine	CPU	RAM	SIMD	PZ tok/s	llama.cpp tok/s	Ratio
Dev laptop	i5-11300H (Tiger Lake, 4P/8T)	16 GB DDR4 dual-ch	AVX-512 VNNI	16.1	~19.7	0.82×
Xeon server	Xeon @ 2.10 GHz (Emerald Rapids, 4C)	260 MiB L3	AVX-512 VNNI	36.25	~19.8 (bitnet.cpp)	1.83×
MacBook Air	i5-5250U (Broadwell, 2C/4T)	8 GB DDR3	SSE4 only	~3.2	~6.0	0.53×

Key observation: On a memory-bandwidth-constrained laptop, llama.cpp is ahead. On a server with high L3 and AVX-512 VNNI, Project Zero wins by 1.83×. We want to know where your machine lands.

---

What We'll Do With Your Data

• Update the Community Results table in this discussion
• Add standout results to BENCHMARK_REPORT.md in the repo
• Use the data to prioritize optimization work — if Project Zero underperforms on a common CPU family, that becomes a target
• Call out the fastest community submission 🏆

---

Known Quirks to Watch For

• T=8 on Tiger Lake/Ice Lake: AVX-512 + hyperthreading causes frequency throttle — T=4 is usually faster. Project Zero auto-detects this.
• ARM/Apple Silicon: NEON path is functional but less optimized than the AVX-512 path. Results on M-series chips are especially interesting.
• Windows native (non-WSL): Build with CMake — cmake -B build && cmake --build build --config Release. Let us know if you hit build issues.
• DeepSeek MoE: Currently 13× behind llama.cpp. We know. See Discussion #1.

---

Questions?

Ask below — happy to help you get the build working or interpret your results.

Every data point helps. Even "it crashed on my machine" is useful information.

[shifulegend]

Dropping the reference numbers here to seed the table.

Intel Xeon (Emerald Rapids, 4-core, 32GB DDR5-5600, single-channel):

Model	PZ	llama.cpp	delta
BitNet b1.58-2B-4T (T=4)	36.25 tok/s	19.77 tok/s	+83%
SmolLM2 360M F16 (T=3)	83.79 tok/s	65.92 tok/s	+27%
SmolLM2 360M F16 (T=4)	73.73 tok/s	49.23 tok/s	+50%
DeepSeek-V2-Lite Q4_K_S	1.90 tok/s	13.79 tok/s	-86% ⚠️

Third-party verification: https://openbenchmarking.org/result/2606207-SHIF-PROJECT42

Intel i5-11300H (Tiger Lake, 4-core, 16GB DDR4):

Model	PZ tok/s
BitNet b1.58-2B-4T (T=4)	51.74

The MoE number is honest - DeepSeek-V2-Lite's expert scatter is a DRAM bandwidth problem we haven't solved yet. BitNet and dense GGUF are where the speedups are.

For submissions: mention CPU, core count, DRAM type/config, and include the llama.cpp baseline on the same machine. Cross-machine tok/s comparisons are hard to interpret without knowing the DRAM bandwidth tier.