[General]: Details about benchmarking process

[saba-er]

Issue description
It seems the details of benchmarking process is not available.
Context
In the benchmarking process, it is not clear what queries are used, how they are generated, and where the responses are logged. What are the roles of PP and TG configurations? Do these parameters set upper bounds on input prompt length (including query) and size of output tokens? It would be very good to clarify a bit with an example to show how this works. Thanks!

[a-ghorbani]

Benchmark Process

PocketPal's benchmark is based on llama.cpp's built-in benchmarking approach.
It measures raw computational throughput - how fast your device can run the model's neural network calculations (not response quality).

Parameters

Parameter	Description	Default	Range
PP	Prompt Processing tokens	512	64-4096
TG	Text Generation tokens	128	32-2048
PL	Pipeline Parallel (sequences)	1	1-4
NR	Number of Repetitions	3	1-10

How It Works

The benchmark does not use actual text prompts. Instead, it generates random token IDs from the model's vocabulary.

Two-Phase Measurement

Phase 1: Prompt Processing (PP)

• Creates a batch of pp random tokens
• Feeds the entire batch to the model at once via llama_decode()
• Measures total time to process
• Result: speed_pp = prompt_tokens / time (tokens/second)

This simulates what happens when you send a message - the model processes your entire input in one pass.

Phase 2: Text Generation (TG)

• Generates tg tokens sequentially, one at a time
• Each iteration: add 1 token → decode → measure
• Result: speed_tg = generated_tokens / time (tokens/second)

This simulates the AI "typing" its response - each token depends on all previous tokens, so they must be generated one-by-one.

Warmup & Averaging

Before measuring, the benchmark runs a 16-token warmup to initialize GPU/NPU resources and stabilize performance.
The entire PP+TG cycle runs nr times (default 3), and results are averaged to reduce variance.

What The Numbers Mean

Metric	What It Measures	User Experience
PP speed	Input processing rate	How long before AI starts responding
TG speed	Output generation rate	How fast text appears ("typing speed")

TG speed is what matters most for chat. Reference points:

• 10+ t/s: Fast
• 7-8 t/s: Conversational
• 2-4 t/s: Sluggish, requires patience
• <2 t/s: Frustratingly slow

Why PP is Faster Than TG

PP processes tokens in parallel batches (GPU-friendly), while TG is inherently sequential (each token needs the previous one).
This is why you'll see PP speeds of 50-200+ t/s but TG speeds of 5-15 t/s on the same device.

Benchmark Configuration Info

The benchmark also reports device configuration:

• n_threads / n_threads_batch: CPU threads used
• n_gpu_layers: Layers offloaded to GPU
• flash_attn: Flash attention setting (0=off, 1=on, 2=auto)
• n_batch / n_ubatch: Batch sizes

Reproducibility

Same model + same quantization + same settings = comparable results.
The random tokens don't affect speed measurements since the neural network performs the same calculations regardless of token content.

---

Sources

• llama.rn: cpp/rn-completion.cpp#L575-L788
• llama.cpp: tools/llama-bench/llama-bench.cpp