Benchmarking Guide

Benchmarking Guide

Use this guide for a small, reproducible TinyTitan measurement. Do not compare a result with another row unless the model, prompt, generated tokens, settings, and stop condition match.

1. Prepare

• Connect a laptop to power and disable Low Power Mode.
• Quit unrelated heavy workloads.
• Use a completed, verified model installation.
• Check memory pressure and require an empty model-process check.

memory_pressure -Q
pgrep -fl 'TinyTitanServer|TinyTitanCLI|TinyTitanPackageTests|swiftpm-testing-helper|mlx_lm|mlx-lm'

Do not kill an existing process merely to run a benchmark. Build once:

swift build -c release

2. Record the environment

git rev-parse HEAD
git status --short
sw_vers
swift --version
system_profiler SPHardwareDataType |
  awk -F': ' '/Model Name|Model Identifier|Chip|Total Number of Cores|Memory/ { print $1 ": " $2 }'

The filtered hardware command omits serial numbers and hardware UUIDs.

3. Run a fixed case

Use one discarded warmup, then three fresh-process measurements with the same command. Normal benchmark requests use the production policy: temperature 0.6, Top-P 0.95, Top-K 20, and presence penalty 0.0. This example deliberately overrides temperature with greedy generation to remove sampling variation:

.build/release/TinyTitanCLI \
  --model models/ornith-1.5_35B_A3B_8Bit \
  --prompt "Explain what a mutex is and when you would use one." \
  --max-new 96 \
  --max-context 262144 \
  --rope-scaling none \
  --kv-bits 8 \
  --temperature 0 \
  --seed 1234

Run only one model process at a time. Preserve standard output and the complete standard-error footer for every measured run.

For the resumable Ornith coding-client and feature matrix, use:

python3 benchmark/coder_cli_benchmark.py
python3 benchmark/coder_cli_benchmark.py --round features
python3 benchmark/coder_cli_benchmark.py --round clients

Round	What it does	Model needed
coder (default)	every client × prompt × (warmup + reps), one server per width, resumable	yes
features	direct OpenAI requests: the cache/fast/concise tracks plus Ornith MTP off/on	yes, and an MoE family
clients	the launcher's client list and this harness's agreeing — binary, version, the launcher's own setup line, and the config this harness writes	yes, installed — the launcher dry run resolves a real install; nothing is loaded

The client list is not repeated here or in the harness: both read TINYTITAN_CLIENTS in tools/tinytitan_models.sh. Four entries are coder clients the coder round can prompt and score — codex, claude, qwen, opencode — and zed is an editor client: the launcher writes its provider config, and its CLI only opens windows and diffs files, so --clients zed is refused with a pointer to --round clients. Claude Code speaks the Anthropic Messages API, so its cases run through the loopback benchmark/claude_openai_adapter.py, which the harness starts itself when claude is among the clients. benchmark/test_coder_clients.py fails if either consumer drifts from the catalogue.

The coder round's --model also takes the dense Qwen 3.5 installs (qwen35-2b, qwen35-4b, qwen35-9b): their prefills are seconds rather than minutes, which is what makes a client check cheap. They have no routed experts, so --round features refuses them by name.

The plain invocation uses Ornith 8-bit, native 262K context, multi-prefix cache, 8-bit KV, MTP off, standard responses, Thinking off, and the base model alias. Pass --quantizations 4 or select a feature round when intentionally changing that profile. The feature round includes native Ornith MTP off/on for the selected weight precision; pass --quantizations 4 8 to test both. Its ordinary feature cells use the production policy. The paired MTP cells override temperature to 0 because MTP currently requires pure-greedy generation; deterministic microbenchmarks do the same. Top-P, Top-K, and presence penalty remain explicit in those request payloads. Review every saved answer; an exit code alone is not a quality result.

Benchmarks select thinking explicitly and default it to off. To qualify the model's real binary reasoning branch, run the same command with TINYTITAN_THINKING_MODE=on. Ornith does not publish Low/Medium/High effort levels, so benchmark labels must remain off or on.

To compare Concise Mode on real Python and Swift tool loops, plus a bounded self-scaffolding workflow, run:

python3 benchmark/ornith_concise_tool_ab.py

This A/B uses the same Ornith 8-bit production profile and changes only Concise Mode. Rendered reasoning is off, matching the coding-client benchmark, so it does not hide the variable behind a second prompt-mode change. Tool execution is restricted to an isolated directory below .build; generated programs are re-run on held-out inputs. The report distinguishes a correct artifact from a completed agent workflow. This test measures whether Ornith can participate in a self-scaffolding data-generation loop—it does not train or update the model during inference.

To compare 4-bit and 8-bit Ornith across Concise and Thinking modes with one chained four-program coding result, run:

python3 benchmark/ornith_four_program_matrix.py

The eight-cell matrix gives each Swift, Python 3.14, TensorFlow LSTM, and PyTorch attention program a fresh tool conversation with up to 25 assistant turns. Each cell also starts a fresh server. Public executions guide repair; separate hidden inputs and source-contract checks decide the final pass. The four hidden outputs must all match to form the expected chained result.

The harness uses conventional local virtual-environment paths by default. Set TINYTITAN_BENCH_PYTHON_314, TINYTITAN_BENCH_TENSORFLOW_PYTHON, or TINYTITAN_BENCH_PYTORCH_PYTHON to point to equivalent environments elsewhere. Compare full wall time only among passing cells; a faster public-only result that fails hidden validation is not a valid winner.

For a server or client benchmark, start one server first and reuse it for the entire fixed configuration. Do not restart between cache-warm requests unless server startup is the behavior being measured.

4. Check the output

A usable result must:

• exit successfully;
• finish at a normal end-of-turn or the declared token limit;
• contain coherent, complete text without a repetition loop; and
• report prompt tokens, generated tokens, prefill or TTFT, and decode rate.

Do not replace a failed result with a new baseline. Record the failure and its exact command.

5. Report

Include:

• commit SHA and clean/dirty status;
• Mac model, chip, RAM, macOS, and Swift version;
• quantization and model-installation path;
• exact command and exit code;
• power mode, memory pressure, and other active workloads;
• complete timing footer and output-quality result; and
• every deviation from this protocol.

Use medians only across truly identical successful runs. Results are measurements of that machine and configuration, not performance ceilings.