v0.5.0 — Exact-config evaluations and model provenance

Highlights

• Explicit speed commands: use llamabench speed -- <native command> to benchmark an exact llama-bench or llama-server configuration. Runner options stay before --; native flags stay after it.
• Opt-in behavior evaluations: llamabench eval runs five bounded visual, virtual-tool-use, and role-play scenarios. Evidence is tied to the exact GGUF hash, backend build, context, KV cache, flash attention, speculative decoding, and native argument vector. These outcomes are separate from speed measurements, not an aggregate intelligence score.
• Hardware-fit context: evaluations let llama.cpp fit the model-native context to available hardware by default, record the effective per-request context, and pin it correctly in reproduction commands, including parallel server slots.
• Canonical model provenance: --base-model identifies the intended base model when publisher metadata is missing; speed runs also accept --active-params for disclosed sparse-model parameter counts. Explicit provenance is validated against the measured artifact before measurement.
• More accurate multi-GPU identity: device grouping respects CUDA visibility, selected devices, split mode, and mixed-GPU configurations; ambiguous selections fail explicitly.

Compatibility and evaluation notes

• Existing v0.4.x drop-in commands and the run, bench, and verify subcommands remain available.
• Explicit speed and eval commands reject inherited LLAMA_ARG_* settings. Put equivalent native flags after -- so the published configuration is complete.
• Eval v2 uses a disclosed sampled profile with seed 42. Visual tasks permit up to 60,000 generated tokens each and can take substantial time on slower hardware.
• Evaluation provenance and evidence handling are hardened, including incomplete visual output, backend identity, and reproduction inputs. Auxiliary artifacts that the evaluation contract cannot hash remain unsupported.

Install or upgrade

curl -fsSL https://llamabench.ai/install.sh | sh
llamabench --version

Prebuilt archives are provided for Linux x86_64, macOS Intel, macOS Apple Silicon, and Windows x86_64. The runner does not bundle llama.cpp or model files.

Full changelog: v0.4.4...v0.5.0

Release source: 46de78c04644a2989119c013cc8bd4ad937517eb.

v0.4.4

Merge pull request #2 from XpressAI/codex/report-system-ram

Report system RAM with benchmark submissions

v0.4.2

Report the host CPU model in submissions (contract's optional hardwar…

v0.4.1

v0.4.1 — submissions record the GGUF sha256 (web-side linking, ADR-01…

v0.4.0

v0.4.0 — drop-in CLI (swap llama-bench/llama-server for llamabench) +…

v0.3.6

Add --server-args: one whitespace-delimited string of llama-server flags

Speculative decoding has many flags; passing them one --server-arg at a
time is tedious. --server-args takes a single string that's split on
whitespace and appended to llama-server's args, e.g.
  --server-args "--spec-type draft-mtp --spec-draft-n-max 2"

Both forms are supported and combined (repeatable --server-arg first, then
the split --server-args). Added allow_hyphen_values so flag-like values
("--foo") are accepted as values — the repeatable --server-arg silently
rejected those before. Test covers the combine.

v0.3.5

Apple Silicon: report the clean chip name ("Apple M4"), not "MTL0 (Ap…

v0.3.4

Progress feedback: download bar + phase headers + verify counter

`run` was a long silent wait. Now it shows what's happening:
- Phase headers: [1/4] Model · [2/4] Benchmark · [3/4] Verify · [4/4] Submit.
- Model download: a live progress bar (bar, %, downloaded/total, speed, ETA)
  on a TTY; piped/CI output keeps the plain start/done lines.
- Verification: a "generating N/total (prompt, turns)…" counter, since the
  multi-turn llama-server pass is the slowest, opaque part.

All terminal-only drawing is gated on stderr.is_terminal() so logs aren't
spammed with carriage returns.

v0.3.3

Detect Apple Silicon (Metal); stop misparsing the device banner

On a Mac the device banner is "ggml_metal_init: ... device: Apple M4" /
"GPU name: Apple M4", but parse_device matched ANY "ggml_... = ..." line and
grabbed config values — so an M4 Mac was recorded with a bogus device name
(e.g. "180 s)") and treated as CPU.

- parse_device: the "ggml_xxx: N = name" form now requires a numeric device
  index before "=" (so config lines like "hasUnifiedMemory = true" are
  ignored), plus explicit Metal "device:"/"GPU name:" parsing.
- detect::gpu_name() fallback now also resolves the Apple chip via sysctl
  (machdep.cpu.brand_string) when the banner yields nothing.
- Report Apple unified memory (sysctl hw.memsize) as VRAM, since that's the
  usable GPU memory; discrete GPUs still get VRAM/bandwidth from the server
  catalog.
- Tests for the Metal device line and the config-line false positives.

v0.3.2

Detect CUDA/HIP GPUs (were recorded as "CPU")

llama.cpp's CUDA/HIP backend prints its device to stderr as
"  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes",
which parse_device didn't match (it only handled the Vulkan/Metal
"ggml_xxx: 0 = <name> | ..." form). With no device parsed the runner
fell back to "CPU", so real GPU runs (e.g. an RTX 3090 doing 85 tok/s on
gemma-4-12b Q4) were recorded as CPU — which also made the site derive
"GPU layers: 0".

- parse_device now also reads the "Device N: <name>, ..." banner.
- Added an nvidia-smi fallback: when the banner yields no device but the
  run used the GPU (ngl != 0), query nvidia-smi for the card name, so a
  GPU run can't be mislabeled CPU even on a build whose banner we don't
  recognize.
- Tests for the CUDA layout + both banner shapes.