Skip to content

v0.5.0 — Exact-config evaluations and model provenance

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 07 Sep 23:29
v0.5.0
46de78c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Explicit speed commands: use llamabench speed -- <native command> to benchmark an exact llama-bench or llama-server configuration. Runner options stay before --; native flags stay after it.
  • Opt-in behavior evaluations: llamabench eval runs five bounded visual, virtual-tool-use, and role-play scenarios. Evidence is tied to the exact GGUF hash, backend build, context, KV cache, flash attention, speculative decoding, and native argument vector. These outcomes are separate from speed measurements, not an aggregate intelligence score.
  • Hardware-fit context: evaluations let llama.cpp fit the model-native context to available hardware by default, record the effective per-request context, and pin it correctly in reproduction commands, including parallel server slots.
  • Canonical model provenance: --base-model identifies the intended base model when publisher metadata is missing; speed runs also accept --active-params for disclosed sparse-model parameter counts. Explicit provenance is validated against the measured artifact before measurement.
  • More accurate multi-GPU identity: device grouping respects CUDA visibility, selected devices, split mode, and mixed-GPU configurations; ambiguous selections fail explicitly.

Compatibility and evaluation notes

  • Existing v0.4.x drop-in commands and the run, bench, and verify subcommands remain available.
  • Explicit speed and eval commands reject inherited LLAMA_ARG_* settings. Put equivalent native flags after -- so the published configuration is complete.
  • Eval v2 uses a disclosed sampled profile with seed 42. Visual tasks permit up to 60,000 generated tokens each and can take substantial time on slower hardware.
  • Evaluation provenance and evidence handling are hardened, including incomplete visual output, backend identity, and reproduction inputs. Auxiliary artifacts that the evaluation contract cannot hash remain unsupported.

Install or upgrade

curl -fsSL https://llamabench.ai/install.sh | sh
llamabench --version

Prebuilt archives are provided for Linux x86_64, macOS Intel, macOS Apple Silicon, and Windows x86_64. The runner does not bundle llama.cpp or model files.

Full changelog: v0.4.4...v0.5.0

Release source: 46de78c04644a2989119c013cc8bd4ad937517eb.

Assets 6
Loading