Skip to content

vision-bench 0.3.2

Choose a tag to compare

@TimeOfAnarchy TimeOfAnarchy released this 26 Oct 14:02

Portable benchmark reports and hardware provenance.

What's new

  • vision-bench export-report turns compatible tagger runs into a self-contained HTML comparison, a Markdown report, aggregate JSON and checksums, ready for a Hugging Face Static Space.
  • Compares quality on a common vocabulary while keeping each model's full-vocabulary results and timing denominators explicit.
  • Includes dataset fingerprints, model revisions and hashes, preprocessing settings, software versions and available hardware details.
  • New tagger runs record best-effort CPU, RAM, OS and visible NVIDIA GPU/driver inventory. Older runs remain supported; missing details are marked as unrecorded.
  • Public exports omit source images, raw predictions, sample identifiers, local paths and logs. See docs/publishing-results.md for the workflow and metadata limitations.

Example

vision-bench export-report ./runs/pixai ./runs/caformer --title "PixAI vs CAFormer" --output ./reports/comparison

Downloads

Install the wheel with Python 3.11 or newer, or use the source archive. Select the runtime extra (cpu, cuda or dml) when running models. Offline report export does not require ONNX Runtime or Django.

Validation

97 core tests passed locally, followed by 7 focused report tests after the final report edits. Ruff checks passed. The wheel and source distribution were built, and report export was verified from an installed wheel without ONNX Runtime or Django. No new GPU model benchmark was run for this release.