Skip to content

Releases

Vision Bench v0.4.0

Choose a tag to compare

@TimeOfAnarchy TimeOfAnarchy released this 19 Nov 13:14

Vision Bench 0.4.0 expands the offline analysis workflow for image taggers and
fine-tuned checkpoints. Use the full scores from a saved run for these analyses.

  • Compare paired bootstrap intervals, per-image quality, label-frequency strata,
    and precision or recall at matched operating points.
  • Fit category thresholds or supported per-tag thresholds on a separate
    calibration run with fit-thresholds. The fitted profile records its source
    and evaluation checks the split for identical image hashes.
  • Compare renamed tags with optional aliases and implications. The report shows
    exact-name coverage alongside mapped coverage.
  • Find per-tag AP and F1 regressions between checkpoints. Use label-audit to
    review sampled unlabeled predictions or hard-images to inspect disagreement
    and low-F1 examples.

Download the wheel for Python 3.11+ and install it in a fresh environment. The
cpu extra installs the ONNX Runtime CPU provider.

python -m pip install "./vision_bench-0.4.0-py3-none-any.whl[cpu]"
vision-bench --help

The source archive includes the generated CPU example. Both package files are
listed in SHA256SUMS.

Tagging guide ·
Autotagging case study ·
Frozen reproduction kit

Vision Bench v0.3.3

Choose a tag to compare

@TimeOfAnarchy TimeOfAnarchy released this 17 Nov 18:18

Vision Bench 0.3.3 improves the local model-comparison workflow and documents the
project's recurring evaluation use case: comparing new image models and
fine-tuned checkpoints while retaining complete predictions for offline analysis.

  • Add the PixAI 1.0 Mexes FP16 profile and optional PyTorch CAFormer resolution experiments.
  • Support bounded ONNX CPU sessions and optional tall-image admission limits for library integrations.
  • Preserve bounded diagnostic headers in the VLM HTTP transport.
  • Rework the README around project motivation, the generated CPU example and a worked autotagging case study. Keep detailed usage in focused guides.
  • Point the project page to current GitHub releases instead of a fixed historical wheel.

The worked autotagging case compares five configured pipelines on the same 1,024
images. Read the published autotagging report
on the project Space.
Model weights, the exact image selection
and raw predictions are not included in this release.

CPU package CI passed on Linux (Python 3.11/3.13) and Windows (Python 3.13).
Locally, the built wheel was installed in a fresh environment outside the
checkout on Windows/Python 3.13; 114 tests, dependency checks, the generated CPU
example and exact offline recomputation passed. Ruff lint and format checks
passed. The five historical experiment runs also recomputed exactly from their
verified saved artifacts. This does not establish new GPU/provider performance
or an exact public rerun of the private image selection.

Release assets: wheel, source archive and SHA256SUMS. Project code and
documentation are MIT; third-party models and data retain their own terms.

vision-bench 0.3.2

Choose a tag to compare

@TimeOfAnarchy TimeOfAnarchy released this 26 Oct 14:02

Portable benchmark reports and hardware provenance.

What's new

  • vision-bench export-report turns compatible tagger runs into a self-contained HTML comparison, a Markdown report, aggregate JSON and checksums, ready for a Hugging Face Static Space.
  • Compares quality on a common vocabulary while keeping each model's full-vocabulary results and timing denominators explicit.
  • Includes dataset fingerprints, model revisions and hashes, preprocessing settings, software versions and available hardware details.
  • New tagger runs record best-effort CPU, RAM, OS and visible NVIDIA GPU/driver inventory. Older runs remain supported; missing details are marked as unrecorded.
  • Public exports omit source images, raw predictions, sample identifiers, local paths and logs. See docs/publishing-results.md for the workflow and metadata limitations.

Example

vision-bench export-report ./runs/pixai ./runs/caformer --title "PixAI vs CAFormer" --output ./reports/comparison

Downloads

Install the wheel with Python 3.11 or newer, or use the source archive. Select the runtime extra (cpu, cuda or dml) when running models. Offline report export does not require ONNX Runtime or Django.

Validation

97 core tests passed locally, followed by 7 focused report tests after the final report edits. Ruff checks passed. The wheel and source distribution were built, and report export was verified from an installed wheel without ONNX Runtime or Django. No new GPU model benchmark was run for this release.