Skip to content

v1.0.0 — sourced, schema-validated rewrite

Choose a tag to compare

View all tags
@alloevil alloevil released this 04 Sep 07:11
· 25 commits to main since this release
v1.0.0
4602d8d

Complete rewrite. Site: https://alloevil.github.io/llm-benchmarks-tracker/ · API: https://alloevil.github.io/llm-benchmarks-tracker/api/v1/index.json

Data

  • 41 benchmarks (24 model-layer, 17 agent-layer), 18 evaluators, 179 sourced results.
  • Every result row records source.url, source.kind (official-leaderboard / paper / independent-evaluation / developer-report / aggregator), accessed date and conditions (tools, reasoning effort, scaffold, pass@k, split, cost).
  • Source mix: 74 official leaderboard · 46 paper · 13 independent · 36 self-reported · 10 aggregator.
  • All 38 arXiv ids verified against arXiv metadata. Six wrong ids from the previous version fixed (ARC-AGI-2, MMMU-Pro, RULER, BFCL, τ-bench, BrowseComp); HLE task count, MMLU year and BrowseComp release date corrected.

Data model

  • data/benchmarks/<id>.json, data/results/<id>.json (append-only ledger), data/evaluators/<id>.json; JSON Schema 2020-12 in schema/.
  • Closed vocabularies for status, contamination_risk, domains, source.kind, evaluator.kind.
  • Bidirectional supersedes / superseded_by chains (SWE-bench → Verified → Pro, ARC-AGI-1 → 2 → 3, …).

Tooling

  • scripts/validate.py: schema + cross-file invariants (filename/id, dangling refs, ascending ledgers, percent range, future dates, undeclared splits, duplicate rows).
  • scripts/build.py: generates README tables, static site, api/v1/*.json, sitemap, robots; --check guards README drift in CI.
  • 21 contract tests; ruff; GitHub Actions for CI, Pages deploy, weekly link check.

Removed

  • docs/*.md, research/*.json, data/timeline.json, hand-written index.html — duplicated, mutually inconsistent copies of the data.
Assets 2
Loading