Skip to content

Verity Benchmark v0.3

Latest
Latest

Choose a tag to compare

View all tags
@Th0rgal Th0rgal released this 14 Oct 09:18
· 15 commits to main since this release
v0.3
188cfaa
This commit was signed with the committer’s verified signature.
Th0rgal Thomas Marchand
GPG key ID: F480B9A3F84BE648
Verified
Learn about vigilant mode.

Detailed verifier-backed artifacts for benchmark version 0.3.

This release contains the frozen 263-task manifest (benchmark-v0.3-version-manifest.json), the deterministic STRAT-50 panel (benchmark-v0.3-panel.json, covering all 41 cases), a machine-readable leaderboard (benchmark-v0.3-leaderboard.json), per-cohort run bundles (.tar.zst), and SHA-256 checksums (SHA256SUMS). Every scored cohort has 50 distinct terminal verifier verdicts on the same panel with the p4_normal budget of 16 attempts and 120 tool calls (reasoning_effort=low; the three Mistral cohorts use provider-default sampling because the Mistral route rejects reasoning_effort). Infrastructure-invalid attempts are excluded from scores.

Across all 12 completed cohorts, 31 / 50 tasks (62.0%) on the frozen STRAT-50 panel were solved by at least one model:

  1. anthropic/claude-opus-5.5 — 27/50 (54.0%)
  2. google/gemini-4-argon-eap — 27/50 (54.0%)
  3. openai/gpt-6-astra — 23/50 (46.0%)
  4. openai/gpt-6.1-sol — 22/50 (44.0%)
  5. muse-code/muse-wonder-1.0 — 19/50 (38.0%)
  6. openai/gpt-6-luna — 12/50 (24.0%)
  7. zai/glm-5.3 — 12/50 (24.0%)
  8. muse-spark-1.3 — 11/50 (22.0%)
  9. xai/grok-4.7 — 9/50 (18.0%)
  10. mistral/mistral-small-latest — 4/50 (8.0%)
  11. mistral/mistral-medium-latest — 3/50 (6.0%)
  12. mistral/labs-leanstral-1-5-1 (Leanstral 1.5) — 2/50 (4.0%)

STRAT-50 reports the direct observed solve rate on the frozen 50-task panel. It is not an unweighted estimate of FULL-263 performance. Benchmark v0.3 uses Lean 4.31.0.

Assets 19
Loading