Skip to content

BenchCAD 2.1-beta

Pre-release
Pre-release

Choose a tag to compare

@BenchCAD BenchCAD released this 16 Oct 19:34
· 18 commits to main since this release
13e1da0

Pre-release. Core data is unchanged. Scores from v2.0 (v2.0-core1) are not comparable with v2.1-beta.

  • Reasoning carried across rounds. Each round, the model gets its earlier reasoning back as its provider returned it: Anthropic's signed thinking blocks, OpenAI's encrypted reasoning items, OpenRouter's reasoning_details. Gemini already got its thought signatures back. Before, only the text went back.
  • Compaction threshold raised from 180k to 800k tokens.
  • Images are sized once when they enter the conversation. Past the request limits they are dropped in chunks (20 MB down to 14 MB; the case's own input images are never dropped), so the conversation's prefix stays stable and carried thinking stays valid.
  • Limits: by default 40 rounds and 500k output tokens per case, thinking included; the case ends at whichever comes first. These limits are provisional and may be adjusted with the labs.
  • Every record and core_summary.json carry harness_version: "2.1-beta".

For a reported score, check out the tag (git checkout v2.1-beta) and run tools/run_core.sh. Don't run from main and don't mix versions.