Repository navigation
BenchCAD 2.1-beta
Pre-release
Pre-release
Pre-release. Core data is unchanged. Scores from v2.0 (v2.0-core1) are not comparable with v2.1-beta.
- Reasoning carried across rounds. Each round, the model gets its earlier reasoning back as its provider returned it: Anthropic's signed thinking blocks, OpenAI's encrypted reasoning items, OpenRouter's
reasoning_details. Gemini already got its thought signatures back. Before, only the text went back. - Compaction threshold raised from 180k to 800k tokens.
- Images are sized once when they enter the conversation. Past the request limits they are dropped in chunks (20 MB down to 14 MB; the case's own input images are never dropped), so the conversation's prefix stays stable and carried thinking stays valid.
- Limits: by default 40 rounds and 500k output tokens per case, thinking included; the case ends at whichever comes first. These limits are provisional and may be adjusted with the labs.
- Every record and
core_summary.jsoncarryharness_version: "2.1-beta".
For a reported score, check out the tag (git checkout v2.1-beta) and run tools/run_core.sh. Don't run from main and don't mix versions.