BenchCAD 2.1

BenchCAD 2.1 replaces the 2.1-beta pre-release. Scores from 2.1 are not comparable with 2.0 or 2.1-beta.

Limits and prompt

• 60 rounds and 500k output tokens per case, thinking included. The case ends at a submit or at whichever limit runs out first, checked between calls. The prompt states both limits, and each observation starts Round N/60 · output tokens spent / 500,000. (2.1-beta: 40 rounds and 500k.) The limits are provisional and may be adjusted with the labs.
• The prompt now says that an episode ending without a submit has no answer and scores 0. Before, it said only that submit ends the episode and is what gets scored.
• Replies the harness throws away and asks again for no longer count against the token budget. These are a reply with no executable block, a silent or broken stream, and an infrastructure failure. Records keep them as tokens.output_discarded and tokens.calls_discarded; tokens.output still counts them, for cost.

Task 6 scorer: rule B and duplicates (ecad 01e98d0, PRs #112 and #113)

• A predicted component may pair only with a ground-truth part that no other part, of any type, is more than 0.005 nearer to. Before, each part's matching gate widened against same-type neighbours only. A second copy of a part, placed on its footprint under another type, could then pair with a nearby part of that type that was never placed.
• On the 21 position boards this removed 16 such matches on 7 boards. A correct answer still scores 1.0 on every board.
• It costs answers with imprecise centres a little. With random centre error, the mean board score is unchanged at sd 0.002, goes from .847 to .837 at sd 0.005, and from .403 to .390 at sd 0.01.
• Duplicates are wrong. Two or more predicted components within 0.0005 of each other, whatever their types, are all unmatched, so the part under them counts as missed. The threshold comes from the ground truth: the closest pair of real parts on any board is 0.000716 apart (two parts on opposite faces, projected into one top view), and no pair is closer than 0.0005. A correct answer still scores 1.0 on every board, and answers with imprecise centres score the same as under rule B alone. Listing a part twice now costs more than leaving it out.
• The scorer digest is now c4799cc240612b038362efc5f649fc05e4bf1ce33babec46f1237092a9a7137d (was f4dd1410…). A task-6 board bound to another digest is refused (evaluator_error).

Data. The case files are unchanged. At the Hugging Face revision v2.1, every task-6 gt/correspondence.json names the new digest, and its hash in case.json, dataset.json (scorer_digest) and MANIFEST.sha256 follow. That is Core 4b6fe17f63 and Public 1ef73a9bb7. tools/run_core.sh downloads that revision, never main. main keeps the 2.0 / 2.1-beta data for those tags.

Updated in place. The v2.1 tag and its data revision were moved on 2026-10-19 to add the duplicate rule. A checkout or download from before the move cannot be mixed with the current one: run_core checks that the data's scorer digest matches the code's, and a mismatch stops the run before any spend instead of scoring.

Checked and unchanged

• A plain solid scores at most 0.2, and only on parts with no holes. We scored a plain 10 mm box against all 34 Core part cases (T1 and T3). The IoU term is 0 on every one. The 0.2 × topology term gives exactly 0.2 on the 9 T3 parts with no holes, 0.05 on the 7 parts with one hole, and about 0 elsewhere; the mean is 0.065. The scoring is unchanged.

Unchanged from 2.1-beta:

• reasoning carried across rounds (Anthropic signed thinking, OpenAI encrypted reasoning, Gemini thought signatures, OpenRouter reasoning_details);
• compaction at 800k tokens;
• images sized once on entry and dropped in chunks.

Every record and core_summary.json carry harness_version: "2.1". For a reported score, check out the tag (git checkout v2.1) and run tools/run_core.sh. Don't run from main and don't mix versions. What 2.1 pins is in releases/benchcad-2.1.json.

BenchCAD 2.1-beta

Pre-release. Core data is unchanged. Scores from v2.0 (v2.0-core1) are not comparable with v2.1-beta.

• Reasoning carried across rounds. Each round, the model gets its earlier reasoning back as its provider returned it: Anthropic's signed thinking blocks, OpenAI's encrypted reasoning items, OpenRouter's reasoning_details. Gemini already got its thought signatures back. Before, only the text went back.
• Compaction threshold raised from 180k to 800k tokens.
• Images are sized once when they enter the conversation. Past the request limits they are dropped in chunks (20 MB down to 14 MB; the case's own input images are never dropped), so the conversation's prefix stays stable and carried thinking stays valid.
• Limits: by default 40 rounds and 500k output tokens per case, thinking included; the case ends at whichever comes first. These limits are provisional and may be adjusted with the labs.
• Every record and core_summary.json carry harness_version: "2.1-beta".

For a reported score, check out the tag (git checkout v2.1-beta) and run tools/run_core.sh. Don't run from main and don't mix versions.

BenchCAD 2.0

BenchCAD 2.0 is the release behind the published leaderboard lines. Those lines now live in results/core_reference.json, marked harness_version: "2.0".

What 2.0 is

• Cases: 100 Core cases across six tasks: T1 6, T2 11, T3 28, T4 25, T5 18, T6 12.
• Dataset: BenchCAD/benchcad-2.0-core version 1.0, tagged v2.0 there (commit d159834275).
• Harness: version 2.0, at this tag (v2.0-core1).
  • 30 rounds per case, with no output-token budget.
  • Earlier rounds go back to the model as text only. The model's thinking is not carried; Gemini replies kept their thought signatures.
  • The context is summarised at 180k tokens.
• T6 scorer digest: f4dd141038e7.

The Public set (BenchCAD/benchcad-2.0-public, uploaded 2026-10-12) is not part of 2.0. run_core.sh at this tag has no --set public.

Reproducing the published lines

git checkout v2.0-core1
tools/run_core.sh --model anthropic/claude-opus-5-5     # ANTHROPIC_API_KEY
tools/run_core.sh --model openai/gpt-6-astra            # OPENAI_API_KEY

run_core.sh at this tag downloads Core and checks its MANIFEST.sha256, its name and version (benchcad-2.0-core 1.0) and the T6 scorer digest before any spend.

Not comparable with 2.1-beta. BenchCAD 2.1-beta changes the setting: reasoning is carried across rounds, the limits are 40 rounds and 500k output tokens, and the context is summarised at 800k. The Core data is the same, but scores from the two versions are not comparable.