BenchCAD 2.1

BenchCAD 2.1 replaces the 2.1-beta pre-release. Scores from 2.1 are not comparable with 2.0 or 2.1-beta.

Limits and prompt

• 60 rounds and 500k output tokens per case, thinking included. The case ends at a submit or at whichever limit runs out first, checked between calls. The prompt states both limits, and each observation starts Round N/60 · output tokens spent / 500,000. (2.1-beta: 40 rounds and 500k.) The limits are provisional and may be adjusted with the labs.
• The prompt now says that an episode ending without a submit has no answer and scores 0. Before, it said only that submit ends the episode and is what gets scored.
• Replies the harness throws away and asks again for no longer count against the token budget. These are a reply with no executable block, a silent or broken stream, and an infrastructure failure. Records keep them as tokens.output_discarded and tokens.calls_discarded; tokens.output still counts them, for cost.

Task 6 scorer: rule B and duplicates (ecad 01e98d0, PRs #112 and #113)

• A predicted component may pair only with a ground-truth part that no other part, of any type, is more than 0.005 nearer to. Before, each part's matching gate widened against same-type neighbours only. A second copy of a part, placed on its footprint under another type, could then pair with a nearby part of that type that was never placed.
• On the 21 position boards this removed 16 such matches on 7 boards. A correct answer still scores 1.0 on every board.
• It costs answers with imprecise centres a little. With random centre error, the mean board score is unchanged at sd 0.002, goes from .847 to .837 at sd 0.005, and from .403 to .390 at sd 0.01.
• Duplicates are wrong. Two or more predicted components within 0.0005 of each other, whatever their types, are all unmatched, so the part under them counts as missed. The threshold comes from the ground truth: the closest pair of real parts on any board is 0.000716 apart (two parts on opposite faces, projected into one top view), and no pair is closer than 0.0005. A correct answer still scores 1.0 on every board, and answers with imprecise centres score the same as under rule B alone. Listing a part twice now costs more than leaving it out.
• The scorer digest is now c4799cc240612b038362efc5f649fc05e4bf1ce33babec46f1237092a9a7137d (was f4dd1410…). A task-6 board bound to another digest is refused (evaluator_error).

Data. The case files are unchanged. At the Hugging Face revision v2.1, every task-6 gt/correspondence.json names the new digest, and its hash in case.json, dataset.json (scorer_digest) and MANIFEST.sha256 follow. That is Core 4b6fe17f63 and Public 1ef73a9bb7. tools/run_core.sh downloads that revision, never main. main keeps the 2.0 / 2.1-beta data for those tags.

Updated in place. The v2.1 tag and its data revision were moved on 2026-10-19 to add the duplicate rule. A checkout or download from before the move cannot be mixed with the current one: run_core checks that the data's scorer digest matches the code's, and a mismatch stops the run before any spend instead of scoring.

Checked and unchanged

• A plain solid scores at most 0.2, and only on parts with no holes. We scored a plain 10 mm box against all 34 Core part cases (T1 and T3). The IoU term is 0 on every one. The 0.2 × topology term gives exactly 0.2 on the 9 T3 parts with no holes, 0.05 on the 7 parts with one hole, and about 0 elsewhere; the mean is 0.065. The scoring is unchanged.

Unchanged from 2.1-beta:

• reasoning carried across rounds (Anthropic signed thinking, OpenAI encrypted reasoning, Gemini thought signatures, OpenRouter reasoning_details);
• compaction at 800k tokens;
• images sized once on entry and dropped in chunks.

Every record and core_summary.json carry harness_version: "2.1". For a reported score, check out the tag (git checkout v2.1) and run tools/run_core.sh. Don't run from main and don't mix versions. What 2.1 pins is in releases/benchcad-2.1.json.