Repository navigation
A13N Harness × Terminal-Bench 4.0: Comparable pass rate and inference cost #1015
Wh1isper
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We evaluated A13N Harness on the 62 Terminal-Bench 4.0 tasks supported by our machine and compared the results with the published official results for the same model and tasks.
With the same model, reasoning effort, and task set, A13N Harness achieved a 58.06% pass rate, compared with 58.77% for official Codex—a gap of about 0.7 percentage points. A13N Harness Cost was $106.88, estimated using the official per-task cache-hit rates; the official known cost per round was $102.53. The observed results show comparable task performance and inference cost.
Configuration
4eb06e8, with local modificationsgpt-6.1-sol)Evaluation process
Harbor prepared each task's Docker environment. The Harness UI App drove the model with native file and shell tools, and each task's verifier assessed the result. Task-declared Skills and MCP services were connected as needed.
A13N Harness retained one final outcome per task. The official comparison used scored outcomes from five final runs per task. For the cost estimate, we retained our input and output token usage, applied each task's official cache-hit rate separately, and summed the estimates across all 62 tasks.
Task scope
Terminal-Bench 4.0 contains 66 tasks. Our machine provided 12 CPU threads and approximately 32 GiB of memory, without an NVIDIA GPU meeting the task requirements. We excluded these four tasks:
fp8-rmsnorm-gemmjax-speedrun-gpumath-eval-graderlive-database-cutoverThe main comparison below uses the same remaining 62 tasks, rather than the official leaderboard's full 66-task score. The README chart also reports a matched 61-task cost comparison, described below.
Pass rate and cost
A13N Harness Cost: Estimated using the official per-task cache-hit rates.
Accounting: Of the 310 official final records, 308 have valid scores. The two missing scores are excluded from the pass-rate denominator, while all 62 tasks remain in the comparison. Official cost per round is the known final-run cost divided by five, with one usage record missing. Both costs use available usage from selected or final runs and exclude other attempts and machine costs.
Matched cost comparison used in the README
For a cost comparison with complete official usage coverage, we exclude
cumulative-layout-shiftfrom both sides because one of its five official final runs has no usage record. The remaining 61 matched tasks cost $98.73 for A13N Harness under the same cache-hit-rate estimate and $97.70 per round for official Codex (the five-run total divided by five). The difference is approximately 1.06%.The README chart keeps the 62-task scored pass rates and labels the cost column separately as 61 matched tasks. Both cost figures use available usage from selected or final runs, excluding other attempts and machine costs. The full 62-task figures above remain unchanged.
Conclusion
A13N Harness passed 36 of 62 tasks, with 26 failures. Its pass rate was about 0.7 percentage points below official Codex with the same model configuration. Under the per-task cache-hit-rate estimate, both sides' known inference costs were approximately $100.
On this Terminal-Bench 4.0 task set, A13N Harness achieved a similar task pass rate to official Codex at a similar inference cost.
All reactions