A13N Harness × FrontierHarness Eval: 21/30 tasks with Kimi K3

[Wh1isper]

We evaluated A13N Harness with Kimi K3 on all 30 FrontierHarness Eval tasks, using the FrontierHarness Eval task set and evaluation methodology. A13N Harness passed 21/30 tasks (70.0%), with $2.93 standardized cost per pass and a 7m 03s median successful-trial time.

The comparison places this local run alongside the 12 published baseline configurations from a frozen upstream snapshot. A13N Harness has the highest observed pass rate in that table; the comparison is provisional because the runtime and provider conditions differ and no matched control was run.

Configuration

Setting	Configuration
Harness	A13N Harness, executed through the real Harness UI App and native tools
Evaluated commit	Agent Foundation c7e287142188dd581ef37daea8fdcfa59f310857
Model	Kimi K3 through an internal OpenAI-compatible gateway; route bailian-kimi-k3
Model settings	No explicit model-setting overrides ({})
Benchmark	FrontierHarness Eval v1.0: 21 Terminal-Bench tasks and 9 DeepSWE tasks
Frozen methodology and baseline	frontier-harness-eval/eval@e837a70
Execution	Harbor 0.23.0 for Terminal-Bench; Pier 0.3.1 for DeepSWE; Docker task containers
Host	Local Apple Silicon/macOS with OrbStack; Linux amd64 task images under emulation
Selection	One canonical valid result per task, including valid failures
Evaluation dates	October 15–16, 2026; final clean trial completed October 16 at 12:50:16 UTC

Instructions

We used our general-purpose Harness UI instructions, without benchmark-specific instruction tuning to reduce token usage or inference cost. The adapter supplied the same fixed task-execution guidance across tasks, alongside the normal Harness UI system prompt. Each benchmark's task instruction was submitted as the user task.

The frozen adapter guidance was:

Complete the user's task in the provided working environment.
Inspect relevant files before editing. Use the available file and shell tools.
Validate your work with appropriate commands. Work autonomously; no human is available.
Report the result accurately. Do not read or modify benchmark grading infrastructure,
reference solutions, or agent installation/configuration files.

This guidance sets execution and evaluation-integrity boundaries; it contains no token-saving directives or task-specific solution hints.

Evaluation process and comparability

The evaluation reused the frozen upstream task instructions, task definitions, task payloads, and verifiers. Harbor/Pier prepared the task containers, installed the frozen Harness bundle, ran the real Harness UI App, and collected task-verifier outcomes. Native file, shell, and image-reading capabilities handled the tasks; the adapter did not replace the agent loop. Task-declared tools were connected where required.

The local setup differs from the published Runta baseline: it uses OrbStack with amd64 emulation, different runner versions, an internal model gateway, and local container provisioning rather than Runta golden-snapshot restores. Task network controls remained in place. The clean Meriyah rerun used an offline Python/dependency cache, model-gateway-only agent egress, and a separate offline verifier. Other tasks used their retained task/suite policies rather than this stricter rerun policy.

The upstream reproduction workflow calls for a matched control to establish comparability. We did not run one. Accordingly, the table below is a provisional comparison with frozen published results, not an official upstream leaderboard rank. Pass-rate, cost, cache, and speed differences combine harness behavior with these execution differences.

Full results

Rank is by observed pass rate; ties share a rank. All rows cover the same 30 task identities. Baseline values come from the frozen upstream results/eval-data.json, not new control runs.

Provisional rank	Harness	Passed	Pass rate	Cost per pass	Cost coverage	Median cache hit rate	Median successful-trial time
1	A13N Harness	21/30	70.0%	$2.9254	100%	84.99%	7m 03s
2	Codex	20/30	66.7%	$3.4683	100%	87.97%	6m 43s
3	Claude Code	19/30	63.3%	$18.3368	97%	67.79%	9m 38s
3	DSH Creator	19/30	63.3%	$3.2849	90%	84.30%	6m 44s
5	DSH PTC	18/30	60.0%	$4.5774	90%	87.19%	7m 44s
5	DSH Standard	18/30	60.0%	$3.4589	90%	86.47%	6m 17s
5	Pi	18/30	60.0%	$2.4330	100%	79.38%	7m 33s
8	DSH Minimal	17/30	56.7%	$4.7180	87%	84.64%	5m 41s
8	Kimi Code	17/30	56.7%	$3.6471	100%	88.03%	7m 56s
8	Oh My Pi	17/30	56.7%	$4.7455	100%	82.15%	6m 46s
11	Exo Harness	16/30	53.3%	$1.0452	100%	70.33%	6m 17s
12	Hermes	15/30	50.0%	$2.9043	93%	85.91%	6m 58s
12	OpenCode	15/30	50.0%	$3.2443	97%	78.41%	6m 27s

Cost coverage is the share of tasks with cost data under each row's reporting scope, rounded to a whole percent. Several baseline cost totals have partial coverage; those costs reflect the available portion. A13N Harness's 100% reported-cost coverage uses the request-exclusion policy below; raw usage is complete for 22/30 tasks. Cache coverage is 100% of successful tasks for every row.

The README chart plots pass rate against cost per pass, with a logarithmic cost axis. The orange star marks A13N Harness; the orange line connects the published baselines' cost/pass-rate frontier only.

Cost, cache, and timing definitions

Costs use the frozen upstream kimi-k3-2026-08-20 pricing table: $3.00 per million fresh-input or cache-write tokens, $0.30 per million cache-read tokens, and $15.00 per million output tokens. These are standardized inference estimates, not a supplier invoice; machine costs are excluded.

Input tokens include cached reads and writes. The calculation charges fresh input after subtracting cache reads and writes, then adds the appropriate cache and output charges. First-cold normalization reprices cached reads in each task's first included root generation at the fresh-input rate. For OpenSSL, the initial request had no response or usage, so normalization starts at the first included generation.

A13N Harness metric	Value
Selected valid tasks	30: 21 PASS, 9 FAIL
Reported standardized fee, before first-cold normalization	$60.9107406
First-cold normalized total	$61.4332878
Cost per pass: normalized total / 21 passes	$2.9253947
Median task cost, after first-cold normalization	$0.4137138
Included / retained request records	1,110 / 1,119
Excluded no-response requests with unavailable usage	9
Reported-cost coverage / raw complete-usage coverage	30/30 / 22/30 tasks
Inclusive input tokens	127,207,748
Cache-read tokens	124,296,192
Cache-write tokens	0
Output tokens	992,481
Median normalized cache hit rate, successful tasks	84.99%; 21/21 successes measured
Token-weighted normalized cache hit rate, successful tasks	97.19%
Median successful full-trial time	422.960818 seconds; 21/21 successes measured
Cumulative valid full-trial time, including failures	39,010.854931 seconds = 10h 50m 11s; 30/30 tasks measured

Only canonical valid PASS and FAIL results contribute to these totals. Earlier smoke tests, infrastructure-invalid attempts, quarantined attempts, and superseded retries are excluded. Nine failed requests with no response and unavailable usage are excluded from the reported fee scope; their records remain in the raw audit. Their provider charges are unknown, rather than assumed to be zero.

Cost per pass divides the first-cold total across all selected tasks, including failures, by the number of passes. It is different from the median task cost. The table's cache metric is the median normalized cache-read/input ratio among successful tasks, rather than a token-weighted aggregate. Timing uses full-trial wall time, including setup and verification; the cumulative figure sums selected trial durations and is not calendar elapsed time.

Result selection and integrity

• Merge-diff: the original attempt exceeded the 900-second agent deadline and is retained as FAIL. A cleanup error masked that timeout in the raw runner status. A later passing retry is excluded, preserving first-valid selection.
• Meriyah: an earlier attempt obtained an upstream reference answer and was quarantined. A subsequent setup-only failure occurred before model execution and is infrastructure-invalid. After repairing installation with an offline cache, the first valid clean rerun was retained as FAIL: 48/49 fail-to-pass tests and 51,469/51,469 pass-to-pass tests passed. The remaining failure concerned for-of with using and of as the binding name. No further quality retry was selected.
• Clean-rerun audit: the retained trajectory has 407 transcript rows and 125 matched tool calls/results, with no omitted or truncated parts. GitHub API and npm lookup attempts returned no answer; a local bundled-parser probe returned filenames and a zero keyword count, not implementation code. No reference answer, hidden grading data, or network bypass was found in this retained trajectory. The original verifier ran separately after agent completion.

Per-task results

The first nine rows are DeepSWE tasks; the remaining 21 are Terminal-Bench tasks. Costs below use the first-cold normalization and reported request scope described above. Times are full-trial seconds. Values are rounded for display; aggregate calculations use unrounded data.

Task	Result	First-cold cost	Time (seconds)
datacurve/anko-typed-variable-bindings	PASS	$3.4208	1732.6
datacurve/arktype-json-schema-refs-dependencies	PASS	$8.9535	4902.6
datacurve/expr-try-catch-errors	FAIL	$5.4191	2708.3
datacurve/fastapi-deprecation-response-headers	PASS	$2.9515	1752.0
datacurve/httpx-multipart-response-parsing	PASS	$2.6642	1828.0
datacurve/katex-multicolumn-array-spans	PASS	$5.3323	2940.4
datacurve/meriyah-explicit-resource-declarations	FAIL	$8.0184	4834.6
datacurve/python-statemachine-state-data-scoping	FAIL	$11.8156	5555.7
datacurve/scc-bounded-memory-spilling	PASS	$6.2242	3065.0
terminal-bench/build-cython-ext	PASS	$1.1801	957.6
terminal-bench/chess-best-move	PASS	$0.2447	709.3
terminal-bench/code-from-image	PASS	$0.0765	151.6
terminal-bench/constraints-scheduling	PASS	$0.1868	266.9
terminal-bench/db-wal-recovery	PASS	$0.2150	359.9
terminal-bench/dna-insert	FAIL	$0.5772	905.0
terminal-bench/extract-elf	FAIL	$0.4291	616.2
terminal-bench/gcode-to-text	PASS	$0.5037	456.9
terminal-bench/git-leak-recovery	PASS	$0.1165	205.4
terminal-bench/kv-store-grpc	FAIL	$0.1933	225.3
terminal-bench/largest-eigenval	FAIL	$0.4283	981.1
terminal-bench/log-summary-date-ranges	PASS	$0.1885	206.4
terminal-bench/merge-diff-arc-agi-task	FAIL	$0.3991	952.3
terminal-bench/modernize-scientific-stack	PASS	$0.1333	250.3
terminal-bench/multi-source-data-merger	PASS	$0.1640	275.1
terminal-bench/openssl-selfsigned-cert	PASS	$0.1162	184.3
terminal-bench/polyglot-c-py	FAIL	$0.4362	652.6
terminal-bench/regex-log	PASS	$0.3059	449.5
terminal-bench/sanitize-git-repo	PASS	$0.3586	423.0
terminal-bench/sqlite-db-truncate	PASS	$0.1714	237.0
terminal-bench/vulnerable-secret	PASS	$0.2096	226.3

A13N Harness solved datacurve/scc-bounded-memory-spilling in 3,065 seconds. None of the 12 published baseline configurations passed that task in the frozen results.

Reproducibility references

The upstream repository owns the task set, evaluation workflow, and published baseline data. This report's tables publish our selected outcomes and aggregate accounting; raw local execution artifacts are retained privately and are not a public reproduction package.

Artifact	Identity
Frozen upstream commit	e837a70bd6beb4e72eeeda62dd06e3bd34f6cb63
Published baseline JSON SHA-256	ffd18213a165e985fd7d876395a3f53e00961253a89bbc88420a41374bc20c4c
Evaluated Foundation commit	c7e287142188dd581ef37daea8fdcfa59f310857
Frozen Harness bundle SHA-256	5c095156374024ecf88d321aaccb2264d4e7b083bf80e28ab0a6c555205663ec
Retained result snapshot SHA-256	bada542cbafef661b8f253fb97d58b054131c3b1a327aca2a3f36b7c6367bb46

Conclusion

On this 30-task run, A13N Harness passed 21 tasks at $2.93 standardized cost per pass, with a 7m 03s median successful-trial time. The frozen Codex baseline passed 20 tasks at $3.47 per pass and 6m 43s. A matched-control evaluation is the next step for attributing these observed differences to the harness itself.