Repository navigation
How It Works
The harness is a short pipeline with one rule at its center: every tool shares the same method, so that adding a fifth tool means adding one file and leaving the harness alone. The pipeline takes one prompt case, sends it in one of two forms to one runner, optionally asks a judge to score the answer, prices the tokens, and appends a single row to a CSV file.
flowchart LR
C["corpus/rosetta_build<br/>human.md, machine.cpn,<br/>answer_key.yaml, task_spec.yaml"] --> H["run_trial()<br/>harness.py"]
H -->|"prompt text + Task"| R["Runner.run()<br/>Claude, Azure OpenAI,<br/>Azure Translator, Ollama"]
R -->|"RunResult: output, tokens,<br/>latency, turns"| H
H -->|"raw output"| J["score_trial()<br/>judge.py"]
J -->|"four 1-5 scores, mean"| H
H --> P["estimate_cost()<br/>pricing.py"]
P --> T["Trial row"]
T --> S["storage.py<br/>trials.csv"]
S --> X["export_xlsx.py<br/>raw_data + summary"]
S --> G["export_sheets.py<br/>Google Sheet"]
The CSV in the middle is the single source of truth. Both exports derive from it and never the reverse, which removes any question about which copy is authoritative when someone edits a spreadsheet by hand.
| Part | File | Role |
|---|---|---|
| Corpus loader | corpus.py |
Reads one case folder into a Case: the human text, the machine text, an answer key, and a task spec |
| Harness | harness.py |
run_trial picks the prompt form, builds a Task for the task-completion suite, calls the runner, calls the judge, prices the result, and returns a Trial
|
| Runners | runners/*.py |
One adapter per tool, each with a name, a model, and run(prompt_text, task) -> RunResult
|
| Judge | judge.py |
Builds the rubric prompt, then parses the reply into four integer scores and a justification |
| Pricing | pricing.py |
A static table of dollars per 1,000 tokens for three (tool, model) pairs; anything else costs 0.0 |
| Storage | storage.py |
Appends one Trial as a row of 17 columns and reads rows back as dictionaries |
| Exports |
export_xlsx.py, export_sheets.py
|
A raw_data sheet plus a summary sheet of means per tool and prompt form |
| CLI | cli.py |
bench export xlsx, bench export sheets, and the bench run stub |
Two points in the table repay a closer look. The runners depend on small client protocols, such as create_message(model, prompt) -> (text, tokens_in, tokens_out) for the Claude adapter, and the real SDK wrapper sits behind that protocol in the same file, which means the runner logic is tested without a network while the wrapper is not tested at all. The Azure Translator runner is the exception to the shared shape: translation does not accept task instructions, so run raises a ValueError when it is given a Task, and the adapter belongs to the comprehension suite only.
The offline demo in examples/offline_demo.py runs two trials, and the first is worth following because every value in it can be checked by hand.
-
load_case("corpus", "rosetta_build")reads four files. The human document is a Markdown view of six policy fields, such asNORMAL_PROMPT_VIEWandCONFLICT_POLICY, and the machine packet carries the same six fields plusSCHEMA_VER=1.0in sevenKEY=VALUElines. -
run_trial(StubRunner(), case, "HUMAN", "TASK_COMPLETION", stub_judge, "demo-1", "2026-01-01T00:00:00Z")selectscase.human_text, which is 1,226 characters long, and builds aTaskfromtask_spec.yaml: "Summarize this policy record for a new teammate joining the Rosetta project in 3 bullet points," with three named fields to call out. - The stub runner returns
"stub answer for a prompt of 1226 characters", reportstokens_inas 1226 (characters standing in for tokens),tokens_outas 8, andlatency_msas 0.0. - Because the suite is task completion and a judge function was supplied,
score_trialsends the rubric prompt to the stub judge. The reply isCORRECTNESS: 3,COMPLETENESS: 3,FAITHFULNESS: 3,CLARITY: 3, and the parser turns it into a mean of 3.0. -
estimate_cost("STUB", "stub-1", 1226, 8)finds no entry in the table and returns 0.0. - The
Trialis appended totrials.csv. The second trial repeats the steps with the machine packet and reports 212 characters. -
export_to_xlsxgroups the rows by tool and prompt form, and thesummarysheet reads:
tool prompt_format mean_latency_ms mean_tokens_in mean_tokens_out mean_quality_score mean_cost_usd n
STUB HUMAN 0 1226 8 3 0 1
STUB MACHINE 0 212 8 3 0 1
Every column except mean_tokens_in is constant by construction, which is the point of the demo: the one varying figure traces directly to the length of a file, so a reader can confirm that the pipeline carried it through without alteration.
The task_success and comprehension_accuracy fields exist in every row and are always empty, because the harness leaves answer-key scoring as a documented follow-up and the comprehension suite therefore has no scorer. The judge is also a function the caller must supply, so the quality column depends on a choice that this repository leaves open. Design Decisions gives the reasons for both, and Limits and Next Steps lists what remains.
Designed and authored by Daniel Boles; implemented with coding agents. Licence: Apache-2.0.