Skip to content

How It Works

Daniel Boles edited this page Nov 6, 2026 · 1 revision

How It Works

The harness is a short pipeline with one rule at its center: every tool shares the same method, so that adding a fifth tool means adding one file and leaving the harness alone. The pipeline takes one prompt case, sends it in one of two forms to one runner, optionally asks a judge to score the answer, prices the tokens, and appends a single row to a CSV file.

Architecture

flowchart LR
    C["corpus/rosetta_build<br/>human.md, machine.cpn,<br/>answer_key.yaml, task_spec.yaml"] --> H["run_trial()<br/>harness.py"]
    H -->|"prompt text + Task"| R["Runner.run()<br/>Claude, Azure OpenAI,<br/>Azure Translator, Ollama"]
    R -->|"RunResult: output, tokens,<br/>latency, turns"| H
    H -->|"raw output"| J["score_trial()<br/>judge.py"]
    J -->|"four 1-5 scores, mean"| H
    H --> P["estimate_cost()<br/>pricing.py"]
    P --> T["Trial row"]
    T --> S["storage.py<br/>trials.csv"]
    S --> X["export_xlsx.py<br/>raw_data + summary"]
    S --> G["export_sheets.py<br/>Google Sheet"]
Loading

The CSV in the middle is the single source of truth. Both exports derive from it and never the reverse, which removes any question about which copy is authoritative when someone edits a spreadsheet by hand.

The main parts

Part File Role
Corpus loader corpus.py Reads one case folder into a Case: the human text, the machine text, an answer key, and a task spec
Harness harness.py run_trial picks the prompt form, builds a Task for the task-completion suite, calls the runner, calls the judge, prices the result, and returns a Trial
Runners runners/*.py One adapter per tool, each with a name, a model, and run(prompt_text, task) -> RunResult
Judge judge.py Builds the rubric prompt, then parses the reply into four integer scores and a justification
Pricing pricing.py A static table of dollars per 1,000 tokens for three (tool, model) pairs; anything else costs 0.0
Storage storage.py Appends one Trial as a row of 17 columns and reads rows back as dictionaries
Exports export_xlsx.py, export_sheets.py A raw_data sheet plus a summary sheet of means per tool and prompt form
CLI cli.py bench export xlsx, bench export sheets, and the bench run stub

Two points in the table repay a closer look. The runners depend on small client protocols, such as create_message(model, prompt) -> (text, tokens_in, tokens_out) for the Claude adapter, and the real SDK wrapper sits behind that protocol in the same file, which means the runner logic is tested without a network while the wrapper is not tested at all. The Azure Translator runner is the exception to the shared shape: translation does not accept task instructions, so run raises a ValueError when it is given a Task, and the adapter belongs to the comprehension suite only.

One trial, followed end to end

The offline demo in examples/offline_demo.py runs two trials, and the first is worth following because every value in it can be checked by hand.

  1. load_case("corpus", "rosetta_build") reads four files. The human document is a Markdown view of six policy fields, such as NORMAL_PROMPT_VIEW and CONFLICT_POLICY, and the machine packet carries the same six fields plus SCHEMA_VER=1.0 in seven KEY=VALUE lines.
  2. run_trial(StubRunner(), case, "HUMAN", "TASK_COMPLETION", stub_judge, "demo-1", "2026-01-01T00:00:00Z") selects case.human_text, which is 1,226 characters long, and builds a Task from task_spec.yaml: "Summarize this policy record for a new teammate joining the Rosetta project in 3 bullet points," with three named fields to call out.
  3. The stub runner returns "stub answer for a prompt of 1226 characters", reports tokens_in as 1226 (characters standing in for tokens), tokens_out as 8, and latency_ms as 0.0.
  4. Because the suite is task completion and a judge function was supplied, score_trial sends the rubric prompt to the stub judge. The reply is CORRECTNESS: 3, COMPLETENESS: 3, FAITHFULNESS: 3, CLARITY: 3, and the parser turns it into a mean of 3.0.
  5. estimate_cost("STUB", "stub-1", 1226, 8) finds no entry in the table and returns 0.0.
  6. The Trial is appended to trials.csv. The second trial repeats the steps with the machine packet and reports 212 characters.
  7. export_to_xlsx groups the rows by tool and prompt form, and the summary sheet reads:
tool  prompt_format  mean_latency_ms  mean_tokens_in  mean_tokens_out  mean_quality_score  mean_cost_usd  n
STUB  HUMAN          0                1226            8                3                   0              1
STUB  MACHINE        0                212             8                3                   0              1

Every column except mean_tokens_in is constant by construction, which is the point of the demo: the one varying figure traces directly to the length of a file, so a reader can confirm that the pipeline carried it through without alteration.

What the structure does not yet do

The task_success and comprehension_accuracy fields exist in every row and are always empty, because the harness leaves answer-key scoring as a documented follow-up and the comprehension suite therefore has no scorer. The judge is also a function the caller must supply, so the quality column depends on a choice that this repository leaves open. Design Decisions gives the reasons for both, and Limits and Next Steps lists what remains.

Clone this wiki locally