Repository navigation
Design Decisions
Six decisions shape the harness, and each is recorded in docs/design.md or visible in the 1,361 lines of Python under src, tests, and examples. Several of them changed between the design and the code, and those are described as they happened.
A single quality number would have been simpler to build and to chart, but it would have blended two different questions: whether a tool can read a prompt, and whether reading it produces a good result. A tool might answer every factual question about the packet correctly and still write a poor summary from it, and a blended score would hide which half had failed.
The design therefore defines a comprehension suite, which asks the same battery of factual questions about either prompt form and scores them against an answer key, and a task-completion suite, which sets one concrete task and scores the output with a judge. The cost is a larger corpus, since each case needs both an answer_key.yaml and a task_spec.yaml, and the comprehension half is unfinished: the five questions in answer_key.yaml load, and nothing scores against them, so comprehension_accuracy stays empty in every row.
The decision would change if a first real run showed the two suites tracking each other so closely that one is redundant.
The results have three audiences, a person reading a spreadsheet, a person using Google Sheets, and any script that wants to recompute a mean, and each could have been given its own storage. Writing directly to a workbook would have been easy, although the first hand edit would then have made the workbook and any other copy disagree about what happened.
The harness appends one row per trial to a CSV and generates the workbook and the Sheets export from it, never the reverse. The trade-off is that a CSV has no types, so the raw_data sheet holds numbers as text, and that the code makes no provision for two processes appending at once. A move to SQLite would be the first change if more than one writer, or any ad hoc querying, became real.
Each runner could have called its vendor SDK directly, which is shorter, or could have received a client object that satisfies a one-method protocol, which is longer and testable. The harness takes the second route: ClaudeRunner depends on create_message(model, prompt) -> (text, tokens_in, tokens_out), and the real AnthropicSDKClient sits behind it in the same file.
The consequence is that the timing, the argument assembly, and the RunResult construction of every runner are tested with fakes, while the four SDK wrappers have no test at all. A fake can agree with a misunderstanding of the real API, so the protocol gives confidence in the logic around a call and none in the call. A small set of recorded responses, or a gated live smoke test that runs only when a key is present, would close the gap.
The design called for one Claude call per trial, with the task spec, the rubric, and the reference output as input. The implementation reduced that to a type alias, JudgeCallFn = Callable[[str], str]: the harness builds the rubric prompt, hands it to the function, and parses the reply, which must contain CORRECTNESS, COMPLETENESS, FAITHFULNESS, and CLARITY lines or the parser raises a KeyError.
Raising on a missing field is deliberate, because a judge that skips a dimension and a harness that fills the gap with a default would produce a quality column that looked complete and meant nothing. The price of the function-shaped judge is that the row does not record which model judged it, so two runs scored by different judges cannot be told apart afterward. Adding a judge-model column, and repeating each judgment to measure its variance, would be the first extension.
Cost per trial needs a price per thousand tokens for each tool, and the choices were to fetch current prices live, to keep a table in the source, or to leave cost out. The design said the exact table needed published rates at implementation time and should not be guessed; the implementation then shipped a three-entry placeholder, and the README says it was not checked.
The weakness is in the default. A tool and model pair that is missing from the table costs 0.0, so a typo in a model name makes a trial look free instead of making it fail. The decision was a shortcut, and it is recorded here as one. A dated table with a source for every entry, and an error or a visible flag for an unknown pair, would replace it.
Two scope decisions were taken early. Two interactive coding assistants were excluded because neither has a stable scriptable interface, which leaves the four tools that do. The independent prompt corpus, meant to show that the first results are not an artifact of Rosetta's own design, was deferred as a Stage 2 that the notes name and do not design.
The implemented layout also came out flatter than the design sketched: judge.py, storage.py, and export_xlsx.py are single modules, and bench run was never wired to the live services, since each tool needs its own credentials and a run is exercised by hand. The design notes say so in one sentence at the head of the architecture section. The first real run, and a second corpus case that Rosetta did not write, would each reopen this scope.
Designed and authored by Daniel Boles; implemented with coding agents. Licence: Apache-2.0.