Skip to content

Evidence

Daniel Boles edited this page Nov 6, 2026 · 1 revision

Evidence

The evidence for this project divides cleanly into what the repository demonstrates and what it leaves untested, and the division is the most useful thing the page can state. The harness works offline, with fakes at every network boundary; the experiment it exists to run has not started. A reader who needs a benchmark result will not find one here, and a reader who needs to judge the quality of the code will find a small, tested, and honestly bounded base.

Status in one paragraph

bench run is a stub: it prints a message to standard error and exits with code 1. No real benchmark results exist, so no claim about which prompt form is faster, cheaper, or better follows from anything in the repository. The price table in src/rosetta_bench/pricing.py holds three entries, was never checked against a published price list, and charges 0.0 for any tool and model pair it does not list, so a cost column from this harness deserves no trust until the table carries sourced values.

Tests

The suite holds 38 tests in 15 files. All 38 passed on the run recorded for this wiki (Python 3.12.10 on Windows), and the suite skips nothing.

File Tests What it checks
test_azure_openai_runner.py 3 The adapter sends prompt text verbatim without a task, appends task instructions with one, and exposes its name and model; all against a fake client
test_claude_runner.py 3 The same three checks for the Claude adapter
test_ollama_runner.py 3 The same three checks for the local Ollama adapter
test_azure_translation_runner.py 3 A round-trip translation, the refusal of a Task, and name and model
test_runners_base.py 2 A fake satisfies the Runner protocol, and the protocol is structural rather than nominal
test_judge.py 6 Prompt building with and without a reference, multi-line justification parsing, the mean score, the call path through score_trial, and the KeyError on a missing field
test_pricing.py 4 A known pair has a positive cost, an unknown pair costs 0.0, zero tokens cost 0.0, and local Ollama is free
test_harness.py 3 A comprehension trial sends no task and selects the machine text, a task trial builds its task from the case spec, and the judge is called when one is supplied
test_models.py 3 Task holds optional reference output, RunResult defaults to one turn, and Trial.to_row emits the canonical columns in order
test_storage.py 2 The CSV is created with a header, and a second append does not repeat it
test_export_xlsx.py 1 The workbook has raw_data and summary sheets
test_export_sheets.py 1 The Sheets export opens the named sheet and writes both worksheets, against a fake client
test_corpus.py 1 A case loads all four of its files
test_cli.py 2 export xlsx writes a file, and run returns a nonzero code with an explanation
test_end_to_end.py 1 Corpus, two trials, CSV, and workbook in one pass

The end-to-end test is the strongest single check, because it asserts across module boundaries. It scores both trials with a fake judge that replies 5, 5, 4, and 5, and asserts that the stored quality score is exactly 4.75, which is the mean of the four.

What the demo shows

examples/offline_demo.py runs the same path without a test runner and prints the two figures below. The tool is a stub that reports the length of the prompt it receives, so the output describes the corpus.

HUMAN    prompt characters: 1226
MACHINE  prompt characters: 212

The ratio is 1226 to 212, about 5.8 to 1. On disk the two files measure 1,251 and 219 bytes in this Windows checkout, because the line endings are CRLF and the human file holds two three-byte em dashes; Python's text mode reads them as 1,226 and 212 characters. The ratio therefore shows that the packet is shorter than the document, which anyone could see by opening the files, and it shows nothing about how a model treats either form.

The workbook the demo writes has two sheets. In raw_data, the numeric columns arrive as text, because the rows are read back from the CSV as strings and written unchanged; the summary sheet converts them and holds real numbers. That is a rough edge in the export, not an error in the arithmetic, and it is listed below.

Check it yourself in 5 minutes

git clone https://github.com/dboles99/rosetta-bench.git
cd rosetta-bench
python -m venv .venv
.venv/bin/pip install -e ".[dev]"      # Windows: .venv\Scripts\pip install -e ".[dev]"
.venv/bin/python -m pytest -q
.venv/bin/python examples/offline_demo.py
.venv/bin/bench run ; echo "exit code: $?"

Expected results: 38 passed, the two prompt-length lines followed by a wrote line naming a CSV and a workbook in a temporary folder, and, for the last command, the "not automated in v1" message with exit code: 1. To see the stub's rows, open the workbook that the demo names and read the summary sheet: both rows carry the tool name STUB, a quality score of 3, and a cost of 0.

Known failures and gaps

  • No real model has been run. The only corpus case is one small example derived from the design of Rosetta itself, so even a successful live run would not support a general claim.
  • bench run is a stub. A live run means writing a short script around run_trial, as shown in Getting Started.
  • The price table is unverified. Three entries, no source, and silent zero for everything else.
  • Live adapters are untested. AnthropicSDKClient, AzureOpenAISDKClient, AzureTranslatorSDKClient, and OllamaHTTPClient make real network calls and sit outside the suite; the Google Sheets export has the same status.
  • No judge client ships with the harness. The caller supplies a function from prompt text to response text, so the judge's model, temperature, and bias are all unrecorded.
  • comprehension_accuracy and task_success are always empty. The answer key is loaded and never scored.
  • Costs are not rounded. A fake run of 100 input tokens and 20 output tokens priced at the table's Claude rates produced a cost of 0.0006000000000000001 in the CSV, which is floating-point noise.
  • The demo's latency is 0.0 because the stub does not wait, so the latency column in the demo workbook carries no information.

Clone this wiki locally