Repository navigation
Evidence
The evidence for this project divides cleanly into what the repository demonstrates and what it leaves untested, and the division is the most useful thing the page can state. The harness works offline, with fakes at every network boundary; the experiment it exists to run has not started. A reader who needs a benchmark result will not find one here, and a reader who needs to judge the quality of the code will find a small, tested, and honestly bounded base.
bench run is a stub: it prints a message to standard error and exits with code 1. No real benchmark results exist, so no claim about which prompt form is faster, cheaper, or better follows from anything in the repository. The price table in src/rosetta_bench/pricing.py holds three entries, was never checked against a published price list, and charges 0.0 for any tool and model pair it does not list, so a cost column from this harness deserves no trust until the table carries sourced values.
The suite holds 38 tests in 15 files. All 38 passed on the run recorded for this wiki (Python 3.12.10 on Windows), and the suite skips nothing.
| File | Tests | What it checks |
|---|---|---|
test_azure_openai_runner.py |
3 | The adapter sends prompt text verbatim without a task, appends task instructions with one, and exposes its name and model; all against a fake client |
test_claude_runner.py |
3 | The same three checks for the Claude adapter |
test_ollama_runner.py |
3 | The same three checks for the local Ollama adapter |
test_azure_translation_runner.py |
3 | A round-trip translation, the refusal of a Task, and name and model |
test_runners_base.py |
2 | A fake satisfies the Runner protocol, and the protocol is structural rather than nominal |
test_judge.py |
6 | Prompt building with and without a reference, multi-line justification parsing, the mean score, the call path through score_trial, and the KeyError on a missing field |
test_pricing.py |
4 | A known pair has a positive cost, an unknown pair costs 0.0, zero tokens cost 0.0, and local Ollama is free |
test_harness.py |
3 | A comprehension trial sends no task and selects the machine text, a task trial builds its task from the case spec, and the judge is called when one is supplied |
test_models.py |
3 |
Task holds optional reference output, RunResult defaults to one turn, and Trial.to_row emits the canonical columns in order |
test_storage.py |
2 | The CSV is created with a header, and a second append does not repeat it |
test_export_xlsx.py |
1 | The workbook has raw_data and summary sheets |
test_export_sheets.py |
1 | The Sheets export opens the named sheet and writes both worksheets, against a fake client |
test_corpus.py |
1 | A case loads all four of its files |
test_cli.py |
2 |
export xlsx writes a file, and run returns a nonzero code with an explanation |
test_end_to_end.py |
1 | Corpus, two trials, CSV, and workbook in one pass |
The end-to-end test is the strongest single check, because it asserts across module boundaries. It scores both trials with a fake judge that replies 5, 5, 4, and 5, and asserts that the stored quality score is exactly 4.75, which is the mean of the four.
examples/offline_demo.py runs the same path without a test runner and prints the two figures below. The tool is a stub that reports the length of the prompt it receives, so the output describes the corpus.
HUMAN prompt characters: 1226
MACHINE prompt characters: 212
The ratio is 1226 to 212, about 5.8 to 1. On disk the two files measure 1,251 and 219 bytes in this Windows checkout, because the line endings are CRLF and the human file holds two three-byte em dashes; Python's text mode reads them as 1,226 and 212 characters. The ratio therefore shows that the packet is shorter than the document, which anyone could see by opening the files, and it shows nothing about how a model treats either form.
The workbook the demo writes has two sheets. In raw_data, the numeric columns arrive as text, because the rows are read back from the CSV as strings and written unchanged; the summary sheet converts them and holds real numbers. That is a rough edge in the export, not an error in the arithmetic, and it is listed below.
git clone https://github.com/dboles99/rosetta-bench.git
cd rosetta-bench
python -m venv .venv
.venv/bin/pip install -e ".[dev]" # Windows: .venv\Scripts\pip install -e ".[dev]"
.venv/bin/python -m pytest -q
.venv/bin/python examples/offline_demo.py
.venv/bin/bench run ; echo "exit code: $?"Expected results: 38 passed, the two prompt-length lines followed by a wrote line naming a CSV and a workbook in a temporary folder, and, for the last command, the "not automated in v1" message with exit code: 1. To see the stub's rows, open the workbook that the demo names and read the summary sheet: both rows carry the tool name STUB, a quality score of 3, and a cost of 0.
- No real model has been run. The only corpus case is one small example derived from the design of Rosetta itself, so even a successful live run would not support a general claim.
-
bench runis a stub. A live run means writing a short script aroundrun_trial, as shown in Getting Started. - The price table is unverified. Three entries, no source, and silent zero for everything else.
-
Live adapters are untested.
AnthropicSDKClient,AzureOpenAISDKClient,AzureTranslatorSDKClient, andOllamaHTTPClientmake real network calls and sit outside the suite; the Google Sheets export has the same status. - No judge client ships with the harness. The caller supplies a function from prompt text to response text, so the judge's model, temperature, and bias are all unrecorded.
-
comprehension_accuracyandtask_successare always empty. The answer key is loaded and never scored. -
Costs are not rounded. A fake run of 100 input tokens and 20 output tokens priced at the table's Claude rates produced a cost of
0.0006000000000000001in the CSV, which is floating-point noise. - The demo's latency is 0.0 because the stub does not wait, so the latency column in the demo workbook carries no information.
Designed and authored by Daniel Boles; implemented with coding agents. Licence: Apache-2.0.