Repository navigation
Getting Started
This page takes a stranger from an empty machine to a passing test run and a finished demo, and it ends with the shape of a live run for readers who hold an API key. Everything before that last section needs no key and no network access.
- Python 3.11 or later; every check recorded on this wiki ran on 3.12.10.
- Git, to clone the repository.
- Network access during install only, because the dependencies include the Anthropic SDK, gspread, google-auth, and openpyxl.
- Nothing else for the tests and the demo. A live run needs an API key for the tool under test, and that part did not run for this wiki.
git clone https://github.com/dboles99/rosetta-bench.git
cd rosetta-bench
python -m venv .venv
.venv/bin/pip install -e ".[dev]" # Windows: .venv\Scripts\pip install -e ".[dev]"The [azure] extra adds the OpenAI SDK for the Azure OpenAI adapter; the base install does not need it.
.venv/bin/python -m pytest -q # Windows: .venv\Scripts\python -m pytest -qExpected output, from a run on Windows with Python 3.12.10:
...................................... [100%]
38 passed in 1.02s
The timing varies from run to run; the count of 38 does not.
.venv/bin/python examples/offline_demo.pyExpected output, with the temporary path shortened because it differs on every run:
HUMAN prompt characters: 1226
MACHINE prompt characters: 212
wrote <temp dir>/trials.csv and <temp dir>/trials.xlsx
The demo loads the one corpus case, runs it through a stub in place of a model, scores it with a stub judge that always answers 3, appends two rows to a CSV, and exports a workbook with a raw_data sheet and a summary sheet. The stub reports the length of the prompt it received, so the two figures describe the files in corpus/rosetta_build/; they say nothing about any model. How It Works follows the same two trials step by step.
A CSV written by the harness converts to a workbook with one command, and this is the only bench subcommand that is fully wired:
.venv/bin/bench export xlsx --csv trials.csv --out trials.xlsxwrote trials.xlsx
bench export sheets --csv trials.csv --sheet-name <name> does the same for Google Sheets, and it needs GOOGLE_SERVICE_ACCOUNT_JSON to point at a service-account file. That route is covered by a test with a fake client only.
Nothing useful yet, and it says so:
.venv/bin/bench run`bench run` is not automated in v1 -- requires live credentials for each tool. Use rosetta_bench.harness.run_trial directly with the concrete client adapters (ClaudeRunner + AnthropicSDKClient, etc.).
The message goes to standard error and the exit code is 1. A live run is a short script of the reader's own, as below.
Copy .env.example to .env for the variable names, then wire a runner and a judge function. The harness takes the judge as a plain function from prompt text to response text, because no judge client ships with it.
from pathlib import Path
from rosetta_bench.corpus import load_case
from rosetta_bench.harness import run_trial
from rosetta_bench.runners.claude_runner import AnthropicSDKClient, ClaudeRunner
from rosetta_bench.storage import append_trial
client = AnthropicSDKClient("<your API key>")
runner = ClaudeRunner("<model name>", client)
def judge_call_fn(prompt: str) -> str:
text, _tokens_in, _tokens_out = client.create_message("<judge model name>", prompt)
return text
case = load_case(Path("corpus"), "rosetta_build")
for n, prompt_format in enumerate(("HUMAN", "MACHINE"), start=1):
trial = run_trial(runner, case, prompt_format, "TASK_COMPLETION",
judge_call_fn, f"t{n}", "2026-11-06T00:00:00Z")
append_trial(trial, Path("trials.csv"))The harness half of this script was run here with a fake client in place of AnthropicSDKClient, and it produced two rows with a judged score of 4.0; the live half, which makes the network call, was not run. Before trusting the cost column, read Limits and Next Steps on the price table.
Designed and authored by Daniel Boles; implemented with coding agents. Licence: Apache-2.0.