Skip to content

Getting Started

Daniel Boles edited this page Nov 6, 2026 · 1 revision

Getting Started

This page takes a stranger from an empty machine to a passing test run and a finished demo, and it ends with the shape of a live run for readers who hold an API key. Everything before that last section needs no key and no network access.

What it needs

  • Python 3.11 or later; every check recorded on this wiki ran on 3.12.10.
  • Git, to clone the repository.
  • Network access during install only, because the dependencies include the Anthropic SDK, gspread, google-auth, and openpyxl.
  • Nothing else for the tests and the demo. A live run needs an API key for the tool under test, and that part did not run for this wiki.

Install

git clone https://github.com/dboles99/rosetta-bench.git
cd rosetta-bench
python -m venv .venv
.venv/bin/pip install -e ".[dev]"      # Windows: .venv\Scripts\pip install -e ".[dev]"

The [azure] extra adds the OpenAI SDK for the Azure OpenAI adapter; the base install does not need it.

First run: the tests

.venv/bin/python -m pytest -q          # Windows: .venv\Scripts\python -m pytest -q

Expected output, from a run on Windows with Python 3.12.10:

......................................                                   [100%]
38 passed in 1.02s

The timing varies from run to run; the count of 38 does not.

Second run: the offline demo

.venv/bin/python examples/offline_demo.py

Expected output, with the temporary path shortened because it differs on every run:

HUMAN    prompt characters: 1226
MACHINE  prompt characters: 212
wrote <temp dir>/trials.csv and <temp dir>/trials.xlsx

The demo loads the one corpus case, runs it through a stub in place of a model, scores it with a stub judge that always answers 3, appends two rows to a CSV, and exports a workbook with a raw_data sheet and a summary sheet. The stub reports the length of the prompt it received, so the two figures describe the files in corpus/rosetta_build/; they say nothing about any model. How It Works follows the same two trials step by step.

Export a CSV by hand

A CSV written by the harness converts to a workbook with one command, and this is the only bench subcommand that is fully wired:

.venv/bin/bench export xlsx --csv trials.csv --out trials.xlsx
wrote trials.xlsx

bench export sheets --csv trials.csv --sheet-name <name> does the same for Google Sheets, and it needs GOOGLE_SERVICE_ACCOUNT_JSON to point at a service-account file. That route is covered by a test with a fake client only.

What bench run does

Nothing useful yet, and it says so:

.venv/bin/bench run
`bench run` is not automated in v1 -- requires live credentials for each tool. Use rosetta_bench.harness.run_trial directly with the concrete client adapters (ClaudeRunner + AnthropicSDKClient, etc.).

The message goes to standard error and the exit code is 1. A live run is a short script of the reader's own, as below.

The shape of a live run

Copy .env.example to .env for the variable names, then wire a runner and a judge function. The harness takes the judge as a plain function from prompt text to response text, because no judge client ships with it.

from pathlib import Path

from rosetta_bench.corpus import load_case
from rosetta_bench.harness import run_trial
from rosetta_bench.runners.claude_runner import AnthropicSDKClient, ClaudeRunner
from rosetta_bench.storage import append_trial

client = AnthropicSDKClient("<your API key>")
runner = ClaudeRunner("<model name>", client)

def judge_call_fn(prompt: str) -> str:
    text, _tokens_in, _tokens_out = client.create_message("<judge model name>", prompt)
    return text

case = load_case(Path("corpus"), "rosetta_build")
for n, prompt_format in enumerate(("HUMAN", "MACHINE"), start=1):
    trial = run_trial(runner, case, prompt_format, "TASK_COMPLETION",
                      judge_call_fn, f"t{n}", "2026-11-06T00:00:00Z")
    append_trial(trial, Path("trials.csv"))

The harness half of this script was run here with a fake client in place of AnthropicSDKClient, and it produced two rows with a judged score of 4.0; the live half, which makes the network call, was not run. Before trusting the cost column, read Limits and Next Steps on the price table.

Clone this wiki locally