Skip to content
Daniel Boles edited this page Nov 6, 2026 · 2 revisions

rosetta-bench

The first number this harness prints describes a file, and the documentation says so before the reader has to ask.

rosetta-bench is a Python harness that runs one prompt in two forms, a verbose Markdown document and a compact KEY=VALUE packet, through several language-model tools, and records time, tokens, cost, and a judged quality score for every trial.

简体中文: rosetta-bench 用同一份提示词的两种写法(详细的 Markdown 文档与紧凑的 KEY=VALUE 数据包)测试不同的大模型工具,记录耗时、词元数、费用和评委评分;目前是一个带 38 项测试的框架,尚无任何真实模型的测试结果。

繁體中文: rosetta-bench 以同一份提示詞的兩種寫法(詳細的 Markdown 文件與精簡的 KEY=VALUE 資料包)測試不同的大型語言模型工具,記錄耗時、詞元數、費用與評審評分;目前是一個附有 38 項測試的框架,尚無任何真實模型的測試結果。

At a glance

What it is A benchmark harness for comparing a human-readable prompt with a machine-readable packet across model tools
Language Python 3.11 or later (run here on 3.12.10)
Tests 38 tests, all passing, no network or API key needed
Status The harness works offline. bench run is a stub, the price table has no verified source, and no real benchmark results exist
Licence Apache-2.0

Why it matters

Instructions written for a language model can be long and readable, short and strict, or both, and teams usually settle the choice on preference because a fair comparison needs the same task, the same model, and the same scoring applied to each form. rosetta-bench turns the preference into an experiment: the task goes to the same tool in each form, and the differences in tokens, cost, speed, and judged quality land in one spreadsheet. The harness runs and its tests pass; however, the experiment itself has not been run against a real model, and this wiki keeps that distinction in front of the reader.

Skills shown

  • Evaluation design: a comprehension suite and a task-completion suite, kept apart so that readability and outcome are not conflated.
  • Interface-first adapters: four tools (Claude, Azure OpenAI, Azure Translator, Ollama) behind one Runner.run method, with the network clients injected so that tests can substitute fakes.
  • Rubric-based model judging: a four-dimension prompt and a strict parser that raises an error when a score is missing.
  • Data handling: an append-only CSV as the single source of truth, exported to .xlsx with openpyxl.
  • Test discipline: 38 tests that finish in about two seconds, plus a demo that needs no key.
  • Documentation that records its own limits, including the ones that weaken the project.

Results

The results are of two kinds, and only one kind exists. For the harness there are real numbers, because 38 of 38 tests pass and the offline demo carries two trials, one per prompt form, from corpus to spreadsheet. For the question the harness was built to answer there are none: no real model has been run, no quality score comes from a real judge, bench run prints a message and exits with code 1, and the price table in pricing.py is a placeholder that nobody has checked against a price list.

The demo does print two figures, and they describe the corpus rather than a model:

HUMAN    prompt characters: 1226
MACHINE  prompt characters: 212

The packet is about 5.8 times shorter than the document in characters, a property of the two files in corpus/rosetta_build/ and not a finding about any tool. See Evidence for how that was produced and what it does not show.

Pages

Clone this wiki locally