Repository navigation
Limits and Next Steps
Daniel Boles edited this page Nov 6, 2026
·
1 revision
The limits of this project are structural rather than cosmetic, because the main one is that the experiment has not been run. A stranger reading the code would find a clean harness; a stranger reading only the README would find the same facts, listed plainly. This page repeats them in one place and then orders the work that would turn the harness into evidence.
| Limit | Consequence |
|---|---|
| No real model has been run | No claim about which prompt form is faster, cheaper, or better follows from this repository |
bench run is a stub |
A live run is a script written around run_trial; the command prints a message and exits with code 1 |
| One corpus case of 1,251 and 219 bytes | Far too small for a general claim, and derived from the design of Rosetta itself, so a favorable result would partly echo its author |
| The price table has three entries and no source | Costs are placeholders, and any unlisted tool and model pair costs 0.0 |
| The four SDK and HTTP wrappers are untested | The logic around each call is tested with fakes, and the call is not |
| The Google Sheets export has never met the real service | It needs a service-account file and is tested against a fake client only |
| No judge client ships with the harness | The caller chooses the judge, and the row does not record which one it was |
comprehension_accuracy and task_success are always empty |
The answer key loads and nothing scores against it |
Numbers in the raw_data sheet are text |
The summary sheet holds real numbers, and the raw sheet needs conversion before it is sorted or summed |
The order below follows dependency, not preference, and no dates are attached.
- Verify the price table. Replace the placeholders with dated, sourced rates, and make an unknown pair an error or a visible flag instead of a zero. Every later cost figure depends on this.
-
Score comprehension. Implement
comprehension_accuracyagainstanswer_key.yaml, exact match first and a judged semantic match as the fallback for free-text answers, which is the order the design notes propose. -
Wire
bench run. Read credentials from the environment, build the runners, and loop over tools, prompt forms, and suites, so that a run is one command and not a hand-written script. - Record the judge. Add a column for the judging model, and repeat each judgment enough times to report its variance, because a quality score without a spread invites more confidence than it earns.
- Add cases that Rosetta did not write. The independent corpus named as Stage 2 in the design notes is the only route to a finding that does not echo the format's own author.
- Repeat trials. Each cell in the demo has n = 1; a result needs several trials per tool, form, and suite, and the summary sheet already reports n.
Until the first two steps are done, the cost and comprehension columns should be treated as scaffolding, and until the fifth is done, any result should be reported as a result on one small case.
Designed and authored by Daniel Boles; implemented with coding agents. Licence: Apache-2.0.