GERB tests whether an irrelevant "red herring" corrupts a language model's economic reasoning. It is built on sixty graduate-level analytical economics problems, each with one objectively verifiable answer and a full step-by-step reference solution. Every problem is fielded in a 2×2 within-task design that crosses a content-preserving red herring, an economically irrelevant passage that leaves the correct answer and the operative mechanism unchanged, with a request for the model to explain itself. Because the manipulation is within task, the red herring is orthogonal to task difficulty by construction, and its effect is identified against model and task fixed effects rather than measured as a raw accuracy drop.
GERB follows a public design, private content model. The design, grading rules, code, and full results are open. The problem statements and reference solutions are held privately so the benchmark stays out of training corpora, and are released only under a data-use agreement. See DATA_ACCESS.md.
GERB shares its sixty-problem corpus with GEMB, the Graduate Economics Metacognition Benchmark, which applies a different protocol to the same problems. The two benchmarks differ in what they measure, not in the underlying problems.
| Path | Contents |
|---|---|
| PROTOCOL.md | The full experimental design, the grading rubric, and the judge protocol |
results/ |
The outcome tables and per-model estimates reported in the paper |
examples/ |
A small number of illustrative problems, enough to convey the style |
| CITATION.cff, CITATION.bib | How to cite GERB |
| DATA_ACCESS.md | How to request the private problem set, and the terms |
The sixty problem statements and their reference solutions. Releasing them publicly would let them leak into the next training run and destroy the benchmark's resistance to contamination, so they are served only under the agreement in DATA_ACCESS.md. Because scoring a model on GERB requires these problems, the benchmark is fully reproducible in method from this repository, and reproducible in data once access is granted.
Thirty-eight language models answered all four versions of all sixty problems, five times each,
in independent stateless calls at temperature zero. An independent judge model graded every
response against the verified answer and the worked solution, and a second judge regraded a
subsample. The red herring lowers the probability of a correct answer by about twelve percentage
points, the effect holds across reasoning types, and it is largest on the problems a model rates
as easy. Full results are in the paper and in results/.
Cite the paper and, once assigned, the archived release DOI. Ready-to-use entries are in
CITATION.bib. The paper's arXiv identifier and the Zenodo DOI are filled in on
release; until then they read <to be assigned>.
The materials in this repository are released under CC BY 4.0, except the code, which is released under the MIT license in LICENSE-CODE.txt. The private problem set is not covered by either license and is governed solely by the data-use agreement in DATA_ACCESS.md.
Piyush Akimitsu, Department of Economics, University at Albany, SUNY. Contact address for access requests is in DATA_ACCESS.md.