Question about the benchmark/scratchpad models

[echai58]

Hi, I had a question about the benchmark models. So it seems like in the recent forecasts, only zero shot and zero shot + freeze values are there.

In the paper, I see mentions of scratchpad and news as options too. Are these no longer run? If so, why? And does that mean that the current benchmarks do not have any recent news, etc.?

[houtanb]

For a few reasons:

• scratchpad / chain of thought prompting worked better in older models than it does in newer models
• our zero short prompts were attaining SOTA performance (backs up the first point) so the scratchpad ones were redundant
• our desire to redirect that budget towards models with different reasoning levels and tool use (forthcoming in the next 1 or 2 rounds)