Best position info

[Medeah]

Is there anywhere I can see which move notation, position info and prompt works best for each model?

[benjaminbrumfield]

Thanks for asking! I haven't dug very far into this yet, but I took a quick look and found a few things worth sharing.

Some background first: for every game, each language model gets a randomly chosen setup. That means SAN or UCI notation, the FEN and/or the move history, a board diagram or not, the list of legal moves or not, and one of several prompt templates. So we can compare setups fairly. I looked at about 32,000 games and 30,000 puzzle attempts, mostly at how often models play illegal moves (which forfeits the game here), plus game results and puzzle solve rates.

Here's what stood out:

• Giving the model the list of legal moves helps the most, by far. Illegal moves dropped from about 7.7 per 100 moves to 2.1. Models forfeited 37% of their games instead of 69%. It helped almost every model, and older models helped the most. gpt-4o-mini went from 35 illegal moves per 100 to under 3. On puzzles it helped weaker models a lot, but the strongest models solved about the same number with or without it.
• Include the FEN. Showing only the move history was the worst setup. Older models really struggle without the FEN (gpt-3.5-turbo made more than twice as many illegal moves). Several of the newest models, like Claude Opus 5.5, Grok 4.6, and Gemini 4 Argon, did about as well either way. They seem to follow the board from the move list alone.
• SAN vs UCI depends on the model. SAN is a little better on average, but not for every model. GPT-4.1 and GPT-4.1 mini did clearly better with UCI, while GPT-4.1 nano did much better with SAN.
• Board diagrams don't help. ASCII, Unicode, and grid diagrams made no difference on puzzles, and in games they did slightly worse.
• What's in the prompt matters more than how it's worded. Across our prompt templates, illegal move rates barely changed. The wording mostly changes how often a model's answer needs a bit of cleanup (extra formatting and so on), not how well it plays.

This was only a quick pass. I haven't looked closely at move quality, at how these settings combine, or at models with only a few games. If you're curious about a specific model, or anything else, let me know and I can take a closer look!

[Medeah]

Cool, thanks for the info