Models
| Model | PR | Average generation time | Cost | Average JSON size | Max JSON size | Notes |
|---|---|---|---|---|---|---|
| GPT 5.6 Sol | #57 | 25m 16s (1516.2s) | $710.82 for 15 builds | 91.58 MiB | 281.62 MiB (worldtree) | Direct OpenAI Responses routing with pro mode, max reasoning, high text verbosity, and a 128,000-token combined reasoning/output cap |
| Grok 4.5 | #58 | 1m 48s (108.2s) | $1.69 for 15 builds | 48.28 MiB | 347.95 MiB (worldtree) | Current xAI flagship via x-ai/grok-4.5; high reasoning; OpenRouter accepted a 262,144-token budget after rejecting the 500,000-token context-sized request |
Notes
Subjectively speaking, GPT-5.6 Sol seems to create the most detailed builds MineBench has seen thus far, while for the most part doing so with great creative choices. I think, personally, there are only a handful of builds I would argue are not clear improvements over GPT-5.5 Pro (like the astronaut and worldtree). On average, GPT-5.6 Sol also creates the largest JSON files across all of its builds by a significant portion. That being said, this model was also the most expensive model MineBench has benchmarked to date; the previous most expensive model was GPT-5.5 Pro at $223.90 – so 5.6 Sol totaled to being over 3x as expensive. If you're lucky enough to ignore the cost, then yes, the model created the most detailed generations yet. For example, in its cottage build, it added a scarecrow in the mini-farm, added clothes drying on a rack, etc. Its builds also seemed to have a better sense of scale and proportions overall.
TLDR: Model is amazing, doesn't tend to be conservative (good or bad depending on your use case), but it's extremely expensive.
Grok-4.5 was almost just as surprising of a model – at least when comparing previous Grok releases. I've never taken Grok to be a very serious model, but its outputs were genuinely surprisingly good; I would call them close to Sonnet 5 / Opus 4.7, though I haven't looked at the model that closely. Its price was honestly the most surprising, to the point where I had to double-check the OpenRouter configuration (it's correct, the model is just that cheap somehow – at least for now). Though it's builds did seem quite 'noisey' in terms of overall detail and shading.
We might benchmark GPT-5.6 Terra if there's enough interest, as that would technically be a closer comparison to GPT-5.5 (as Sol is technically the successor to GPT-5.5 Pro, which would also explain the cost).
What's Changed
Models
- Added GPT 5.6 Sol with direct OpenAI Responses routing and an OpenRouter Sol Pro fallback, both using pro mode, max reasoning effort, strict structured output, and a 128,000-token combined reasoning/output cap. #57
- Added Grok 4.5 with direct xAI routing, OpenRouter fallback, high reasoning effort, and a 500,000-token context-bounded output request; the benchmark's OpenRouter route accepted the 262,144-token fallback. #58
Changelog
- #57 (feat) add GPT 5.6 Sol benchmark support @Ammaar-Alam
- #58 (feat) add Grok 4.5 benchmark support @Ammaar-Alam